Github user gerashegalov commented on the issue:
https://github.com/apache/spark/pull/20575
Thanks for suggestions, I will look into them. I agree that a solution into
the right direction will definitely involve changing the write call path. I did
not go down this path because I have no control over my customer's Spark
versions (at least for now).
There is another long standing issue with FsHistoryProvider. It uses
`rename` which is inefficient on S3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
