Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/21048
if this is being looked at again, it'd be nice to have a reference back end
which just did the write straight to the destination: this is exactly what all
the public cloud stores (s3, azure-*, gcs) offer as their atomic write. There'd
be no need to make things clever and automatically switch; just make it
something people can ask for.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
