Chen He created SPARK-13628:
-------------------------------
Summary: Temporary intermediate output file should be renamed
before copying to destination filesystem
Key: SPARK-13628
URL: https://issues.apache.org/jira/browse/SPARK-13628
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 1.6.0
Reporter: Chen He
Spark Executor will dump temporary file into local temp dir, copy it to
destination filesystem, and then, rename it. It could be costly for Blobstore
(such as openstack swift) which do the actual copy when file is renamed. If it
does not affect other components, we may switch the sequence of copy and rename
so that Spark can use Blobstore as final output destination.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
