Merge pull request #466 from liyinan926/file-overwrite-new Allow files added through SparkContext.addFile() to be overwritten
This is useful for the cases when a file needs to be refreshed and downloaded by the executors periodically. For example, a possible use case is: the driver periodically renews a Hadoop delegation token and writes it to a token file. The token file needs to be downloaded by the executors whenever it gets renewed. However, the current implementation throws an exception when the target file exists and its contents do not match those of the new source. This PR adds an option to allow files to be overwritten to support use cases similar to the above. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/84670f27 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/84670f27 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/84670f27 Branch: refs/heads/master Commit: 84670f2715392859624df290c1b52eb4ed4a9cb1 Parents: 3d5c03e 584323c Author: Reynold Xin <r...@apache.org> Authored: Mon Jan 27 17:08:35 2014 -0800 Committer: Reynold Xin <r...@apache.org> Committed: Mon Jan 27 17:08:35 2014 -0800 ---------------------------------------------------------------------- .../scala/org/apache/spark/util/Utils.scala | 49 ++++++++++++++------ docs/configuration.md | 7 +++ 2 files changed, 41 insertions(+), 15 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/84670f27/core/src/main/scala/org/apache/spark/util/Utils.scala ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/84670f27/docs/configuration.md ----------------------------------------------------------------------
