Chris Kanich created SPARK-19417:
------------------------------------
Summary: spark.files.overwrite is ignored
Key: SPARK-19417
URL: https://issues.apache.org/jira/browse/SPARK-19417
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.0
Reporter: Chris Kanich
I have not been able to get Spark to actually overwrite a file after I have
changed it on the driver node, re-called addFile, and then used it on the
executors again. Here's a failing test.
{code}
test("can overwrite files when spark.files.overwrite is true") {
val dir = Utils.createTempDir()
val file = new File(dir, "file")
try {
Files.write("one", file, StandardCharsets.UTF_8)
sc = new SparkContext(new
SparkConf().setAppName("test").setMaster("local-cluster[1,1,1024]")
.set("spark.files.overwrite", "true"))
sc.addFile(file.getAbsolutePath)
def getAddedFileContents(): String = {
sc.parallelize(Seq(0)).map { _ =>
scala.io.Source.fromFile(SparkFiles.get("file")).mkString
}.first()
}
assert(getAddedFileContents() === "one")
Files.write("two", file, StandardCharsets.UTF_8)
sc.addFile(file.getAbsolutePath)
assert(getAddedFileContents() === "onetwo")
} finally {
Utils.deleteRecursively(dir)
sc.stop()
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
