Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4881#discussion_r25805285
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -389,16 +389,30 @@ class UtilsSuite extends FunSuite with
ResetSystemProperties {
val tempDir = Utils.createTempDir()
val innerTempDir = Utils.createTempDir(tempDir.getPath)
val tempFile = File.createTempFile("someprefix", "somesuffix",
innerTempDir)
+ val tempFile1 = File.createTempFile("someprefix1", "somesuffix1",
innerTempDir)
+ val tempFile2 = File.createTempFile("someprefix2", "somesuffix2",
innerTempDir)
+ val tempFile3 = File.createTempFile("someprefix3", "somesuffix3",
tempDir)
val targetDir = new File("target-dir")
+ val renameTargetDir = new File("rename-target-dir")
+ val fileTargetDir = new File("file-target-dir")
+ val multiFileTargetDir = new File("multi-file-target-dir")
Files.write("some text", tempFile, UTF_8)
+ Files.write("some text", tempFile1, UTF_8)
+ Files.write("some text", tempFile2, UTF_8)
try {
val path = new Path("file://" + tempDir.getAbsolutePath)
val conf = new Configuration()
val fs = Utils.getHadoopFileSystem(path.toString, conf)
+ // Testing subdirs are copied across
Utils.fetchHcfsFile(path, targetDir, fs, new SparkConf(), conf,
false)
assert(targetDir.exists())
assert(targetDir.isDirectory())
+ // Testing to make sure it doesn't error if the dir already exists
+ Utils.fetchHcfsFile(path, targetDir, fs, new SparkConf(), conf,
false)
+ val newTempFile = new File(targetDir, tempFile3.getName)
--- End diff --
I think this is where our patches diverge a bit.
The behavior I expect when I upload a directory "foo", is that there will
be a directory called "foo" inside the target directory, so that when I do
`SparkFiles.get("foo")` I get the location of that directory.
Your patch seems to place the contents of "foo" under the target dir, which
is not what I'd expect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
