Github user trystanleftwich commented on a diff in the pull request:
https://github.com/apache/spark/pull/4881#discussion_r25806270
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -389,16 +389,30 @@ class UtilsSuite extends FunSuite with
ResetSystemProperties {
val tempDir = Utils.createTempDir()
val innerTempDir = Utils.createTempDir(tempDir.getPath)
val tempFile = File.createTempFile("someprefix", "somesuffix",
innerTempDir)
+ val tempFile1 = File.createTempFile("someprefix1", "somesuffix1",
innerTempDir)
+ val tempFile2 = File.createTempFile("someprefix2", "somesuffix2",
innerTempDir)
+ val tempFile3 = File.createTempFile("someprefix3", "somesuffix3",
tempDir)
val targetDir = new File("target-dir")
+ val renameTargetDir = new File("rename-target-dir")
+ val fileTargetDir = new File("file-target-dir")
+ val multiFileTargetDir = new File("multi-file-target-dir")
Files.write("some text", tempFile, UTF_8)
+ Files.write("some text", tempFile1, UTF_8)
+ Files.write("some text", tempFile2, UTF_8)
try {
val path = new Path("file://" + tempDir.getAbsolutePath)
val conf = new Configuration()
val fs = Utils.getHadoopFileSystem(path.toString, conf)
+ // Testing subdirs are copied across
Utils.fetchHcfsFile(path, targetDir, fs, new SparkConf(), conf,
false)
assert(targetDir.exists())
assert(targetDir.isDirectory())
+ // Testing to make sure it doesn't error if the dir already exists
+ Utils.fetchHcfsFile(path, targetDir, fs, new SparkConf(), conf,
false)
+ val newTempFile = new File(targetDir, tempFile3.getName)
--- End diff --
Ok, fair enough misunderstanding on my behalf, I've tested your patch and
it works in my case, which I expected. I'm happy to close this as likewise I
just want this solved for the next release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
