dongjoon-hyun commented on pull request #32983: URL: https://github.com/apache/spark/pull/32983#issuecomment-864494488
Although HADOOP-16878 is reopened, it's only reverted from `branch-3.3.1`. `trunk` still has it. - https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java#L402-L405 Here is my understanding. From Apache Spark side, - We are hitting Jenkins failure and HADOOP-16878 looks like a breaking behavior change which is inappropriate in the maintenance release (3.3.0 -> 3.3.1). - Spark has three cases. - `!compareFs(srcFs, destFs)`: This is safe because we will not have this exception. - `"file".equals(srcFs.getScheme)`: This is safe because this cannot be a `false` alarm. - `force=true`: - For the `good` alarm part, Spark works in the same way. - For the `false` alarm part, Spark is safe because we use `force = true` only for copying `localConfArchive` instead of a general copy. ```scala val localConfArchive = new Path(createConfArchive(confsToOverride).toURI()) copyFileToRemote(destDir, localConfArchive, replication, symlinkCache, force = true, destName = Some(LOCALIZED_CONF_ARCHIVE)) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
