Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/17748#discussion_r113325881
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -363,25 +362,29 @@ protected File initRecoveryDb(String dbFileName) {
// make sure to move all DBs to the recovery path from the old
NM local dirs.
// If another DB was initialized first just make sure all the
DBs are in the same
// location.
- File newLoc = new File(_recoveryPath.toUri().getPath(),
dbFileName);
- if (!newLoc.equals(f)) {
+ Path newLoc = new Path(_recoveryPath, dbName);
+ Path copyFrom = new Path(f.toURI());
+ if (!newLoc.equals(copyFrom)) {
+ logger.info("Moving " + copyFrom + " to: " + newLoc);
try {
- Files.move(f.toPath(), newLoc.toPath());
+ // The move here needs to handle moving non-empty
directories across NFS mounts
+ FileSystem fs = FileSystem.getLocal(_conf);
+ fs.rename(copyFrom, newLoc);
--- End diff --
The implementations looked different when I looked at LocalFileSystem - and
was wondering if there was a perf implementation to the change.
If it is a one time thing used infrequently, then it might not be a concern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
