Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/17748#discussion_r113192444
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -363,25 +362,29 @@ protected File initRecoveryDb(String dbFileName) {
// make sure to move all DBs to the recovery path from the old
NM local dirs.
// If another DB was initialized first just make sure all the
DBs are in the same
// location.
- File newLoc = new File(_recoveryPath.toUri().getPath(),
dbFileName);
- if (!newLoc.equals(f)) {
+ Path newLoc = new Path(_recoveryPath, dbName);
+ Path copyFrom = new Path(f.toURI());
+ if (!newLoc.equals(copyFrom)) {
+ logger.info("Moving " + copyFrom + " to: " + newLoc);
try {
- Files.move(f.toPath(), newLoc.toPath());
+ // The move here needs to handle moving non-empty
directories across NFS mounts
+ FileSystem fs = FileSystem.getLocal(_conf);
+ fs.rename(copyFrom, newLoc);
--- End diff --
didn't measure it but can't imagine there is much difference. I didn't see
any noticable delays in NM startup during testing. It is a one time operation
on startup to move the DB if changing to use the recovery dir. If you have a
specific concern here I can go do some measurements?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
