symious opened a new pull request #35569: URL: https://github.com/apache/spark/pull/35569
### What changes were proposed in this pull request? In `commitJob` and `abortJob` of HadoopMapReduceCommitProtocol, the stagingDir was deleted, but for cases like the following example, the stagingDir was not created in the first place. If the underLayer FileSystem is HDFS, there won't be any logs, but deleting a nonexistent file with FileSystem.delete would also acquire the WRITE LOCK of NameNode, if such operations are frequent, it would hurt the performance of NN. For other FileSystem, like Alluxio, a warning log was thrown as follows. AbstractFileSystem: delete failed: alluxio.exception.FileDoesNotExistException: Path "/test/spark/t10/.spark-staging-3cfe0d7c-3749-44de-9da7-f811a40aa4af" does not exist. This ticket is to add an existence check before the actual deleting operation, which could help eliminate the WARN log or not hurt HDFS' performance. ### Why are the changes needed? The original case might hurt HDFS' performance and have WARNING logs for Alluxio. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? unit test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
