This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 074894ce70e [MINOR][SHUFFLE] Include IOException in warning log of
finalizeShuffleMerge
074894ce70e is described below
commit 074894ce70e90717cbc81f7e6abc53d10872cda3
Author: Ted Yu <[email protected]>
AuthorDate: Sat Jan 21 15:56:40 2023 -0600
[MINOR][SHUFFLE] Include IOException in warning log of finalizeShuffleMerge
### What changes were proposed in this pull request?
This PR adds `ioe` to the warning log of `finalizeShuffleMerge`.
### Why are the changes needed?
With `ioe` logged, user would have more clue as to the root cause.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing test suite.
Closes #39654 from tedyu/shuffle-ioe.
Authored-by: Ted Yu <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
.../org/apache/spark/network/shuffle/RemoteBlockPushResolver.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
index fb3f8109a1a..a2e8219228a 100644
---
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
+++
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
@@ -814,8 +814,9 @@ public class RemoteBlockPushResolver implements
MergedShuffleFileManager {
}
} catch (IOException ioe) {
logger.warn("{} attempt {} shuffle {} shuffleMerge {}: exception
while " +
- "finalizing shuffle partition {}", msg.appId,
msg.appAttemptId, msg.shuffleId,
- msg.shuffleMergeId, partition.reduceId);
+ "finalizing shuffle partition {}. Exception message: {}",
msg.appId,
+ msg.appAttemptId, msg.shuffleId, msg.shuffleMergeId,
partition.reduceId,
+ ioe.getMessage());
} finally {
partition.closeAllFilesAndDeleteIfNeeded(false);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
