steveloughran commented on code in PR #6468:
URL: https://github.com/apache/hadoop/pull/6468#discussion_r1484756242
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java:
##########
@@ -264,9 +326,14 @@ public void abortTask(TaskAttemptContext context) throws
IOException {
try (DurationInfo d = new DurationInfo(LOG,
"Abort task %s", context.getTaskAttemptID());
CommitContext commitContext = initiateTaskOperation(context)) {
- getCommitOperations().abortAllSinglePendingCommits(attemptPath,
- commitContext,
- true);
+ if (isTrackMagicCommitsInMemoryEnabled(context.getConfiguration())) {
+ List<SinglePendingCommit> pendingCommits =
loadPendingCommitsFromMemory(context);
+ for (SinglePendingCommit singleCommit : pendingCommits) {
+ commitContext.abortSingleCommit(singleCommit);
+ }
+ } else {
+ getCommitOperations().abortAllSinglePendingCommits(attemptPath,
commitContext, true);
Review Comment:
looked at the hadoop stuff and yes, it does it in the same process too
```
this is called from a task's process to clean
up a single task's output that can not yet been committed. This may be
called multiple times for the same task, but for different task attempts.
```
so nothing to worry about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
