otterc commented on code in PR #35906:
URL: https://github.com/apache/spark/pull/35906#discussion_r903112657
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -583,6 +686,7 @@ public MergeStatuses
finalizeShuffleMerge(FinalizeShuffleMerge msg) {
bitmaps.toArray(new RoaringBitmap[bitmaps.size()]),
Ints.toArray(reduceIds),
Longs.toArray(sizes));
}
+ writeAppAttemptShuffleMergeInfoToDB(appAttemptShuffleMergeId);
Review Comment:
I think based on an earlier comment, this was moved here. However, I think
we should write to db when we update the state that the partition is finalized.
When a particular reduce partition fails to finalize, we don't add it to the
result. However, we still mark the shuffle to be finalized. In that case we
should update the database there as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
