AngersZhuuuu commented on code in PR #2117:
URL:
https://github.com/apache/incubator-celeborn/pull/2117#discussion_r1407084339
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/celeborn/SortBasedPusher.java:
##########
@@ -213,6 +217,10 @@ public long pushData() throws IOException {
long freedBytes = freeMemory();
inMemSorter.freeMemory();
+ // Reset the in-memory sorter's pointer array only after freeing up the
memory pages holding
+ // the records. Otherwise, if the task is over allocated memory, then
without freeing the
+ // memory pages, we might not be able to get memory for the pointer
array.
+ taskContext.taskMetrics().incMemoryBytesSpilled(freedBytes);
Review Comment:
Yea, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
