Weixin-Xu commented on code in PR #13492:
URL: https://github.com/apache/gluten/pull/13492#discussion_r4421506285
##########
backends-bolt/src-uniffle/main/java/org/apache/spark/shuffle/writer/BoltUniffleColumnarShuffleWriter.java:
##########
@@ -224,29 +222,36 @@ public long spill(MemoryTarget self, Spiller.Phase phase,
long size) {
return;
}
long startTime = System.nanoTime();
- GlutenSplitResult splitResult;
+ BoltSplitResult splitResult;
try {
splitResult = shuffleWriterJniWrapper.stop(nativeShuffleWriter);
} catch (IOException e) {
throw new RssException(e);
}
columnarDep.metrics().get("shuffleWallTime").get().add(System.nanoTime() -
startTime);
+ // AQE needs the uncompressed size, including batches flushed by stop().
+ columnarDep
+ .metrics()
+ .get("dataSize")
+ .get()
+ .add(Arrays.stream(splitResult.getRawPartitionLengths()).sum());
+ // Bolt's evict time includes compression; the common writer reported push
time separately.
+ long pushTime =
+ Math.max(0L, splitResult.getTotalPushTime() -
splitResult.getTotalCompressTime());
columnarDep
.metrics()
.get("splitTime")
Review Comment:
Moved splitTime into the common Bolt shuffle metrics so it is available for
both hash- and sort-classified dependencies. Added a Uniffle regression above
the sort-partition threshold that exercises writer shutdown and verifies the
commit path completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
