xumingming commented on code in PR #3670:
URL: https://github.com/apache/celeborn/pull/3670#discussion_r3233086066
##########
client/src/main/java/org/apache/celeborn/client/ShuffleClientImpl.java:
##########
@@ -1404,6 +1398,23 @@ public int mergeData(
false);
}
+ @Override
+ public void computeBatchCRC(
+ int shuffleId,
+ int mapId,
+ int attemptId,
+ int partitionId,
+ byte[] data,
+ int offset,
+ int length) {
+ if (!shuffleIntegrityCheckEnabled) {
+ return;
+ }
+ final String mapKey = Utils.makeMapKey(shuffleId, mapId, attemptId);
+ PushState pushState = getPushState(mapKey);
Review Comment:
What we are doing here is computing CRC per batch, the CRC computation on
the data payload (CRC32.update over potentially megabytes) dominates the
per-batch cost, making the string allocation noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
