attilapiros commented on a change in pull request #22024: [SPARK-25034][CORE]
Remove allocations in onBlockFetchSuccess
URL: https://github.com/apache/spark/pull/22024#discussion_r257702269
##########
File path:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
##########
@@ -160,7 +160,13 @@ private[spark] class TorrentBroadcast[T: ClassTag](obj:
T, id: Long)
releaseLock(pieceId)
case None =>
bm.getRemoteBytes(pieceId) match {
- case Some(b) =>
+ case Some(splitB) =>
+
+ // Checksum computation and further computations require the data
+ // from the ChunkedByteBuffer to be merged, so we we merge it
now.
+ val bMergedData = splitB.toArray
Review comment:
As I see when checksum is not enabled then merge is not needed.
And you can go further you can extend the checksum calculation to work on an
array of chunks as I think it is just calling Alder32's `update` in a iteration
for all the chunks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
