attilapiros commented on a change in pull request #28708:
URL: https://github.com/apache/spark/pull/28708#discussion_r435103701
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -650,6 +657,19 @@ private[spark] class BlockManager(
blockId: BlockId,
level: StorageLevel,
classTag: ClassTag[_]): StreamCallbackWithID = {
+ // Delegate shuffle blocks here to resolver if supported
+ if (blockId.isShuffle || blockId.isInternalShuffle) {
Review comment:
I think `isShuffle` name on its own is only for requesting shuffle files
and not for storing the shuffle blocks so they are important during fetching
but that's all.
But if they would be somehow useful then the future coder has to touch the
method `putShuffleBlockAsStream` too where we matched for the
`ShuffleIndexBlockId` and `ShuffleDataBlockId` and many more places what we
cannot foresee right now. My suggestion is let's do the minimal we need here
the one we have proof about it really works and needed (follow the YAGNI
principle).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
