viirya commented on a change in pull request #23693: [SPARK-26768][CORE]Remove
useless code in BlockManager
URL: https://github.com/apache/spark/pull/23693#discussion_r253257192
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -632,18 +632,8 @@ private[spark] class BlockManager(
*/
def getLocalBytes(blockId: BlockId): Option[BlockData] = {
logDebug(s"Getting local block $blockId as bytes")
- // As an optimization for map output fetches, if the block is for a
shuffle, return it
- // without acquiring a lock; the disk store never deletes (recent) items
so this should work
- if (blockId.isShuffle) {
- val shuffleBlockResolver = shuffleManager.shuffleBlockResolver
- // TODO: This should gracefully handle case where local block is not
available. Currently
- // downstream code will throw an exception.
- val buf = new ChunkedByteBuffer(
-
shuffleBlockResolver.getBlockData(blockId.asInstanceOf[ShuffleBlockId]).nioByteBuffer())
- Some(new ByteBufferBlockData(buf, true))
- } else {
- blockInfoManager.lockForReading(blockId).map { info =>
doGetLocalBytes(blockId, info) }
- }
+ assert(!blockId.isShuffle, "Unexpected ShuffleBlockId -- cf. SPARK-26768")
Review comment:
I think instead of mentioning this JIRA number, maybe it is more helpful to
suggest to use `getBlockData` for shuffle block.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
