vanzin commented on a change in pull request #23688: [SPARK-25035][Core]
Avoiding memory mapping at disk-stored blocks replication
URL: https://github.com/apache/spark/pull/23688#discussion_r254905627
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -221,6 +221,178 @@ private[spark] class BlockManager(
new BlockManager.RemoteBlockDownloadFileManager(this)
private val maxRemoteBlockToMem =
conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)
+ /**
+ * @param blockSize the decrypted size of the block
+ */
+ private abstract class BlockStoreUpdater[T](
+ blockSize: Long,
+ blockId: BlockId,
+ level: StorageLevel,
+ classTag: ClassTag[T],
+ tellMaster: Boolean,
+ keepReadLock: Boolean) {
+
+ protected def byteBuffer: ChunkedByteBuffer
+
+ protected def saveToDiskStore(): Unit
+
+ private def saveToMemoryStore(): Boolean = {
+ if (level.deserialized) {
+ val values =
+ serializerManager.dataDeserializeStream(blockId,
byteBuffer.toInputStream())(classTag)
Review comment:
If the goal here is to save memory, you could also do something here.
e.g., in the temp file case, instead of reading the data into a
`ChunkedByteBuffer` first, your could return an input stream directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
