Github user vincent-grosbois commented on a diff in the pull request:
https://github.com/apache/spark/pull/21933#discussion_r207557109
--- Diff:
core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala ---
@@ -70,6 +70,8 @@ private[spark] class SerializerManager(
private[this] val compressRdds = conf.getBoolean("spark.rdd.compress",
false)
// Whether to compress shuffle output temporarily spilled to disk
private[this] val compressShuffleSpill =
conf.getBoolean("spark.shuffle.spill.compress", true)
+ // Size of the chunks to be used in the ChunkedByteBuffer
+ private[this] val chunkSizeMb =
conf.getSizeAsMb("spark.memory.chunkSize", "4m").toInt
--- End diff --
I renamed it to spark.memory.serializer.chunkSize
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
