Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21933#discussion_r209496412
--- Diff:
core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala ---
@@ -70,6 +70,8 @@ private[spark] class SerializerManager(
private[this] val compressRdds = conf.getBoolean("spark.rdd.compress",
false)
// Whether to compress shuffle output temporarily spilled to disk
private[this] val compressShuffleSpill =
conf.getBoolean("spark.shuffle.spill.compress", true)
+ // Size of the chunks to be used in the ChunkedByteBuffer
+ private[this] val chunkSizeMb =
conf.getSizeAsMb("spark.memory.serializer.chunkSize", "4m").toInt
--- End diff --
@vincent-grosbois WDTY about this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
