advancedxy commented on a change in pull request #26085: [SPARK-29434] [Core]
Improve the MapStatuses Serialization Performance
URL: https://github.com/apache/spark/pull/26085#discussion_r333810252
##########
File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala
##########
@@ -807,13 +808,17 @@ private[spark] object MapOutputTracker extends Logging {
private val BROADCAST = 1
// Serialize an array of map output locations into an efficient byte format
so that we can send
- // it to reduce tasks. We do this by compressing the serialized bytes using
GZIP. They will
+ // it to reduce tasks. We do this by compressing the serialized bytes using
Zstd. They will
// generally be pretty compressible because many map outputs will be on the
same hostname.
def serializeMapStatuses(statuses: Array[MapStatus], broadcastManager:
BroadcastManager,
isLocal: Boolean, minBroadcastSize: Int): (Array[Byte],
Broadcast[Array[Byte]]) = {
- val out = new ByteArrayOutputStream
- out.write(DIRECT)
- val objOut = new ObjectOutputStream(new GZIPOutputStream(out))
+ import scala.language.reflectiveCalls
+ val out = new ByteArrayOutputStream(4096) {
Review comment:
Just curious, is there any specific reason to choose 4096 here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
