cloud-fan commented on issue #26611: [SPARK-29939][CORE] Add a conf for CompressionCodec for Ser/Deser of MapOutputStatus URL: https://github.com/apache/spark/pull/26611#issuecomment-557362370 If it's justified for event log, then it's justified for map status as well. I think we should also give a different value for the event log codec. A possible use case: people develop a custom codec for RDD/shuffle and set it with `spark.io.compression.codec`. But this codec doesn't work well with event log/map status, so ideally codec of these two should not be affected by `spark.io.compression.codec`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
