DB Tsai created SPARK-29576: ------------------------------- Summary: Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus Key: SPARK-29576 URL: https://issues.apache.org/jira/browse/SPARK-29576 Project: Spark Issue Type: New Feature Components: Spark Core Affects Versions: 2.4.4 Reporter: DB Tsai Fix For: 3.0.0
Instead of using ZStd codec directly, we use Spark's CompressionCodec which wraps ZStd codec in buffered stream to avoid overhead excessive of JNI call while trying to compress small amount of data. Also, by using Spark's CompressionCodec, we can easily to make it configurable in the future if needed. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org