[
https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811935#comment-15811935
]
Thomas Graves commented on SPARK-19112:
---------------------------------------
For any sort of Hadoop output formats we get it for free but for the internal
and shuffle outputs I think we have to add support to
spark.io.compression.codec for this.
I'm also wondering if there are other output formats like parquet that we would
want to add this option to (spark.sql.parquet.compression.codec), but I haven't
investigated those enough.
> add codec for ZStandard
> -----------------------
>
> Key: SPARK-19112
> URL: https://issues.apache.org/jira/browse/SPARK-19112
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Thomas Graves
> Priority: Minor
>
> ZStandard: https://github.com/facebook/zstd and
> http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was
> recently released. Hadoop
> (https://issues.apache.org/jira/browse/HADOOP-13578) and others
> (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it.
> Zstd seems to give great results => Gzip level Compression with Lz4 level CPU.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
