[
https://issues.apache.org/jira/browse/SPARK-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169160#comment-15169160
]
Hyukjin Kwon edited comment on SPARK-11691 at 2/26/16 3:10 PM:
---------------------------------------------------------------
This issue deals with a bit more generalized compression options compareing to
the issue I gave here. However, for general Hadoop compression configurations,
they can only be applied to JSON, CSV and TEXT datasources which I already
submitted some PRs and they were merged. So, although the issues themselves are
slightly different, I think the PRs I submitted cover this.
Also, I think we can't just assume all the {{HadoopFsRelation}}s support
compression. For ORC and Parquet, they might have to be dealt with differently
due to dofferent configuration keys and supporting codecs.
Should we then move all the issues about compression codecs for each data
source to this issue as sub-tasks?
was (Author: hyukjin.kwon):
This issue deals with a bit more generalized compression options compareing to
the issue I gave here. However, for general Hadoop compression configurations,
they can only be applied to JSON, CSV and TEXT datasources which I already
submitted some PRs and they were merged. So, although the issues themselves are
slightly different, I think the PRs I submitted cover this.
Also, I think we can't just assume all the {{HadoopFsRelation}}s support
compression. For ORC and Parquet, they might have to be dealt with differently
due to dofferent configuration keys and supporting codecs.
> Allow to specify compression codec in HadoopFsRelation when saving
> -------------------------------------------------------------------
>
> Key: SPARK-11691
> URL: https://issues.apache.org/jira/browse/SPARK-11691
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Jeff Zhang
>
> Currently, there's no way to specify compression codec when saving data frame
> to hdfs. It would nice to allow specify compression codec in DataFrameWriter
> just as we did in RDD api
> {code}
> def saveAsTextFile(path: String, codec: Class[_ <: CompressionCodec]): Unit =
> withScope {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
