[
https://issues.apache.org/jira/browse/SPARK-47829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18022182#comment-18022182
]
Romain Ardiet commented on SPARK-47829:
---------------------------------------
Is there an update on this?
> Text Datasource supports zstd compression codec
> -----------------------------------------------
>
> Key: SPARK-47829
> URL: https://issues.apache.org/jira/browse/SPARK-47829
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: xorsum
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> {code:java}
> drop table if exists tmp_compressed_datasource;CREATE TABLE
> tmp_compressed_datasource (key STRING) USING TEXT OPTIONS (compression
> 'zstd');INSERT INTO TABLE tmp_compressed_datasource values
> ('a'),('b'),('c');select * from tmp_compressed_datasource ;
> {code}
> {code:java}
> java.lang.IllegalArgumentException: Codec [zstd] is not available. Known
> codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.
> at
> org.apache.spark.sql.catalyst.util.CompressionCodecs$.getCodecClassName(CompressionCodecs.scala:53)
> at
> org.apache.spark.sql.execution.datasources.text.TextOptions.$anonfun$compressionCodec$1(TextOptions.scala:38)
> at scala.Option.map(Option.scala:230)
> at
> org.apache.spark.sql.execution.datasources.text.TextOptions.<init>(TextOptions.scala:38)
> at
> org.apache.spark.sql.execution.datasources.text.TextOptions.<init>(TextOptions.scala:33)
> at
> org.apache.spark.sql.execution.datasources.text.TextFileFormat.prepareWrite(TextFileFormat.scala:72)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:123)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:190)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
