[
https://issues.apache.org/jira/browse/SPARK-60217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandip Agarwala updated SPARK-60217:
------------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Add a decompressionCodec read option to text-based file sources to override
> extension-based codec detection
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60217
> URL: https://issues.apache.org/jira/browse/SPARK-60217
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Sandip Agarwala
> Priority: Major
>
> Text-based file sources (CSV, JSON, XML, text) choose the decompression codec
> for each input file solely from its file name extension (e.g. `.gz` -> gzip).
> Files whose extension does not match their actual compression cannot be read
> correctly: a gzip-compressed file named `data.csv` is read as garbage, and a
> plain-text file named `data.csv.gz` fails to decompress. Writers already
> support a `compression` option, but readers have no way to override the
> extension-based detection.
> This ticket proposes a new read-only data source option,
> `decompressionCodec`, for CSV, JSON, XML and text:
> {code}
> spark.read.option("decompressionCodec", "gzip").csv("/path/to/data.csv")
> {code}
> * Accepts the short names supported by the `compression` write option
> (`none`, `uncompressed`, `bzip2`, `deflate`, `gzip`, `lz4`, `snappy`), plus
> `zstd`/`zstandard`, or a fully qualified Hadoop `CompressionCodec` class name.
> * When set, the codec is applied to every input file instead of inferring it
> from the extension; `none`/`uncompressed` reads files as-is even if they
> carry a compression extension.
> * Applies uniformly to line-based, multiLine, wholetext and schema-inference
> reads, for both DSv1 and DSv2.
> * Splittability follows the forced codec (e.g. a forced bzip2 file remains
> splittable, a forced gzip file does not).
> * The existing write-side `compression` option keeps being ignored on read. A
> separate option name is used deliberately so that tables and pipelines that
> persist or share `compression` (e.g. `saveAsTable` / `CREATE TABLE ...
> OPTIONS (compression ...)`) do not change behavior.
> * Not supported (fails with a clear error) for archive inputs (`.tar`,
> `.tar.gz`, `.tgz`, `.zip`, `.7z`), whose container format is determined by
> the extension, and when the legacy Hadoop `LineRecordReader` is enabled
> (`spark.sql.execution.datasources.hadoopLineRecordReader.enabled=false`),
> which cannot honor the option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
