Sandip Agarwala created SPARK-60217:
---------------------------------------
Summary: Add a decompressionCodec read option to text-based file
sources to override extension-based codec detection
Key: SPARK-60217
URL: https://issues.apache.org/jira/browse/SPARK-60217
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Sandip Agarwala
Text-based file sources (CSV, JSON, XML, text) choose the decompression codec
for each input file solely from its file name extension (e.g. `.gz` -> gzip).
Files whose extension does not match their actual compression cannot be read
correctly: a gzip-compressed file named `data.csv` is read as garbage, and a
plain-text file named `data.csv.gz` fails to decompress. Writers already
support a `compression` option, but readers have no way to override the
extension-based detection.
This ticket proposes a new read-only data source option, `decompressionCodec`,
for CSV, JSON, XML and text:
{code}
spark.read.option("decompressionCodec", "gzip").csv("/path/to/data.csv")
{code}
* Accepts the short names supported by the `compression` write option (`none`,
`uncompressed`, `bzip2`, `deflate`, `gzip`, `lz4`, `snappy`), plus
`zstd`/`zstandard`, or a fully qualified Hadoop `CompressionCodec` class name.
* When set, the codec is applied to every input file instead of inferring it
from the extension; `none`/`uncompressed` reads files as-is even if they carry
a compression extension.
* Applies uniformly to line-based, multiLine, wholetext and schema-inference
reads, for both DSv1 and DSv2.
* Splittability follows the forced codec (e.g. a forced bzip2 file remains
splittable, a forced gzip file does not).
* The existing write-side `compression` option keeps being ignored on read. A
separate option name is used deliberately so that tables and pipelines that
persist or share `compression` (e.g. `saveAsTable` / `CREATE TABLE ... OPTIONS
(compression ...)`) do not change behavior.
* Not supported (fails with a clear error) for archive inputs (`.tar`,
`.tar.gz`, `.tgz`, `.zip`, `.7z`), whose container format is determined by the
extension, and when the legacy Hadoop `LineRecordReader` is enabled
(`spark.sql.execution.datasources.hadoopLineRecordReader.enabled=false`), which
cannot honor the option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
