sandip-db opened a new pull request, #59435:
URL: https://github.com/apache/spark/pull/59435
### What changes were proposed in this pull request?
This PR adds a read-only data source option, `decompressionCodec`, to the
CSV, JSON, XML and text file sources. When set, it forces the codec used to
decompress every input file instead of inferring the codec from each file name
extension:
```scala
spark.read.option("decompressionCodec", "gzip").csv("/path/to/data.csv")
```
- Accepts the short names supported by the `compression` write option
(`none`, `uncompressed`, `bzip2`, `deflate`, `gzip`, `lz4`, `snappy`), plus
`zstd`/`zstandard`, or a fully qualified Hadoop `CompressionCodec` class name.
Names are case-insensitive.
- `none`/`uncompressed` reads the files as-is, even when the file name
carries a compression extension.
How it works:
- `HadoopCodecStreams.getDecompressionCodec` is the single place every
text-based reader already uses to pick a codec. It now honors an internal
Hadoop conf key
(`org.apache.spark.sql.execution.datasources.decompressionCodec`) before
falling back to extension-based detection.
- `CompressionCodecs.setReadCodecConfiguration` puts the parsed option into
the per-scan Hadoop conf before it is broadcast. It is called by every V1
`FileFormat.buildReader`, every V2 `Scan.createReaderFactory`, and the
multiLine schema-inference paths. Line-based inference reuses `TextFileFormat`,
which forwards the options. When the option is absent, the key is explicitly
unset, so a value inherited from the session or Hadoop configuration is never
honored.
- Splittability follows the forced codec: a forced bzip2 file stays
splittable, a forced gzip file does not, and `none` keeps the file splittable.
V2 scans report the forced codec as the reason a file is not split.
- The option is parsed once in `FileSourceOptions`. The name lives in
`DataSourceOptions` and is registered by the four supporting formats.
Not supported. Both fail with a clear `UNSUPPORTED_FEATURE` error on the
driver while planning, so `ignoreCorruptFiles` cannot hide them:
- archive inputs (`.tar`, `.tar.gz`, `.tgz`, `.zip`, `.7z`), whose container
format is determined by the extension
(`UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE`);
- the legacy Hadoop `LineRecordReader`
(`spark.sql.execution.datasources.hadoopLineRecordReader.enabled=false`), which
cannot be told which codec to use
(`UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_WITH_LEGACY_LINE_READER`).
### Why are the changes needed?
Text-based file sources choose the decompression codec solely from the file
name extension. Files whose extension does not match their actual compression
cannot be read correctly:
- a gzip-compressed file named `data.csv` is read as garbage;
- a plain-text file named `data.csv.gz` fails to decompress.
Writers already have a `compression` option, but readers had no way to
override extension-based detection.
A separate option name is used on purpose instead of honoring `compression`
on read. `compression` is often persisted in table options, for example by
`saveAsTable` or `CREATE TABLE ... OPTIONS (compression 'gzip')`, or shared
between read and write option maps. Honoring it on read would change how
existing tables and pipelines read their files.
### Does this PR introduce _any_ user-facing change?
Yes. There is a new read option, `decompressionCodec`, for the CSV, JSON,
XML and text data sources, documented in
`docs/sql-data-sources-{csv,json,text,xml}.md`. Existing reads are unchanged;
in particular, `compression` is still ignored on read.
### How was this patch tested?
New tests in `CSVSuite`, `JsonSuite`, `XmlSuite` and `TextSuite`, which run
on DSv1 and DSv2 where applicable:
- every codec, including zstd, read from a file whose name has no codec
extension, with and without multiLine/wholetext and schema inference;
- `none`/`uncompressed` on a plain file with a `.gz` name;
- codec class names and case-insensitive short names (CSV);
- table options in `CREATE TABLE ... USING csv OPTIONS (...)` and Structured
Streaming reads (CSV);
- splitting with forced bzip2, gzip and `none`, and the interaction with
multiLine (CSV, text);
- the V2 unsplittable reason (`CSVv2Suite`, `TextV2Suite`);
- `compression` alone still does not force a codec on read, and an inherited
internal conf key is ignored;
- invalid codec names, and the legacy line reader error.
Shared tests in `ArchiveReadSuiteBase` and `TextArchiveReadBase` check that
`decompressionCodec` is rejected for every archive type and format, including
with `ignoreCorruptFiles=true` and for archives mixed with loose files.
Ran locally:
```
build/sbt 'core/testOnly *SparkThrowableSuite'
build/sbt 'sql/testOnly *TextV1Suite *TextV2Suite *CSVv1Suite *CSVv2Suite
*JsonV1Suite *JsonV2Suite *XmlSuite *ArchiveReadSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
