akshatshenoi-db opened a new pull request, #56784:
URL: https://github.com/apache/spark/pull/56784
### What changes were proposed in this pull request?
Add zip (`.zip`) support to the streaming archive reader (`ArchiveReader`),
extending the existing tar support (`.tar`/`.tar.gz`/`.tgz`) and continuing the
archive-reader series: SPARK-57135 (CSV read), SPARK-57321 (CSV inference),
SPARK-57419 (JSON), SPARK-57478 (text), SPARK-57479 (XML), SPARK-57481 (Avro).
The archive read/inference integration is format-agnostic -- every data
source dispatches through `ArchiveReader.isArchivePath` and
`ArchiveReader(path).readEntries(...)` -- so zip works for every data source
already wired up (CSV, JSON, text, XML, Avro) with no per-data-source changes.
The change is concentrated in `ArchiveReader`. The entry-streaming engine
(lazy one-entry-at-a-time advance, directory/dotfile/marker skipping with the
same `ignoredPathSegmentRegex` filter as a loose-file listing, close-shielded
entry streams, eager-first-entry error surfacing, task-completion cleanup) is
hoisted from `TarArchiveReader` into the abstract base. Since
`TarArchiveInputStream` and `ZipArchiveInputStream` both extend
commons-compress `ArchiveInputStream`, the base steps entries via
`getNextEntry` directly and a subclass implements only
`openArchiveStream(conf): ArchiveInputStream[_ <: ArchiveEntry]`:
```
ArchiveReader (abstract) -- shared readEntries engine + entry
filtering
+- TarArchiveReader -- opens a TarArchiveInputStream
(explicit .tgz gunzip)
+- ZipArchiveReader (new) -- opens a ZipArchiveInputStream
```
- `TarArchiveReader` is reduced to opening a `TarArchiveInputStream`
(keeping the `.tgz` explicit gunzip, with a defensive close so a gzip-header
failure can't leak the base stream); behavior is unchanged.
- New `ZipArchiveReader` opens a `ZipArchiveInputStream` (zip entries are
individually deflated, so no Hadoop codec layer is applied). It streams local
file headers sequentially, matching the tar reader's pure-streaming model; a
few unusual zips (e.g. a deflated entry whose size is recorded only in a
trailing data descriptor) are not streamable this way.
- `isArchivePath` / `apply` now recognize and dispatch `.zip` (`apply`
matches the tar and zip extensions explicitly and throws on anything else).
- The `spark.sql.files.archive.reader.enabled` doc is simplified to describe
the user-facing behavior without enumerating formats or implementation details.
No new flag -- zip rides the existing (default-off) gate.
### Why are the changes needed?
The archive reader already supports CSV, JSON, text, XML, and Avro over tar.
Zip is one of the most common archive containers for shipped data, and
extending the same opt-in archive path to zip lets users read and infer schema
from files packed in a `.zip` without unpacking them first, with the same
directory-read parity the rest of the series guarantees -- and, because the
integration is format-agnostic, across every data source already wired up.
### Does this PR introduce _any_ user-facing change?
Yes. When `spark.sql.files.archive.reader.enabled` is set (default `false`),
the CSV, JSON, text, XML, and Avro data sources can now read `.zip` archives in
addition to tar archives -- each archive is read as a single split and its
entries are streamed through the data source's parser (never unpacked to disk),
as if the entries were separate files, during both scan and schema inference.
Previously only `.tar`/`.tar.gz`/`.tgz` were recognized; `.zip` files were
treated as ordinary (non-archive) files. With the flag at its default, behavior
is unchanged.
### How was this patch tested?
- `ArchiveReaderSuite` -- added `.zip` cases for `isArchivePath` dispatch
and `readEntries` (empty/single/multiple entries, directory and dotfile/marker
skipping, lazy advance, non-closing entry stream), alongside the existing tar
cases that guard the refactor.
- New zip format suites mounted on the existing shared scaffolding:
`CSVHeaderZipArchiveReadSuite`, `CSVHeaderlessZipArchiveReadSuite`,
`JSONZipArchiveReadSuite`, `XMLZipArchiveReadSuite`, `TextZipArchiveReadSuite`,
and `AvroZipArchiveReadSuite`. The text read tests were extracted into a
container-agnostic `TextArchiveReadBase` (with the tar suite slimmed to match
the CSV/JSON/XML base+container pattern), and
`ZipArchiveReadBase`/`ZipArchiveTestUtils` are the zip peers of
`TarArchiveReadBase`/`TarArchiveTestUtils`.
- Re-ran all tar archive suites (including Avro) to confirm the
shared-engine refactor is behavior-preserving.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
