akshatshenoi-db opened a new pull request, #57411:
URL: https://github.com/apache/spark/pull/57411
### What changes were proposed in this pull request?
This adds **7z** (`.7z`) as a third archive container to the archive-reader
feature (gated by
`spark.sql.files.archive.reader.enabled`, default off), alongside tar and
zip. It continues the
series: SPARK-57135/57321 CSV, SPARK-57419 JSON, SPARK-57478 text,
SPARK-57479 XML, SPARK-57481
Avro, SPARK-57705 zip container, SPARK-58110 the `SupportsArchiveFormat`
trait, SPARK-57590 Parquet.
The container engine is format-agnostic: data sources dispatch through the
trait's `isArchivePath`
+ `readArchiveEntries`, so CSV, JSON, text, XML, and Avro get 7z with no
per-data-source change.
Unlike tar and zip, 7z keeps its entry index at the *end* of the file and
can compress entries
together, so it cannot be read from a forward-only stream. The port:
- adds a read-only `HadoopSeekableByteChannel` over `FSDataInputStream` so
`SevenZFile` can seek to
the footer index in place (the archive is never copied to local disk);
- wraps `SevenZFile` in a `SevenZArchiveInputStream` adapter that fits the
existing
`ArchiveInputStream` cursor the engine already consumes, so
`openArchiveStream` gains a single
`.7z` case and the streaming engine itself is unchanged;
- adds `.7z` to `isArchivePath` and the unsupported-archive error message.
commons-compress reads 7z but treats its xz (LZMA/LZMA2) codec as an
optional dependency, so
`org.tukaani:xz` (already version-managed in the parent POM) is declared in
`sql/core/pom.xml`.
It also consolidates the per-container leaf test suites into each format's
`*ArchiveReadBase` (and
merges the CSV header/headerless bases), so adding a container no longer
multiplies suite files;
each format gains a `<Format>SevenZArchiveReadSuite` beside its existing
tar/zip suites.
### Why are the changes needed?
7z is a common archive format for shipping many small files compressed
together. Extending the
archive reader to 7z lets users read tarred/zipped/7z'd data without
unpacking first, and exercises
the seekable-container path (the previous containers were all forward-only
streams).
### Does this PR introduce _any_ user-facing change?
Yes, gated by `spark.sql.files.archive.reader.enabled` (default `false`).
When enabled, reading a
path ending in `.7z` returns the rows of its entries, the same as an
equivalent directory; when
disabled (the default), behavior is unchanged.
### How was this patch tested?
Added a `<Format>SevenZArchiveReadSuite` for CSV (header and headerless),
JSON, text, XML, and Avro
(each binding the format to the new
`SevenZArchiveReadBase`/`SevenZArchiveTestUtils`), so 7z runs the
full format-agnostic and format-specific archive test matrix. Added 7z cases
to
`SupportsArchiveFormatSuite` covering `.7z` dispatch, entry ordering, and
directory/dotfile/marker
skipping through the seek-based reader.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
