akshatshenoi-db opened a new pull request, #57814:
URL: https://github.com/apache/spark/pull/57814
### What changes were proposed in this pull request?
When an archive entry is itself an archive
(`.tar`/`.tar.gz`/`.tgz`/`.zip`/`.7z`), recurse
into it and stream its entries, instead of handing the nested archive's raw
bytes to the
format parser. Nested entries report the full `outer!/inner!/leaf` logical
name.
- `readArchiveEntries` delegates to a depth-aware `streamEntries` that
recurses when an
entry is an archive.
- A new `openNestedArchiveStream` opens a container from an already-open
entry stream. Tar
reads the stream directly; zip and 7z read their index by seeking, which
an entry stream
cannot do, so the entry is spilled to a local file and opened from there.
- New config `spark.sql.files.archive.reader.maxNestingDepth` (default 10)
bounds the
recursion; exceeding it fails the read with `MAX_ARCHIVE_DEPTH_EXCEEDED`,
guarding
against zip bombs and cyclic archives.
- `isArchiveFileName` is factored out of `isArchivePath` so an entry name is
matched as a
plain string: an entry name is arbitrary, and `Path` would misparse one
containing a
colon as a URI scheme.
### Why are the changes needed?
Archives commonly contain other archives, and today those inner archives are
handed to the
format parser as opaque bytes, so their contents are unreadable. Recursing
makes an archive
of archives behave like a directory tree of the data files it ultimately
holds.
### Does this PR introduce any user-facing change?
Yes, gated by `spark.sql.files.archive.reader.enabled` (default false). With
archive reading
enabled, a nested archive's entries are now read instead of its raw bytes,
and the new
`spark.sql.files.archive.reader.maxNestingDepth` config (default 10) bounds
recursion depth.
### How was this patch tested?
`ArchiveReadSuiteBase` gains nested-archive tests that run for every
container (recursion
reads like a directory, a nested archive alongside plain entries, three
levels of nesting,
empty nested, hidden entries in nested, the depth limit, and corrupt-nested
under
`ignoreCorruptFiles`), plus a `writeNestedArchive` helper.
`SupportsArchiveFormatSuite`
covers what only the engine sees: one archive nesting tar, zip, and 7z
inside each other, a
nested archive whose name contains a colon, and the depth-limit boundary.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
