akshatshenoi-db opened a new pull request, #57238:
URL: https://github.com/apache/spark/pull/57238
### What changes were proposed in this pull request?
Reading tar/zip archives of data files (gated by
`spark.sql.files.archive.reader.enabled`, default off) is currently wired
through a standalone `ArchiveReader` abstract class with
`TarArchiveReader`/`ZipArchiveReader` subclasses; a data source obtains archive
support by calling into that free-standing class from its read and inference
paths.
This refactor replaces that class hierarchy with a `SupportsArchiveFormat`
trait that a file-based data source mixes in to gain archive support, treating
an archive like a directory of the entries it contains. The trait owns the
shared machinery:
- the streaming surface -- `isArchivePath`, `readArchiveEntries`,
`lineIterator` -- for formats whose per-file reader consumes an `InputStream`
(CSV, JSON, XML, text, Avro), and
- the read-localization surface -- `readLocalizedEntries`,
`copyEntryToLocalFile` -- for random-access formats that need a complete file
on disk (Parquet/ORC footers).
Container selection (tar/tgz/zip) moves into a single `openArchiveStream`
extension match, so the `TarArchiveReader`/`ZipArchiveReader` subclasses are
removed. Schema inference is deliberately not on the trait -- each format owns
its own -- so there is no shared `inferArchiveSchema`. The companion object
keeps two statics for callers that have no trait instance: `isArchivePath` and
a `readArchiveEntries` forwarder (for test suites and executor-side inference
whose `mapPartitions` closures cannot capture an instance).
Class hierarchy before:
```
abstract class ArchiveReader
<- TarArchiveReader
<- ZipArchiveReader
```
after:
```
trait SupportsArchiveFormat // mixed into
CSVDataSource/JsonDataSource/XmlDataSource/TextFileFormat/AvroFileFormat
object SupportsArchiveFormat // isArchivePath, readArchiveEntries
(statics for instance-less callers)
```
CSV/JSON/XML/text/Avro now `extends ... with SupportsArchiveFormat` and call
the inherited `readArchiveEntries`/`lineIterator` -- a mechanical swap.
### Why are the changes needed?
The standalone-class shape made adding archive support to a new format a
matter of threading calls to a free-standing class through the format's read
and inference paths. Modeling "this data source can read archives" as a trait
the format mixes in is a better fit: the shared streaming/localization
machinery lives in one place, a format opts in by mixing in the trait and
calling inherited methods, and container selection is centralized in one
extension match instead of a subclass per container. This is a structural
cleanup ahead of adding more archive-capable formats.
### Does this PR introduce _any_ user-facing change?
No. This is a behavior-preserving refactor; the archive read/infer semantics
are unchanged.
### How was this patch tested?
No behavior change, so this relies on the existing archive test suites: the
`ArchiveReaderSuite` engine tests (`isArchivePath` dispatch and
`readArchiveEntries` -- entry ordering, gzip handling, dir/dotfile skipping,
lazy advance, the non-closing entry stream, cleanup, and the non-streamable-zip
case) and the per-format tar/zip read suites. The call-site references were
updated from `ArchiveReader` to `SupportsArchiveFormat` and the suites' test
names/docs updated to match.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
