[
https://issues.apache.org/jira/browse/SPARK-58110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Shenoi updated SPARK-58110:
----------------------------------
Description:
Reading tar/zip archives (gated by spark.sql.files.archive.reader.enabled,
default off) is currently wired through a standalone ArchiveReader abstract
class with TarArchiveReader/ZipArchiveReader subclasses; a data source obtains
archive support by calling into that free-standing class from its read and
inference paths.
This refactor replaces that class hierarchy with a SupportsArchiveFormat trait
that a file-based data source mixes in to gain archive support, treating an
archive like a directory of its entries. The trait owns the shared machinery:
- the streaming surface, isArchivePath, readArchiveEntries, lineIterator, for
formats whose per-file reader consumes an InputStream (CSV, JSON, XML, text,
Avro), and
- the read-localization surface, readLocalizedEntries, copyEntryToLocalFile,
for random-access formats that need a complete file on disk (Parquet/ORC
footers).
Container selection (tar/tgz/zip) moves into a single openArchiveStream
extension match, so the TarArchiveReader/ZipArchiveReader subclasses are
removed. The companion object keeps two statics for callers that have no trait
instance: isArchivePath (the prefetching readers) and a readArchiveEntries
forwarder (test suites and executor-side inference whose mapPartitions closures
cannot capture an instance).
This is a mechanical, behavior-preserving change: the existing archive
read/infer semantics and the per-format suites are unchanged. It makes adding
archive support to a new format a matter of mixing in the trait rather than
threading calls through a shared class.
> [SQL] Extract a SupportsArchiveFormat trait for archive-read data sources
> -------------------------------------------------------------------------
>
> Key: SPARK-58110
> URL: https://issues.apache.org/jira/browse/SPARK-58110
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Priority: Major
>
> Reading tar/zip archives (gated by spark.sql.files.archive.reader.enabled,
> default off) is currently wired through a standalone ArchiveReader abstract
> class with TarArchiveReader/ZipArchiveReader subclasses; a data source
> obtains archive support by calling into that free-standing class from its
> read and inference paths.
> This refactor replaces that class hierarchy with a SupportsArchiveFormat
> trait that a file-based data source mixes in to gain archive support,
> treating an archive like a directory of its entries. The trait owns the
> shared machinery:
> - the streaming surface, isArchivePath, readArchiveEntries, lineIterator,
> for formats whose per-file reader consumes an InputStream (CSV, JSON, XML,
> text, Avro), and
> - the read-localization surface, readLocalizedEntries, copyEntryToLocalFile,
> for random-access formats that need a complete file on disk (Parquet/ORC
> footers).
> Container selection (tar/tgz/zip) moves into a single openArchiveStream
> extension match, so the TarArchiveReader/ZipArchiveReader subclasses are
> removed. The companion object keeps two statics for callers that have no
> trait instance: isArchivePath (the prefetching readers) and a
> readArchiveEntries forwarder (test suites and executor-side inference whose
> mapPartitions closures cannot capture an instance).
> This is a mechanical, behavior-preserving change: the existing archive
> read/infer semantics and the per-format suites are unchanged. It makes adding
> archive support to a new format a matter of mixing in the trait rather than
> threading calls through a shared class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
