[
https://issues.apache.org/jira/browse/SPARK-57135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-57135:
-----------------------------------
Assignee: Akshat Shenoi
> [SQL] Support reading CSV files inside tar archives
> ---------------------------------------------------
>
> Key: SPARK-57135
> URL: https://issues.apache.org/jira/browse/SPARK-57135
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
>
> Spark cannot currently read CSV files packaged inside tar archives (.tar,
> .tar.gz, .tgz); users must unpack them externally first.
> This adds opt-in support (spark.sql.files.archive.reader.enabled, default
> false) for reading such archives through the CSV data source by streaming
> each entry through the CSV parser, without materializing entries to local
> disk:
> * A streaming ArchiveReader opens the tar once and yields one bounded
> InputStream per entry, advancing lazily so memory stays bounded regardless of
> archive size. Directories are skipped, along with any entry Spark's own file
> listing would filter out — dot- and underscore-prefixed names (e.g. .{_}x,
> .DS_Store, _SUCCESS, _committed{_}*) and anything under a dot-/underscore-
> prefixed directory (e.g. a leftover _temporary/) — applied per path
> component, so an archive parses like a directory of the same files. .tar.gz
> is decompressed via Hadoop's codec factory; .tgz is gunzipped explicitly.
> ArchiveReader is an abstract base (TarArchiveReader is the only
> implementation today), so other archive formats can be added as additive
> subclasses.
> * CSVFileFormat treats archives as non-splittable (one split per archive)
> and streams each entry through UnivocityParser, handling each entry as a
> standalone CSV file (headers, multiLine, delimiters, column pruning).
> Scope: CSV reads over tar only. Schema inference from archives, and other
> file formats (e.g. JSON, text, XML), are left to follow-ups. Streaming
> supports formats parseable sequentially; formats needing random access
> (Parquet/ORC footers) cannot stream from a tar and are out of scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
