Akshat Shenoi created SPARK-57321:
-------------------------------------
Summary: [SQL] Support schema inference of CSV files inside tar
archives
Key: SPARK-57321
URL: https://issues.apache.org/jira/browse/SPARK-57321
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.3.0
Reporter: Akshat Shenoi
Spark cannot currently read CSV files packaged inside tar archives (.tar,
.tar.gz, .tgz); users must unpack them externally first.
This adds opt-in support (spark.sql.files.archive.reader.enabled, default
false) for reading such archives through the CSV data source by streaming each
entry through the CSV parser, without materializing entries to local disk:
* A streaming ArchiveReader opens the tar once and yields one bounded
InputStream per entry, advancing lazily so memory
stays bounded regardless of archive size. Directories and dot-prefixed
entries are skipped. .tar.gz is decompressed via
Hadoop's codec factory; .tgz is gunzipped explicitly. ArchiveReader is an
abstract base (TarArchiveReader is the only
implementation today), so other archive formats can be added as additive
subclasses.
* CSVFileFormat treats archives as non-splittable (one split per archive) and
streams each entry through UnivocityParser,
handling each entry as a standalone CSV file (headers, multiLine, delimiters,
column pruning).
Scope: CSV reads over tar only. Schema inference from archives, and other file
formats (e.g. JSON, text, XML), are left to follow-ups. Streaming supports
formats parseable sequentially; formats needing random access (Parquet/ORC
footers) cannot stream from a tar and are out of scope.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
