[
https://issues.apache.org/jira/browse/SPARK-57419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Shenoi updated SPARK-57419:
----------------------------------
Description: SPARK-57135 added support for reading CSV files packed in tar
archives (.tar/.tar.gz/.tgz) and SPARK-57321 added schema inference for them,
both gated by spark.sql.files.archive.reader.enabled; this extends the same
capability to the JSON data source. When spark.sql.files.archive.reader.enabled
is true, the V1 JSON data source reads a tar archive as if it were a directory
of its entries: each entry is streamed through ArchiveReader (never unpacked to
disk) and parsed exactly like a standalone JSON file, for both line-delimited
and multi-line JSON. Schema inference reads every archive entry together with
any loose files alongside it in a single JsonInferSchema pass, so the inferred
schema matches a directory read of the same files. The whole archive is a
single non-splittable unit, and a corrupt/missing archive is skipped as a unit
under ignoreCorruptFiles/ignoreMissingFiles. The DSv2 JSON reader does not
support archives, so it refuses to infer a schema for archive inputs (raising
UNABLE_TO_INFER_SCHEMA) rather than mis-reading raw archive bytes. Unlike CSV,
JSON needs no per-entry header handling (records are self-describing, so one
parser serves every entry) and no mergeSchema-style branching (JsonInferSchema
already merges record types by field name across all inputs, so one pass is
itself the union). This change also unifies the archive test suites: the
format-agnostic inference and complex-type tests are hoisted into
ArchiveReadSuiteBase behind capability hooks (supportsSchemaInference,
supportsComplexTypes) so CSV, JSON, and future archive formats share them
rather than each duplicating them. (was: SPARK-57135 added opt-in reading of
CSV files packaged in tar archives (.tar, .tar.gz, .tgz), but only with an
explicit schema — schema inference was out of scope, and inferring without a
schema errors (UNABLE_TO_INFER_SCHEMA).
This follow-up adds schema inference for tar archives, so
spark.read.csv("data.tar") (with spark.sql.files.archive.reader.enabled=true)
infers a schema instead of erroring, matching how a directory of the same CSV
files is inferred:
- CSVDataSource.inferSchema streams each archive's entries through the existing
ArchiveReader (never unpacking to disk), tokenizes each entry like a standalone
CSV file (dropping its header row when header=true), and feeds all entries'
rows into a single CSVInferSchema pass keyed on the first entry's header — the
same first-header, type-widening model used for a multi-file CSV read.
- When archives and loose CSV files are read together, the two inferred schemas
are merged positionally with CSV-aware type widening.
- ignoreCorruptFiles / ignoreMissingFiles are honored at archive granularity,
matching the loose-file path.
- Reuses the spark.sql.files.archive.reader.enabled config from SPARK-57135; no
new config.
Scope: CSV over tar, building on SPARK-57135. Inference for other file formats
(JSON, text, XML) follows their respective read support.
Tests: directory parity, all archive formats agree, corrupt-archive skip among
good ones, cross-entry type widening, and mixed archive + loose-file inference.)
> [SQL] Support read & schema inference of JSON files inside tar archives
> -----------------------------------------------------------------------
>
> Key: SPARK-57419
> URL: https://issues.apache.org/jira/browse/SPARK-57419
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> SPARK-57135 added support for reading CSV files packed in tar archives
> (.tar/.tar.gz/.tgz) and SPARK-57321 added schema inference for them, both
> gated by spark.sql.files.archive.reader.enabled; this extends the same
> capability to the JSON data source. When
> spark.sql.files.archive.reader.enabled is true, the V1 JSON data source reads
> a tar archive as if it were a directory of its entries: each entry is
> streamed through ArchiveReader (never unpacked to disk) and parsed exactly
> like a standalone JSON file, for both line-delimited and multi-line JSON.
> Schema inference reads every archive entry together with any loose files
> alongside it in a single JsonInferSchema pass, so the inferred schema matches
> a directory read of the same files. The whole archive is a single
> non-splittable unit, and a corrupt/missing archive is skipped as a unit under
> ignoreCorruptFiles/ignoreMissingFiles. The DSv2 JSON reader does not support
> archives, so it refuses to infer a schema for archive inputs (raising
> UNABLE_TO_INFER_SCHEMA) rather than mis-reading raw archive bytes. Unlike
> CSV, JSON needs no per-entry header handling (records are self-describing, so
> one parser serves every entry) and no mergeSchema-style branching
> (JsonInferSchema already merges record types by field name across all inputs,
> so one pass is itself the union). This change also unifies the archive test
> suites: the format-agnostic inference and complex-type tests are hoisted into
> ArchiveReadSuiteBase behind capability hooks (supportsSchemaInference,
> supportsComplexTypes) so CSV, JSON, and future archive formats share them
> rather than each duplicating them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
