akshatshenoi-db opened a new pull request, #57414:
URL: https://github.com/apache/spark/pull/57414
### What changes were proposed in this pull request?
`JsonDataSource` and `XmlDataSource` each carried a custom per-entry,
line-delimited
archive-schema-inference branch in `inferWithArchives` (added alongside
JSON/XML archive support):
for single-line mode they streamed every archive entry and loose file, split
the bytes into lines,
and fed them through a bespoke `JsonInferSchema` / `XmlInferSchema` pass.
This duplicated the
format's existing single-line inference path
(`TextInput{Json,Xml}DataSource.infer` via
`createBaseDataset` / `inferFromDataset`), which already tokenizes
line-delimited records by
delegating to the Text data source.
That duplication was only necessary because the Text data source could not
read archives. Since
SPARK-57705 added archive support to `TextFileFormat`, that is no longer the
case, so this PR
removes the custom single-line branch in both `inferWithArchives` methods.
Single-line archive
schema inference now falls through the dispatch in `inferSchema` to the
normal `infer` path — the
exact same path a directory or loose-file read takes — so it is identical to
a directory read by
construction. `inferWithArchives` moves into the multi-line data source and
is multi-line-only
(it reads whole streams / `rowTag`-tokenizes, which has no line model to
reuse).
It also removes the DSv2 archive-inference refuse guard: DSv2 + archive
reads are unsupported, so
the `supportsArchiveScan` parameter on `inferSchema` — which made the DSv2
path return `None`
(`UNABLE_TO_INFER_SCHEMA`) for archive inputs — is now dead and is dropped
from `inferSchema` and
its CSV/JSON callers (`{Csv,Json}FileFormat`, `{CSV,Json}Table`). CSV also
drops the parameter's
gate on the archive-inference pass, which now always runs when an archive
input is present, matching
the V1 behavior.
### Why are the changes needed?
The custom single-line inference branch was a workaround for the Text data
source's lack of archive
support; that support now exists, so the workaround is dead code that can
diverge from the real
single-line read path. Removing it makes single-line archive inference
identical to a directory read
by construction rather than by a parallel implementation. The
`supportsArchiveScan` guard is
likewise dead once the parameter has no caller that needs it.
### Does this PR introduce _any_ user-facing change?
No. This is a behavior-preserving refactor; single-line archive inference
produces the same schema
(now via the shared path), multi-line inference is unchanged, and the DSv2
path still does not read
archives.
### How was this patch tested?
Existing archive test suites already assert that single-line archive
inference equals the directory
read (`inferredSchema(archive) == inferredSchema(dir)`), cross-entry type
widening, corrupt-entry
skip, and null-across-entries widening — these are unchanged and remain the
safety net
(`JSON{Tar,Zip}ArchiveReadSuite`, `XML{Tar,Zip}ArchiveReadSuite`). The two
DSv2-refuse tests are
removed along with the guard they covered.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
