[
https://issues.apache.org/jira/browse/SPARK-57479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-57479.
---------------------------------
Resolution: Fixed
Issue resolved by pull request 56572
[https://github.com/apache/spark/pull/56572]
> [SQL] Read and infer XML schema from tar archives
> -------------------------------------------------
>
> Key: SPARK-57479
> URL: https://issues.apache.org/jira/browse/SPARK-57479
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419)
> tar-archive support, extend the XML data source to read and infer schema from
> .tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
> When the flag is enabled, the V1 XML source treats a tar archive as a
> directory of its entries: each entry is streamed through the StaxXmlParser
> (never unpacked to disk) and parsed exactly like a standalone XML file. Both
> the read and schema-inference paths handle each entry per parsing mode,
> mirroring a non-archive read — multi-line parses each entry as one document;
> single-line treats each line as a record (run through a FailureSafeParser for
> the same per-record corrupt-record handling as a non-archive read).
> Schema inference makes a single XmlInferSchema pass over every archive entry
> together with any loose files, so the inferred schema matches a directory
> read of the same files (a field typed in one input but absent in another
> widens; a NullType field survives to one final canonicalization rather than
> being collapsed per input).
> The whole archive is one non-splittable unit. XML has no DSv2 reader, so the
> archive scan is V1-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
