[
https://issues.apache.org/jira/browse/SPARK-57479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Shenoi updated SPARK-57479:
----------------------------------
Description:
Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419)
tar-archive support, extend the XML data source to read and infer schema from
.tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
When the flag is enabled, the V1 XML source treats a tar archive as a directory
of its entries: each entry is streamed through the StaxXmlParser (never
unpacked to disk) and parsed exactly like a standalone XML file. Both the read
and schema-inference paths handle each entry per parsing mode, mirroring a
non-archive read — multi-line parses each entry as one document; single-line
treats each line as a record (run through a FailureSafeParser for the same
per-record corrupt-record handling as a non-archive read).
Schema inference makes a single XmlInferSchema pass over every archive entry
together with any loose files, so the inferred schema matches a directory read
of the same files (a field typed in one input but absent in another widens; a
NullType field survives to one final canonicalization rather than being
collapsed per input).
The whole archive is one non-splittable unit. XML has no DSv2 reader, so the
archive scan is V1-only.
was:
Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419)
tar-archive support, extend the XML data source to read and infer schema from
.tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
When the flag is enabled, the V1 XML source treats a tar archive as a directory
of its entries: each entry is streamed through the StaxXmlParser (never
unpacked to disk) and tokenized into its rowTag-delimited records, exactly like
a standalone XML file. Schema inference makes a single XmlInferSchema pass over
every archive entry together with any loose files, so the inferred schema
matches a directory read of the same files (a field typed in one input but
absent in another widens; a NullType field survives to one final
canonicalization rather than being collapsed per input).
The whole archive is one non-splittable unit. XML has no DSv2 reader, so the
archive scan is V1-only.
> [SQL] Read and infer XML schema from tar archives
> -------------------------------------------------
>
> Key: SPARK-57479
> URL: https://issues.apache.org/jira/browse/SPARK-57479
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419)
> tar-archive support, extend the XML data source to read and infer schema from
> .tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
> When the flag is enabled, the V1 XML source treats a tar archive as a
> directory of its entries: each entry is streamed through the StaxXmlParser
> (never unpacked to disk) and parsed exactly like a standalone XML file. Both
> the read and schema-inference paths handle each entry per parsing mode,
> mirroring a non-archive read — multi-line parses each entry as one document;
> single-line treats each line as a record (run through a FailureSafeParser for
> the same per-record corrupt-record handling as a non-archive read).
> Schema inference makes a single XmlInferSchema pass over every archive entry
> together with any loose files, so the inferred schema matches a directory
> read of the same files (a field typed in one input but absent in another
> widens; a NullType field survives to one final canonicalization rather than
> being collapsed per input).
> The whole archive is one non-splittable unit. XML has no DSv2 reader, so the
> archive scan is V1-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
