Akshat Shenoi created SPARK-57481:
-------------------------------------
Summary: [SQL] Read and infer Avro schema from tar archives
Key: SPARK-57481
URL: https://issues.apache.org/jira/browse/SPARK-57481
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Akshat Shenoi
Assignee: Akshat Shenoi
Fix For: 4.3.0
Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419)
tar-archive support, extend the XML data source to read and infer schema from
.tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
When the flag is enabled, the V1 XML source treats a tar archive as a directory
of its entries: each entry is streamed through the StaxXmlParser (never
unpacked to disk) and tokenized into its rowTag-delimited records, exactly like
a standalone XML file. Schema inference makes a single XmlInferSchema pass over
every archive entry together with any loose files, so the inferred schema
matches a directory read of the same files (a field typed in one input but
absent in another widens; a NullType field survives to one final
canonicalization rather than being collapsed per input).
The whole archive is one non-splittable unit. XML has no DSv2 reader, so the
archive scan is V1-only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
