andygrove opened a new issue, #290:
URL: https://github.com/apache/datafusion-spark/issues/290

   Part of #271 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `from_xml(xmlStr, schema[, options])` parses one XML record into a `STRUCT`. 
It was added in Spark 4.0.0.
   
   - **Schema.** `schema` must be a foldable string: a DDL string (`'a INT, b 
DOUBLE'` or `'STRUCT<...>'`) or
     `schema_of_xml(...)`. A non-literal schema fails with 
`INVALID_SCHEMA.NON_STRING_LITERAL`. A bad DDL string
     fails with `PARSE_SYNTAX_ERROR`. The top level must be `STRUCT`, or 
`VARIANT` from 4.1. Otherwise analysis
     fails with `DATATYPE_MISMATCH.INVALID_XML_SCHEMA`. A `MAP` with a 
non-`STRING` key anywhere in the schema
     gives `DATATYPE_MISMATCH.INVALID_XML_MAP_KEY_TYPE`.
   - **Options.** `options` must be a `map(...)` of string literals. Otherwise 
the error is
     `INVALID_OPTIONS.NON_MAP_FUNCTION` or `INVALID_OPTIONS.NON_STRING_TYPE`. 
Wrong arity gives
     `WRONG_NUM_ARGS.WITHOUT_SUGGESTION`.
   - **Result.** The result type is the schema with nullability forced 
everywhere (`asNullable`). It is nullable
     and NULL-intolerant (NULL in, NULL out). The input is a string 
(collation-aware), and the function is time
     zone aware.
   - **Root element.** The root element's name is ignored. Its attributes 
become fields and its children are
     matched against the schema, so `rowTag` has no effect: `XmlFunctionsSuite` 
shows `<foo>`, `<ROW>` and `<p>`
     all parsing.
   
   Parsing uses `StaxXmlParser` over a StAX reader. The reader is not 
namespace-aware and coalesces text. It does
   not support DTDs or external entities, and skips comments, processing 
instructions and DTD events. Records go
   through `FailureSafeParser` with options from `XmlOptions`.
   
   ### Examples
   
   ```sql
   SELECT from_xml('<p><a>1</a><b>0.8</b></p>', 'a INT, b DOUBLE');
   -- {"a":1,"b":0.8}
   SELECT from_xml('<p><time>26/08/2015</time></p>', 'time Timestamp', 
map('timestampFormat', 'dd/MM/yyyy'));
   -- {"time":2015-08-26 00:00:00}
   SELECT 
from_xml('<p><teacher>Alice</teacher><student><name>Bob</name><rank>1</rank></student><student><name>Charlie</name><rank>2</rank></student></p>',
 'STRUCT<teacher: STRING, student: ARRAY<STRUCT<name: STRING, rank: INT>>>');
   -- 
{"teacher":"Alice","student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}
   SELECT from_xml('<p><a>1</a></p>', 'struct<a:array<int>>');
   -- {"a":[1]}
   SELECT from_xml('<p><a>1</a><b>"2"</b></p>', 'struct<a:int,b:string>');
   -- {"a":1,"b":""2""}
   SELECT from_xml('<p><a>1</a><a>"2"</a></p>', 'struct<a:array<int>>');
   -- {"a":[1]}
   SELECT from_xml('<p><a>1</a><a></a></p>', 'struct<a:array<int>>');
   -- {"a":[1,null]}
   SELECT from_xml('<p><a attr="1"><b>2</b></a></p>', 
'struct<a:map<string,int>>');
   -- {"a":{"_attr":1,"b":2}}
   SELECT from_xml('<p><d>12/15 2012</d><t>12/15 2012 15:15:15</t>}</p>', 'd 
date, t timestamp', map('dateFormat', 'MM/dd yyyy', 'timestampFormat', 'MM/dd 
yyyy HH:mm:ss'));
   -- {"d":2012-12-15,"t":2012-12-15 15:15:15}
   SELECT from_xml('<p><d>02-29</d></p>', 'd date', map('dateFormat', 'MM-dd'));
   -- {"d":null}
   SELECT from_xml('<p><a>1</a><a>2</a><a>3</a></p>', 
schema_of_xml('<p><a>1</a><a>2</a></p>'));
   -- {"a":[1,2,3]}
   ```
   
   These come from the `ExpressionDescription` and `xml-functions.sql.out` 
(4.2).
   
   ### Edge cases to test
   
   - **Modes** (`mode`, case-insensitive; an unknown value falls back to 
`PERMISSIVE` with a warning):
     - `PERMISSIVE` (the default). A malformed record gives a struct of NULLs, 
not a NULL struct. `<ROW><a>1</ROW>`
       with `a INT` gives `{"a":null}`.
     - A conversion failure inside an element keeps the partial row (the `[1]` 
example above).
     - If the schema contains the corrupt-record column 
(`spark.sql.columnNameOfCorruptRecord`, default
       `_corrupt_record`, or the `columnNameOfCorruptRecord` option), that 
column gets the raw input string.
     - `FAILFAST` raises `MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION`.
     - `DROPMALFORMED` raises `PARSE_MODE_UNSUPPORTED` ("doesn't support the 
DROPMALFORMED mode").
   - **Corrupt column in the middle.** `XmlFunctionsSuite` ("corrupt record 
column in the middle") asserts that
     schema `a INT, _unparsed STRING, b INT` on the valid 
`<ROW><a>1</a><b>12</b></ROW>` gives `(1, null, null)`.
     `FailureSafeParser` reads the parsed row by its index in the schema 
without the corrupt column, so
     every field after the corrupt column is shifted. Decide whether to copy 
this bug and document the decision.
   - **Type conversion** (`convertTo` / `castTo`, after trimming when 
`ignoreSurroundingSpaces=true`, the default):
     - `INT`, `BIGINT` and `FLOAT`/`DOUBLE` strip a leading `+` or `-`, parse 
the rest with Java
       `toInt`/`toLong`/`toDouble`, and then negate. By code reading this means 
`-2147483648` and
       `-9223372036854775808` overflow and become NULL.
     - `FLOAT`/`DOUBLE` fall back to a locale `NumberFormat`.
     - `TINYINT` and `SMALLINT` use `toByte`/`toShort` directly, and `DECIMAL` 
uses the locale decimal parser.
     - `BOOLEAN` accepts `true`, `false`, `1` and `0`, case-insensitive.
     - Any failure nulls the field, or raises in `FAILFAST`.
   - **Empty values.** An empty element (`<a></a>` or `<a/>`) is NULL for 
non-string types. For `STRING` it is
     `""` unless `nullValue` is `""`. A text value equal to `nullValue` 
(default: none) becomes NULL.
   - **`STRING` field with child elements.** The field gets the inner markup as 
a string
     (`currentStructureAsString`).
   - **Attributes.** Attributes become fields named `attributePrefix + name` 
(default `_`). With
     `excludeAttribute=true` they are dropped. An element that has both 
attributes and text maps the text to
     `valueTag` (default `_VALUE`): `<a x="1">5</a>` with `a STRUCT<_VALUE: 
INT, _x: INT>`. Option checks:
     `valueTag` must be non-empty and different from `attributePrefix`.
   - **Arrays.** Repeated elements fill an `ARRAY` field. A single occurrence 
gives a one-element array.
     `wildcardColName` (default `xs_any`) collects unmatched elements as 
`STRING` or `ARRAY<STRING>`.
   - **Field names.** Matching follows `spark.sql.caseSensitive` (default 
false, so it is case-insensitive).
     Unknown elements are skipped and missing ones are NULL. 
`ignoreNamespace=true` strips the `prefix:`.
   - **Datetimes.** `timestampFormat` has no default in read mode unless the 
legacy parser policy is on. Also
     cover `timestampNTZFormat`, `dateFormat` (an invalid date such as `02-29` 
without a year gives NULL),
     `timeFormat` (4.2, `TIME(p)`), `locale`, `timeZone`, 
`spark.sql.session.timeZone`, DST transitions, and
     `spark.sql.legacy.timeParserPolicy`.
   - **Binary.** `BINARY` fields (4.1+) are parsed according to 
`spark.sql.binaryOutputStyle`.
   - **Bad options.** Validation errors from `require` 
(`IllegalArgumentException`) cover an empty `rowTag`, a
     `rowTag` with `<` or `>`, an empty `valueTag`, and a non-boolean value for 
`excludeAttribute` and other
     boolean options.
   - **Inputs.** NULL input gives NULL. Also test the empty string, 
whitespace-only input, multiple root elements,
     trailing content after the root, and Unicode in names and text.
   - **Round-trip.** `from_xml(to_xml(s), schema)` should round-trip structs 
and arrays (`XmlFunctionsSuite`).
   
   ### Differences across Spark versions
   
   - Not available in 3.4 or 3.5.
   - 4.1: adds the `VARIANT` schema and the `singleVariantColumn` option, 
`BINARY` parsing, and the
     `useLegacyXMLParser` option and `spark.sql.legacy.useLegacyXMLParser` 
config. These only affect the file
     reader.
   - 4.2: adds the `TIME` type and the `timeFormat` option.
   
   ### Implementation notes
   
   - Spark: `XmlToStructs` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala`
     and `XmlToStructsEvaluator` in 
`.../expressions/xml/XmlExpressionEvalUtils.scala`. The parser is
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala`,
 with
     `StaxXmlParserUtils.scala` and `XmlOptions.scala`. Modes are in 
`catalyst/util/FailureSafeParser.scala`.
   - DataFusion core has no XML functions.
   - Add a streaming XML parser dependency. `quick-xml` maps closely to the 
StAX event model; it doesn't process
     DTDs, and namespaces are opt-in. This crate has no XML dependency today. 
The `xpath*` family needs a DOM and
     an XPath engine instead (see the `xpath` issue), so the two may end up 
with different crates.
   - This crate has no Spark DDL schema parser. Share the approach chosen for 
`from_json`: either parse the
     literal at planning time, or let callers such as Comet pass the Arrow type 
directly.
   - A first version could support `PERMISSIVE` and `FAILFAST`, primitive, 
struct, array and string-keyed map
     types, attributes and `valueTag`, and leave `VARIANT`, XSD validation 
(`rowValidationXSDPath`) and datetime
     patterns for follow-ups.
   - Tests: add `crates/sqllogictest/test_files/xml/from_xml.slt` and document 
the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to