andygrove opened a new issue, #292:
URL: https://github.com/apache/datafusion-spark/issues/292

   Part of #271 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `schema_of_xml(xml[, options])` infers the Spark type of one XML record and 
returns it as a DDL-formatted
   `STRING` (`DataType.sql`), for example `STRUCT<a: BIGINT, b: STRING>`. It 
was added in Spark 4.0.0.
   
   - **Arguments.** `xml` must be foldable. A column fails with 
`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`, and
     `schema_of_xml(null)` fails with `DATATYPE_MISMATCH.UNEXPECTED_NULL`. 
`options` is a `map(...)` of string
     literals using the `XmlOptions` keys.
   - **Result.** The result is non-nullable and in practice a constant, 
typically passed as the schema of
     `from_xml`.
   - **Root element.** Its name is ignored. Its attributes and children become 
the fields.
   
   Inference is `XmlInferSchema.infer(xml)` followed by `canonicalizeType`, 
with `XmlOptions` in the UTC time
   zone and case sensitivity from `spark.sql.caseSensitive`. A `StructType` 
result whose fields are all removed
   by canonicalization becomes `STRUCT<>`.
   
   ### Examples
   
   ```sql
   SELECT schema_of_xml('<p><a>1</a></p>');
   -- STRUCT<a: BIGINT>
   SELECT schema_of_xml('<p><a attr="2">1</a><a>3</a></p>', 
map('excludeAttribute', 'true'));
   -- STRUCT<a: ARRAY<BIGINT>>
   SELECT schema_of_xml('<p><a>1</a><b>"2"</b></p>');
   -- STRUCT<a: BIGINT, b: STRING>
   SELECT schema_of_xml('<record><time>14:30:45</time></record>', map('rowTag', 
'record'));
   -- STRUCT<time: TIMESTAMP>
   SELECT 
schema_of_xml(regexp_replace('<ROW><item_id>1</item_id><item_price>0.1</item_price></ROW>',
 'item_', ''));
   -- STRUCT<id: BIGINT, price: DOUBLE>
   SELECT schema_of_xml('<ROW><id></id></ROW>');
   -- STRUCT<id: STRING>
   SELECT schema_of_xml('<ROW><a>1<ROW>', map('mode', 'PERMISSIVE'));
   -- STRUCT<_corrupt_record: STRING>
   SELECT schema_of_xml(42);
   -- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE (4.1+)
   ```
   
   These come from the `ExpressionDescription`, `xml-functions.sql.out`, 
`XmlFunctionsSuite` and `XmlSuite`. The
   `regexp_replace` case shows that any foldable expression is accepted, not 
only a literal.
   
   ### Edge cases to test
   
   - **Value inference** (`inferFrom`, after trimming when 
`ignoreSurroundingSpaces=true`, the default). The rules
     are tried in this order:
     1. empty: `NullType`, which becomes `STRING`;
     2. `Long`-parseable (optional sign, first character a digit): `BIGINT`;
     3. with `prefersDecimal=true`, decimal-parseable: `DECIMAL(max(p, s), s)`;
     4. double-parseable, but not ending in `d`, `D`, `f` or `F`: `DOUBLE`;
     5. `true` or `false`, case-insensitive: `BOOLEAN`;
     6. a date per `dateFormat`, only when `preferDate=true` (the default, and 
forced off under the legacy time
        parser policy): `DATE`;
     7. `TIMESTAMP_NTZ`, only when `spark.sql.timestampType=TIMESTAMP_NTZ` or 
the legacy policy is on;
     8. a timestamp per `timestampFormat`, or the default parser: `TIMESTAMP` 
(`14:30:45` above);
     9. anything else: `STRING`.
   
     With `inferSchema=false`, every value is `STRING`.
   - **Structure.**
     - Elements become struct fields sorted by name.
     - A repeated element becomes `ARRAY<...>`, with the element types merged 
by `compatibleType`. That function
       uses `findTightestCommonType` and has these rules:
       - `DOUBLE` with `DECIMAL` gives `DOUBLE`;
       - a decimal that would need precision above 38 becomes `DOUBLE`;
       - `TIMESTAMP_NTZ` with `TIMESTAMP` gives `TIMESTAMP`;
       - structs merge their fields;
       - an array with a scalar gives an array;
       - a struct with a `valueTag` field merged with a scalar widens the value 
field;
       - anything else gives `STRING`.
     - Attributes become `_`-prefixed fields (`attributePrefix`). An element 
with attributes and text gets a
       `_VALUE` field (`valueTag`).
   - **Case sensitivity.** With `spark.sql.caseSensitive=false`, `<foo>`, 
`<Foo>` and `<FOO>` merge into one array
     field named after the first name seen. With `true`, they are three fields.
   - **Other options.** `ignoreNamespace`, `excludeAttribute`, `nullValue`, 
`dateFormat`, `timestampFormat`,
     `timestampNTZFormat` and `locale`.
   - **Malformed input.**
     - `PERMISSIVE` returns `STRUCT<_corrupt_record: STRING>`, using 
`columnNameOfCorruptRecord` or the
       `spark.sql.columnNameOfCorruptRecord` config.
     - `FAILFAST` raises `MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION`, and 
`badRecord` is the corrupt column
       name (`_corrupt_record`), not the input.
     - `DROPMALFORMED` raises `PARSE_MODE_UNSUPPORTED` with `funcName` `` 
`schema_of_xml` ``.
   - **Other inputs.** Also test an empty element (`STRING`), deeply nested 
structs, mixed content, and Unicode
     element names.
   
   ### Differences across Spark versions
   
   - Not available in 3.4 or 3.5.
   - 4.1 adds the `UNEXPECTED_INPUT_TYPE` check for a non-`STRING` foldable 
argument (`schema_of_xml(42)`). In 4.0
     that query has no golden result.
   - 4.2: `TIME` values are still inferred as `TIMESTAMP` (golden above). There 
is no `TIME` inference.
   
   ### Implementation notes
   
   - Spark: `SchemaOfXml` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala`
     (`RuntimeReplaceable`, a `StaticInvoke` of 
`XmlExpressionEvalUtils.schemaOfXml`), with inference in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala`.
   - DataFusion core has no equivalent.
   - The argument is a literal, so the kernel can run once (at planning time or 
on the first batch) and return a
     scalar. It needs a Spark-style `DataType.sql` printer (`STRUCT<a: 
BIGINT>`, upper-case, `: ` separators).
     Share it with `schema_of_json` and `schema_of_csv` if they exist by then.
   - Use the same XML tokenizer as `from_xml` (`quick-xml` is a natural fit). 
The crate has no XML dependency
     today. Put it in `crates/spark/src/function/xml/`.
   - Tests: add `crates/sqllogictest/test_files/xml/schema_of_xml.slt` and 
document the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to