andygrove opened a new issue, #291:
URL: https://github.com/apache/datafusion-spark/issues/291

   Part of #271 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `to_xml(expr[, options])` serializes a `STRUCT` (or a `VARIANT`, from 4.1) 
to an XML string. It was added in
   Spark 4.0.0.
   
   - **Arguments.** Any other input type fails analysis with 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
     (`requiredType` `"STRUCT"`). `options` must be a `map(...)` of string 
literals. Otherwise the error is
     `INVALID_OPTIONS.NON_MAP_FUNCTION` or `INVALID_OPTIONS.NON_STRING_TYPE`. 
Wrong arity gives
     `WRONG_NUM_ARGS.WITHOUT_SUGGESTION` (`expectedNum` `[1, 2]`).
   - **Result.** The result is a nullable `STRING` (the session default string 
type). The function is
     NULL-intolerant and time zone aware.
   
   Output comes from `StaxXmlGenerator`, which wraps a Woodstox 
`XMLStreamWriter` with structure validation off.
   
   - **Row element.** The struct is written as one element named `rowTag` 
(default `ROW`). No XML declaration and
     no `rootTag` wrapper are written; those are file-writer only.
   - **Indentation.** The default `indent` is four spaces, which gives 
newline-separated, indented output.
     `indent` set to `''` gives compact output.
   - **Fields.** A field whose name starts with `attributePrefix` (default 
`_`), and is not `valueTag` (default
     `_VALUE`), becomes an attribute without the prefix. Attributes are written 
before child elements. The
     `valueTag` field is written as the element's text.
   - **NULL values.** When `nullValue` is not set (the default), NULL fields 
are omitted. Otherwise the
     `nullValue` text is written.
   - **Arrays.** An array field becomes repeated elements with the field's 
name. An array nested directly in an
     array uses `arrayElementName` (default `item`).
   - **Maps.** A map becomes child elements keyed by the map key. Keys with the 
attribute prefix become
     attributes.
   
   ### Examples
   
   ```sql
   SELECT to_xml(named_struct('a', 1, 'b', 2));
   -- <ROW>
   --     <a>1</a>
   --     <b>2</b>
   -- </ROW>
   SELECT to_xml(named_struct('a', 1, 'b', 2), map('indent', ''));
   -- <ROW><a>1</a><b>2</b></ROW>
   SELECT to_xml(named_struct('time', to_timestamp('2015-08-26', 
'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy', 'indent', ''));
   -- <ROW><time>26/08/2015</time></ROW>
   SELECT to_xml(named_struct('time', TIME'14:30:45.123456'), map('rowTag', 
'record', 'indent', ''));
   -- <record><time>14:30:45.123456</time></record>
   SELECT to_xml(array(named_struct('a', 1, 'b', 2)));
   -- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
   SELECT to_xml(map('a', 1));
   -- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
   ```
   
   These come from the `ExpressionDescription` and `xml-functions.sql.out` 
(4.2). `XmlFunctionsSuite` also has
   these cases:
   
   - with session time zone `America/Los_Angeles`, the timestamp `1800-01-01 
00:00:00` is written as
     `1800-01-01T00:00:00.000-07:52:58`;
   - with `timestampFormat` `yyyy-MM-dd HH:mm:ss.SSSSSS`, it writes `2019-11-18 
11:56:00.123456`.
   
   ### Edge cases to test
   
   - **Value formatting.**
     - Numbers use Java `toString`, so `DOUBLE` and `FLOAT` print as `1.0E10`, 
`NaN` and `Infinity`, and
       `DECIMAL` uses `Decimal.toString`. Booleans print as `true` or `false`.
     - `DATE` uses `dateFormat` (default `yyyy-MM-dd`).
     - `TIMESTAMP` uses `timestampFormat`. The default is 
`yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]`;
       `spark.sql.legacy.timeParserPolicy=LEGACY` changes it to `...ss.SSSXXX`, 
and the internal
       `supportSecondOffsetFormat` setting changes the offset to `[XXXXX]`. The 
value is rendered in the session
       time zone or the `timeZone` option. Test DST and historical (LMT) 
offsets, as in the 1800 case above.
     - `TIMESTAMP_NTZ` uses `timestampNTZFormat` (default 
`yyyy-MM-dd'T'HH:mm:ss[.SSS]`), and `TIME` uses
       `timeFormat` (4.2).
     - `BINARY` follows `spark.sql.binaryOutputStyle`.
   - **Escaping and names.**
     - `<`, `&` and `>` in text, and quotes in attributes, are escaped by the 
writer.
     - Field names that aren't valid XML names (`'1a'`, a name with a space) 
fail when `validateName=true`
       (default). Check the exact error against Spark.
   - **Empty struct values.** Nested NULL structs are omitted. A struct whose 
fields are all NULL writes an empty
     element. Compare the exact output (`<ROW></ROW>` or `<ROW/>`) against 
Spark.
   - **Attributes and `valueTag`.** For `named_struct('_id', 1, '_VALUE', 
'x')`, attribute fields are written
     first, whatever the field order.
   - **Arrays.** Arrays of primitives, arrays of structs, and NULL array 
elements (omitted unless `nullValue` is
     set). Nested arrays use `arrayElementName`.
   - **Option validation.** `attributePrefix` must be non-empty (`require` 
fails otherwise), and `rowTag` must
     not contain angle brackets.
   - **Round-trips.** `to_xml` followed by `from_xml` with the same schema 
should round-trip.
   - NULL input gives NULL.
   
   ### Differences across Spark versions
   
   - Not available in 3.4 or 3.5.
   - 4.1: accepts `VARIANT` input, and adds the `singleVariantColumn` handling 
in `StaxXmlGenerator.write`.
   - 4.2: adds `TIME` and the `timeFormat` option.
   
   ### Implementation notes
   
   - Spark: `StructsToXml` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala`,
     `StructsToXmlEvaluator` in 
`.../expressions/xml/XmlExpressionEvalUtils.scala`, and the writer in
     
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlGenerator.scala`
 with
     `IndentingXMLStreamWriter.scala`. Options are in `XmlOptions.scala`.
   - DataFusion core has no XML functions.
   - A writer can be hand-rolled, or built on `quick-xml`'s `Writer`. The crate 
has no XML dependency today.
     Match the indentation rules of `IndentingXMLStreamWriter` exactly. Reuse 
the datetime formatting helpers
     that `to_json` and `date_format` use, if they exist by then.
   - Tests: add `crates/sqllogictest/test_files/xml/to_xml.slt` and document 
the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to