andygrove opened a new issue, #291:
URL: https://github.com/apache/datafusion-spark/issues/291
Part of #271 and apache/datafusion-comet#7300.
### What the function does
`to_xml(expr[, options])` serializes a `STRUCT` (or a `VARIANT`, from 4.1)
to an XML string. It was added in
Spark 4.0.0.
- **Arguments.** Any other input type fails analysis with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
(`requiredType` `"STRUCT"`). `options` must be a `map(...)` of string
literals. Otherwise the error is
`INVALID_OPTIONS.NON_MAP_FUNCTION` or `INVALID_OPTIONS.NON_STRING_TYPE`.
Wrong arity gives
`WRONG_NUM_ARGS.WITHOUT_SUGGESTION` (`expectedNum` `[1, 2]`).
- **Result.** The result is a nullable `STRING` (the session default string
type). The function is
NULL-intolerant and time zone aware.
Output comes from `StaxXmlGenerator`, which wraps a Woodstox
`XMLStreamWriter` with structure validation off.
- **Row element.** The struct is written as one element named `rowTag`
(default `ROW`). No XML declaration and
no `rootTag` wrapper are written; those are file-writer only.
- **Indentation.** The default `indent` is four spaces, which gives
newline-separated, indented output.
`indent` set to `''` gives compact output.
- **Fields.** A field whose name starts with `attributePrefix` (default
`_`), and is not `valueTag` (default
`_VALUE`), becomes an attribute without the prefix. Attributes are written
before child elements. The
`valueTag` field is written as the element's text.
- **NULL values.** When `nullValue` is not set (the default), NULL fields
are omitted. Otherwise the
`nullValue` text is written.
- **Arrays.** An array field becomes repeated elements with the field's
name. An array nested directly in an
array uses `arrayElementName` (default `item`).
- **Maps.** A map becomes child elements keyed by the map key. Keys with the
attribute prefix become
attributes.
### Examples
```sql
SELECT to_xml(named_struct('a', 1, 'b', 2));
-- <ROW>
-- <a>1</a>
-- <b>2</b>
-- </ROW>
SELECT to_xml(named_struct('a', 1, 'b', 2), map('indent', ''));
-- <ROW><a>1</a><b>2</b></ROW>
SELECT to_xml(named_struct('time', to_timestamp('2015-08-26',
'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy', 'indent', ''));
-- <ROW><time>26/08/2015</time></ROW>
SELECT to_xml(named_struct('time', TIME'14:30:45.123456'), map('rowTag',
'record', 'indent', ''));
-- <record><time>14:30:45.123456</time></record>
SELECT to_xml(array(named_struct('a', 1, 'b', 2)));
-- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
SELECT to_xml(map('a', 1));
-- error: DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
```
These come from the `ExpressionDescription` and `xml-functions.sql.out`
(4.2). `XmlFunctionsSuite` also has
these cases:
- with session time zone `America/Los_Angeles`, the timestamp `1800-01-01
00:00:00` is written as
`1800-01-01T00:00:00.000-07:52:58`;
- with `timestampFormat` `yyyy-MM-dd HH:mm:ss.SSSSSS`, it writes `2019-11-18
11:56:00.123456`.
### Edge cases to test
- **Value formatting.**
- Numbers use Java `toString`, so `DOUBLE` and `FLOAT` print as `1.0E10`,
`NaN` and `Infinity`, and
`DECIMAL` uses `Decimal.toString`. Booleans print as `true` or `false`.
- `DATE` uses `dateFormat` (default `yyyy-MM-dd`).
- `TIMESTAMP` uses `timestampFormat`. The default is
`yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]`;
`spark.sql.legacy.timeParserPolicy=LEGACY` changes it to `...ss.SSSXXX`,
and the internal
`supportSecondOffsetFormat` setting changes the offset to `[XXXXX]`. The
value is rendered in the session
time zone or the `timeZone` option. Test DST and historical (LMT)
offsets, as in the 1800 case above.
- `TIMESTAMP_NTZ` uses `timestampNTZFormat` (default
`yyyy-MM-dd'T'HH:mm:ss[.SSS]`), and `TIME` uses
`timeFormat` (4.2).
- `BINARY` follows `spark.sql.binaryOutputStyle`.
- **Escaping and names.**
- `<`, `&` and `>` in text, and quotes in attributes, are escaped by the
writer.
- Field names that aren't valid XML names (`'1a'`, a name with a space)
fail when `validateName=true`
(default). Check the exact error against Spark.
- **Empty struct values.** Nested NULL structs are omitted. A struct whose
fields are all NULL writes an empty
element. Compare the exact output (`<ROW></ROW>` or `<ROW/>`) against
Spark.
- **Attributes and `valueTag`.** For `named_struct('_id', 1, '_VALUE',
'x')`, attribute fields are written
first, whatever the field order.
- **Arrays.** Arrays of primitives, arrays of structs, and NULL array
elements (omitted unless `nullValue` is
set). Nested arrays use `arrayElementName`.
- **Option validation.** `attributePrefix` must be non-empty (`require`
fails otherwise), and `rowTag` must
not contain angle brackets.
- **Round-trips.** `to_xml` followed by `from_xml` with the same schema
should round-trip.
- NULL input gives NULL.
### Differences across Spark versions
- Not available in 3.4 or 3.5.
- 4.1: accepts `VARIANT` input, and adds the `singleVariantColumn` handling
in `StaxXmlGenerator.write`.
- 4.2: adds `TIME` and the `timeFormat` option.
### Implementation notes
- Spark: `StructsToXml` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala`,
`StructsToXmlEvaluator` in
`.../expressions/xml/XmlExpressionEvalUtils.scala`, and the writer in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlGenerator.scala`
with
`IndentingXMLStreamWriter.scala`. Options are in `XmlOptions.scala`.
- DataFusion core has no XML functions.
- A writer can be hand-rolled, or built on `quick-xml`'s `Writer`. The crate
has no XML dependency today.
Match the indentation rules of `IndentingXMLStreamWriter` exactly. Reuse
the datetime formatting helpers
that `to_json` and `date_format` use, if they exist by then.
- Tests: add `crates/sqllogictest/test_files/xml/to_xml.slt` and document
the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]