andygrove opened a new issue, #281:
URL: https://github.com/apache/datafusion-spark/issues/281
Part of #271 and apache/datafusion-comet#7300.
### What the function does
`xpath(xml, path)` evaluates an XPath 1.0 expression against an XML string
and returns an `ARRAY<STRING>` with
the DOM node value (`Node.getNodeValue()`) of every node in the resulting
node-set, in document order. Both
arguments are strings. From 4.0 any collation is accepted, including trim
collations, and the element type is the
session default string type. `path` must be foldable. A column path fails
analysis with
`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` (`inputName` = `` `path` ``). The
result is always nullable. Available
since Spark 2.0.0. It was ported from Hive's `UDFXPathUtil`.
This issue also specifies the shared evaluator behind `xpath_boolean`,
`xpath_short`, `xpath_int`,
`xpath_long`, `xpath_float`, `xpath_double` / `xpath_number` and
`xpath_string`. Those issues cover only their
own result conversion and point back here. All nine functions should share
one parser and evaluator.
The node value is the text for text, CDATA and comment nodes and the value
for attribute nodes. It is NULL
for element nodes, so `a/b` returns `[null,null,null]` while `a/b/text()`
returns the strings.
### Examples
```sql
SELECT
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b/text()');
-- ["b1","b2","b3"]
SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');
-- [null,null,null]
SELECT
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/text()');
-- []
SELECT
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/*/text()');
-- ["b1","b2","b3","c1","c2"]
SELECT xpath('<a><b class="bb">b1</b><b>b2</b><b>b3</b><c
class="bb">c1</c><c>c2</c></a>', 'a/*[@class="bb"]/text()');
-- ["b1","c1"]
SELECT coalesce(xpath('<a></a>', 'a')[0], 'nul');
-- nul
```
These come from the `ExpressionDescription`, `XPathExpressionSuite` and the
Hive `udf_xpath` golden files
that `HiveCompatibilitySuite` runs.
### Shared semantics (all `xpath*` functions)
- **NULL and empty input.** The result is NULL if `xml` is NULL or `''`, or
if `path` is NULL or `''`
(`XPathEvaluator.evaluate`). The check is a plain `isEmpty`, so a
whitespace-only `xml` is parsed and fails.
- **Lazy path compilation.** The path is compiled (and cached) on the first
row whose `xml` is non-empty. An
invalid path such as `'!#$'` or `'a/text('` therefore raises nothing when
every row is NULL or empty. When it
does fail, the error is a plain `RuntimeException("Invalid XPath '<path>'"
+ cause message)`, not a Spark error
condition.
- **Invalid XML.** `builder.parse` throws a `SAXParseException`. This is
wrapped in
`RuntimeException("Error loading expression '<path>'", e)`. For `'<a>/a>'`
the root cause message is
`XML document structures must start and end within the same entity.` There
is no NULL-on-error mode, and
`spark.sql.ansi.enabled` has no effect. A result-type conversion failure
inside `XPathExpression.evaluate`
(an `XPathExpressionException`) is reported as `RuntimeException("Invalid
XML document: <msg>\n<xml>")`.
- **Parser.** The parser is the JDK `DocumentBuilderFactory` with default
settings, so it is not
namespace-aware, does not validate, keeps whitespace-only text nodes, and
does not coalesce CDATA. It reads
from a `String` reader, so the `encoding` in an XML declaration is
ignored. Only two features are set:
`external-general-entities` and `external-parameter-entities` are both
false.
- The internal DTD subset is processed, so internal entities (`<!DOCTYPE a
[<!ENTITY e "x">]><a>&e;</a>`)
expand.
- An external entity reference expands to nothing. The `embedFailure` test
in `UDFXPathUtilSuite` shows
`xpath_string` returning `''` for `<!ENTITY embed SYSTEM "file">`.
- Loading an external DTD (`load-external-dtd`) is not disabled. A native
implementation should never
fetch it, and should document that as a known divergence.
- **Namespaces.** No `NamespaceContext` is installed and the DOM is not
namespace-aware. A prefixed name test
such as `ns:b` cannot be resolved, so the usual workaround is
`*[local-name()='b']`. Spark's tests don't
cover documents with `xmlns`, so the behavior of `a/b` on `<a
xmlns="urn:x"><b>1</b></a>` must be checked
against Spark before an output is asserted.
- **Engine.** This is the JDK's built-in XPath 1.0 (Xalan-derived). Required
features include all axes,
predicates, `@attr`, `//`, `text()`, `node()`, `comment()`, unions (`|`),
and the core function library:
`count`, `sum`, `string`, `number`, `boolean`, `concat`, `contains`,
`starts-with`, `substring*`,
`normalize-space`, `translate`, `local-name`, `name`, `position`, `last`,
`not`, `floor`, `ceiling` and
`round`. Spark registers no variable or function resolvers, so `$var` and
extension functions fail.
- **Determinism.** The functions are deterministic and per-row. Spark marks
them null-intolerant, so filters
get an `IsNotNull(xml)` (see `XPathFunctionsSuite`).
### Edge cases to test (`xpath`)
- NULL `xml`, NULL `path`, `''` for each, and a non-literal path (analysis
error).
- An empty node-set gives `[]`, not NULL (`a/text()` above).
- Element, attribute (`a/b/@id`), text, comment and CDATA nodes, and mixed
content. Element nodes give NULL
array entries.
- An expression that doesn't return a node-set, such as `count(a/b)` or
`'x'`. By code reading, the JDK can't
convert the result to `NODESET` and throws an `XPathExpressionException`,
which Spark reports as
`Invalid XML document: ...`. Confirm the exact message against Spark.
- Whitespace-only text nodes between elements: `a/node()` on `<a> <b/> </a>`
includes them.
- Unicode text and multi-byte characters in element names.
- Malformed XML, a DTD with an internal entity, and an external entity
(which expands to empty).
### Differences across Spark versions
- 3.4 and 3.5: `inputTypes` is `(StringType, StringType)`, and the
`NON_FOLDABLE_INPUT` parameter is
`inputName = "path"` without backticks. The expression is
`CodegenFallback` with `NullIntolerant`.
- 4.0+: the expression is `RuntimeReplaceable` (an `Invoke` on an
`XPathEvaluator`). Collated string inputs are
accepted, and `inputName` is `` `path` ``. Results are unchanged.
- 4.1 and 4.2: no changes in `xpath.scala` or `UDFXPathUtil.java`.
### Implementation notes
- Spark: `XPathList` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala`,
the evaluators in `.../expressions/xml/XmlExpressionEvalUtils.scala`, and
the parser and engine in
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/xml/UDFXPathUtil.java`.
- DataFusion core has no XML or XPath functions.
- `crates/spark/src/function/xml/mod.rs` exists but registers nothing. The
`.slt` files under
`crates/sqllogictest/test_files/xml/` (`xpath.slt`, `xpath_boolean.slt`,
`xpath_string.slt`) only hold
commented-out queries ported from Sail, and should be enabled.
- The crate has no XML dependency today, so one must be added. Options:
- `sxd-document` + `sxd-xpath`: the only pure-Rust XPath 1.0 engine. The
last release was 2019 (0.4.x).
Check its DTD and internal-entity support, its number-to-string
formatting, its `NaN` handling, and its
error behavior against Xalan.
- `roxmltree` or `quick-xml` + a hand-written XPath 1.0 evaluator.
`roxmltree` expands internal entities and
does not resolve external ones. Neither crate ships an XPath engine.
- `xee-xpath` targets XPath 3.1. Its typing, comparisons, `sum()` of
non-numeric strings and number
formatting differ from 1.0, so it would need a 1.0 compatibility layer.
- `libxml` (libxml2 bindings) has complete XPath 1.0 but adds a C
dependency.
Whichever is chosen, the XPath 1.0 behaviors to verify are:
string-to-number (no exponent, no leading `+`,
surrounding whitespace allowed, anything else `NaN`); number-to-string
(`3`, not `3.0`; `NaN`, `Infinity`);
node-set comparisons with existential semantics; and the string value of
an element (concatenated
descendant text).
- Compile the path once per batch, since it is a literal, and return an
error for a non-scalar path.
- Tests: add `crates/sqllogictest/test_files/xml/xpath.slt` (replace the
commented stub) and document the
function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]