andygrove opened a new issue, #281:
URL: https://github.com/apache/datafusion-spark/issues/281

   Part of #271 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `xpath(xml, path)` evaluates an XPath 1.0 expression against an XML string 
and returns an `ARRAY<STRING>` with
   the DOM node value (`Node.getNodeValue()`) of every node in the resulting 
node-set, in document order. Both
   arguments are strings. From 4.0 any collation is accepted, including trim 
collations, and the element type is the
   session default string type. `path` must be foldable. A column path fails 
analysis with
   `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` (`inputName` = `` `path` ``). The 
result is always nullable. Available
   since Spark 2.0.0. It was ported from Hive's `UDFXPathUtil`.
   
   This issue also specifies the shared evaluator behind `xpath_boolean`, 
`xpath_short`, `xpath_int`,
   `xpath_long`, `xpath_float`, `xpath_double` / `xpath_number` and 
`xpath_string`. Those issues cover only their
   own result conversion and point back here. All nine functions should share 
one parser and evaluator.
   
   The node value is the text for text, CDATA and comment nodes and the value 
for attribute nodes. It is NULL
   for element nodes, so `a/b` returns `[null,null,null]` while `a/b/text()` 
returns the strings.
   
   ### Examples
   
   ```sql
   SELECT 
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b/text()');
   -- ["b1","b2","b3"]
   SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b');
   -- [null,null,null]
   SELECT 
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/text()');
   -- []
   SELECT 
xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/*/text()');
   -- ["b1","b2","b3","c1","c2"]
   SELECT xpath('<a><b class="bb">b1</b><b>b2</b><b>b3</b><c 
class="bb">c1</c><c>c2</c></a>', 'a/*[@class="bb"]/text()');
   -- ["b1","c1"]
   SELECT coalesce(xpath('<a></a>', 'a')[0], 'nul');
   -- nul
   ```
   
   These come from the `ExpressionDescription`, `XPathExpressionSuite` and the 
Hive `udf_xpath` golden files
   that `HiveCompatibilitySuite` runs.
   
   ### Shared semantics (all `xpath*` functions)
   
   - **NULL and empty input.** The result is NULL if `xml` is NULL or `''`, or 
if `path` is NULL or `''`
     (`XPathEvaluator.evaluate`). The check is a plain `isEmpty`, so a 
whitespace-only `xml` is parsed and fails.
   - **Lazy path compilation.** The path is compiled (and cached) on the first 
row whose `xml` is non-empty. An
     invalid path such as `'!#$'` or `'a/text('` therefore raises nothing when 
every row is NULL or empty. When it
     does fail, the error is a plain `RuntimeException("Invalid XPath '<path>'" 
+ cause message)`, not a Spark error
     condition.
   - **Invalid XML.** `builder.parse` throws a `SAXParseException`. This is 
wrapped in
     `RuntimeException("Error loading expression '<path>'", e)`. For `'<a>/a>'` 
the root cause message is
     `XML document structures must start and end within the same entity.` There 
is no NULL-on-error mode, and
     `spark.sql.ansi.enabled` has no effect. A result-type conversion failure 
inside `XPathExpression.evaluate`
     (an `XPathExpressionException`) is reported as `RuntimeException("Invalid 
XML document: <msg>\n<xml>")`.
   - **Parser.** The parser is the JDK `DocumentBuilderFactory` with default 
settings, so it is not
     namespace-aware, does not validate, keeps whitespace-only text nodes, and 
does not coalesce CDATA. It reads
     from a `String` reader, so the `encoding` in an XML declaration is 
ignored. Only two features are set:
     `external-general-entities` and `external-parameter-entities` are both 
false.
     - The internal DTD subset is processed, so internal entities (`<!DOCTYPE a 
[<!ENTITY e "x">]><a>&e;</a>`)
       expand.
     - An external entity reference expands to nothing. The `embedFailure` test 
in `UDFXPathUtilSuite` shows
       `xpath_string` returning `''` for `<!ENTITY embed SYSTEM "file">`.
     - Loading an external DTD (`load-external-dtd`) is not disabled. A native 
implementation should never
       fetch it, and should document that as a known divergence.
   - **Namespaces.** No `NamespaceContext` is installed and the DOM is not 
namespace-aware. A prefixed name test
     such as `ns:b` cannot be resolved, so the usual workaround is 
`*[local-name()='b']`. Spark's tests don't
     cover documents with `xmlns`, so the behavior of `a/b` on `<a 
xmlns="urn:x"><b>1</b></a>` must be checked
     against Spark before an output is asserted.
   - **Engine.** This is the JDK's built-in XPath 1.0 (Xalan-derived). Required 
features include all axes,
     predicates, `@attr`, `//`, `text()`, `node()`, `comment()`, unions (`|`), 
and the core function library:
     `count`, `sum`, `string`, `number`, `boolean`, `concat`, `contains`, 
`starts-with`, `substring*`,
     `normalize-space`, `translate`, `local-name`, `name`, `position`, `last`, 
`not`, `floor`, `ceiling` and
     `round`. Spark registers no variable or function resolvers, so `$var` and 
extension functions fail.
   - **Determinism.** The functions are deterministic and per-row. Spark marks 
them null-intolerant, so filters
     get an `IsNotNull(xml)` (see `XPathFunctionsSuite`).
   
   ### Edge cases to test (`xpath`)
   
   - NULL `xml`, NULL `path`, `''` for each, and a non-literal path (analysis 
error).
   - An empty node-set gives `[]`, not NULL (`a/text()` above).
   - Element, attribute (`a/b/@id`), text, comment and CDATA nodes, and mixed 
content. Element nodes give NULL
     array entries.
   - An expression that doesn't return a node-set, such as `count(a/b)` or 
`'x'`. By code reading, the JDK can't
     convert the result to `NODESET` and throws an `XPathExpressionException`, 
which Spark reports as
     `Invalid XML document: ...`. Confirm the exact message against Spark.
   - Whitespace-only text nodes between elements: `a/node()` on `<a> <b/> </a>` 
includes them.
   - Unicode text and multi-byte characters in element names.
   - Malformed XML, a DTD with an internal entity, and an external entity 
(which expands to empty).
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5: `inputTypes` is `(StringType, StringType)`, and the 
`NON_FOLDABLE_INPUT` parameter is
     `inputName = "path"` without backticks. The expression is 
`CodegenFallback` with `NullIntolerant`.
   - 4.0+: the expression is `RuntimeReplaceable` (an `Invoke` on an 
`XPathEvaluator`). Collated string inputs are
     accepted, and `inputName` is `` `path` ``. Results are unchanged.
   - 4.1 and 4.2: no changes in `xpath.scala` or `UDFXPathUtil.java`.
   
   ### Implementation notes
   
   - Spark: `XPathList` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala`,
     the evaluators in `.../expressions/xml/XmlExpressionEvalUtils.scala`, and 
the parser and engine in
     
`sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/xml/UDFXPathUtil.java`.
   - DataFusion core has no XML or XPath functions.
   - `crates/spark/src/function/xml/mod.rs` exists but registers nothing. The 
`.slt` files under
     `crates/sqllogictest/test_files/xml/` (`xpath.slt`, `xpath_boolean.slt`, 
`xpath_string.slt`) only hold
     commented-out queries ported from Sail, and should be enabled.
   - The crate has no XML dependency today, so one must be added. Options:
     - `sxd-document` + `sxd-xpath`: the only pure-Rust XPath 1.0 engine. The 
last release was 2019 (0.4.x).
       Check its DTD and internal-entity support, its number-to-string 
formatting, its `NaN` handling, and its
       error behavior against Xalan.
     - `roxmltree` or `quick-xml` + a hand-written XPath 1.0 evaluator. 
`roxmltree` expands internal entities and
       does not resolve external ones. Neither crate ships an XPath engine.
     - `xee-xpath` targets XPath 3.1. Its typing, comparisons, `sum()` of 
non-numeric strings and number
       formatting differ from 1.0, so it would need a 1.0 compatibility layer.
     - `libxml` (libxml2 bindings) has complete XPath 1.0 but adds a C 
dependency.
     Whichever is chosen, the XPath 1.0 behaviors to verify are: 
string-to-number (no exponent, no leading `+`,
     surrounding whitespace allowed, anything else `NaN`); number-to-string 
(`3`, not `3.0`; `NaN`, `Infinity`);
     node-set comparisons with existential semantics; and the string value of 
an element (concatenated
     descendant text).
   - Compile the path once per batch, since it is a literal, and return an 
error for a non-scalar path.
   - Tests: add `crates/sqllogictest/test_files/xml/xpath.slt` (replace the 
commented stub) and document the
     function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to