andygrove opened a new issue, #283:
URL: https://github.com/apache/datafusion-spark/issues/283

   Part of #271 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `xpath_double(xml, path)` evaluates an XPath 1.0 expression, converts the 
result with the XPath `number()`
   rules (`XPathConstants.NUMBER`, a Java `Double`), and returns it as 
`DOUBLE`. `xpath_number` is a registered
   alias of the same expression: `expression[XPathDouble]("xpath_number", 
true)`, and its pretty name is
   `xpath_number`. Both arguments are strings (collation-aware from 4.0), and 
`path` must be foldable. The result
   is always nullable. Available since Spark 2.0.0.
   
   Parsing, DTD and entity handling, namespaces, NULL and empty-string 
handling, invalid-XML and invalid-path
   errors, and lazy path compilation are shared with the family. They are 
specified in the `xpath` issue, and
   this function should reuse that evaluator. This issue covers only the 
numeric conversion. `xpath_float`,
   `xpath_int`, `xpath_long` and `xpath_short` take the same double and narrow 
it.
   
   ### Examples
   
   ```sql
   SELECT xpath_double('<a><b>1</b><b>2</b></a>', 'sum(a/b)');
   -- 3.0
   SELECT xpath_double('<a>this is not a number</a>', 'a');
   -- NaN
   SELECT xpath_double('<a>this 2 is not a number</a>', 'a');
   -- NaN
   SELECT xpath_double('<a><b>2000000000</b><c>40000000000</c></a>', 'a/b * 
a/c');
   -- 8.0E19
   SELECT xpath_double('<a>try a boolean</a>', 'a = 10');
   -- 0.0
   SELECT xpath_double('<a><b class="odd">1</b><b class="even">2</b><b 
class="odd">4</b><c>8</c></a>', 'a/b');
   -- 1.0
   SELECT xpath_double('<a><b class="odd">1</b><b class="even">2</b><b 
class="odd">4</b><c>8</c></a>', 'sum(a/*)');
   -- 15.0
   SELECT xpath_double('<a><b class="odd">1</b><b class="even">2</b><b 
class="odd">4</b><c>8</c></a>', 'sum(a/b[@class="odd"])');
   -- 5.0
   SELECT xpath_number('<a><b>1.0</b><b>2.1</b></a>', 'sum(a/b)');
   -- 3.1
   ```
   
   These come from the `ExpressionDescription`, the Hive `udf_xpath_double` 
golden files run by
   `HiveCompatibilitySuite`, `XPathExpressionSuite` and `XPathFunctionsSuite`. 
In `UDFXPathUtilSuite`, `a/c[2]`
   on `<c>-77</c>` gives `-77.0`, and an attribute `k="foo"` gives `NaN`.
   
   ### Edge cases to test
   
   - Conversion rules from XPath 1.0 `number()`:
     - a node-set uses the string value of its first node in document order 
(`a/b` above gives `1.0`, not the sum);
     - a boolean gives `1.0` or `0.0` (`a = 10` above);
     - a string is parsed with XPath's grammar. Surrounding whitespace is 
allowed, as are an optional leading `-`
       and digits with an optional `.`. There is no exponent and no leading 
`+`. Anything else is `NaN`, for
       example `'1e3'`, `'+1'`, `'1,000'`, `'Infinity'` and `'0x10'`.
   - Empty node-set. The usage string says "zero if no match is found", but 
XPath 1.0 converts an empty node-set
     through `number('')`, which is `NaN`. Verify `xpath_double('<a/>', 'a/b')` 
against Spark before asserting
     either value (the integer variants map `NaN` to 0 anyway).
   - `NaN`, `Infinity` and `-Infinity` can be produced by arithmetic (`0 div 
0`, `1 div 0`, `-1 div 0`) and must
     come back as the IEEE values. `-0` (`-0 * 1`) should keep its sign bit.
   - Large and imprecise values: `8.0E19` above, and `sum()` over many decimal 
strings, which uses double
     arithmetic (`1.0 + 2.1` gives `3.1` in the suite).
   - `NULL` or `''` for either argument gives NULL. A non-literal `path` gives 
`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`.
     Malformed XML and an invalid path raise runtime errors (see the `xpath` 
issue). ANSI mode has no effect.
   - `xpath_number` must return the same values and show `xpath_number` as its 
display name.
   
   ### Differences across Spark versions
   
   None found beyond the shared ones in the `xpath` issue.
   
   ### Implementation notes
   
   - Spark: `XPathDouble` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala`.
     It calls `XPathDoubleEvaluator` (`.../xml/XmlExpressionEvalUtils.scala`), 
which calls
     `UDFXPathUtil.evalNumber`. The `xpath_number` alias is in 
`FunctionRegistry.scala`.
   - DataFusion core has no equivalent.
   - Build it on the shared evaluator from the `xpath` issue in 
`crates/spark/src/function/xml/`. A single
     `evaluate_number(xml, path) -> Option<f64>` helper can serve 
`xpath_double`, `xpath_float`, `xpath_int`,
     `xpath_long` and `xpath_short`. Register `xpath_number` as an alias.
   - Tests: `crates/sqllogictest/test_files/xml/xpath_double.slt`, which also 
covers `xpath_number`. Document
     the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to