andygrove opened a new issue, #288:
URL: https://github.com/apache/datafusion-spark/issues/288
Part of #271 and apache/datafusion-comet#7300.
### What the function does
`xpath_short(xml, path)` evaluates an XPath 1.0 expression, converts the
result with XPath `number()` into a
Java `Double`, and narrows it with `Double.shortValue()`, giving a
`SMALLINT`. Both arguments are strings
(collation-aware from 4.0), and `path` must be foldable. The result is
always nullable. Available since Spark
2.0.0.
The `number()` conversion is in the `xpath_double` issue. Parsing, DTD and
entity handling, namespaces, NULL
handling and errors are in the `xpath` issue. This issue covers only the
double-to-short step.
The narrowing is Java's `(short) d`, which does not saturate. JLS 5.1.3
first converts the double to `int`,
with saturation and `NaN` becoming 0, and then keeps the low 16 bits. In
Rust this is `(d as i32) as i16`, not
`d as i16`, which would saturate at `i16::MAX`.
### Examples
```sql
SELECT xpath_short('<a><b>1</b><b>2</b></a>', 'sum(a/b)');
-- 3
SELECT xpath_short('<a>this is not a number</a>', 'a');
-- 0
SELECT xpath_short('<a><b>2000000000</b><c>40000000000</c></a>', 'a/b *
a/c');
-- -1
SELECT xpath_short('<a>try a boolean</a>', 'a = 10');
-- 0
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b
class="odd">4</b><c>8</c></a>', 'a/b');
-- 1
SELECT xpath_short('<a><b class="odd">1</b><b class="even">2</b><b
class="odd">4</b><c>8</c></a>', 'sum(a/*)');
-- 15
SELECT xpath_short('<a><b class="odd">10000</b><b class="even">2</b><b
class="odd">4</b><c>8</c></a>', 'sum(a/b[@class="odd"])');
-- 10004
```
These come from the `ExpressionDescription`, the Hive `udf_xpath_short`
golden files run by
`HiveCompatibilitySuite`, and `XPathExpressionSuite`. The `-1` comes from
`(short) (int) 8.0E19 = (short) 2147483647 = -1`.
### Edge cases to test
- Wrap-around, by code reading:
- `<a>40000</a>` gives `-25536`;
- `<a>32768</a>` gives `-32768`;
- `<a>65536</a>` gives `0`;
- `-1 div 0` gives `(short) Integer.MIN_VALUE`, which is `0`;
- `1 div 0` gives `-1`.
- Truncation toward zero before wrapping: `<a>-2.7</a>` gives `-2`.
- `NaN` gives `0`. This covers non-numeric text, `0 div 0`, and the empty
node-set, which is `NaN` in
XPath 1.0.
- `NULL` or `''` arguments give NULL. A non-literal path gives
`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`.
Malformed XML and an invalid path raise runtime errors (see the `xpath`
issue).
### Differences across Spark versions
None found beyond the shared ones in the `xpath` issue.
### Implementation notes
- Spark: `XPathShort` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xml/xpath.scala`.
It calls `XPathShortEvaluator` (`ret.shortValue()`) in
`.../xml/XmlExpressionEvalUtils.scala`.
- DataFusion core has no equivalent.
- Reuse the shared `number()` evaluator from the `xpath` and `xpath_double`
issues in
`crates/spark/src/function/xml/`, and narrow with `(d as i32) as i16`.
- Tests: add `crates/sqllogictest/test_files/xml/xpath_short.slt` and
document the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]