andygrove opened a new issue, #219:
URL: https://github.com/apache/datafusion-spark/issues/219

   Part of #178 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `reverse(expr)` reverses its input, and the result has the same type as the 
input:
   - A STRING is reversed character by character, by Unicode code point.
   - An ARRAY has its elements reversed. `containsNull` is unchanged, and NULL 
elements are kept in their mirrored
     positions.
   - A BINARY value (Spark 4.2.0+) has its bytes reversed.
   
   The input types are 
`TypeCollection(StringTypeWithCollation(supportsTrimCollation = true), 
BinaryType, ArrayType)`
   with `ImplicitCastInputTypes`. Any other atomic type is implicitly cast to 
STRING first, so
   `reverse(3215)` returns the string `'5123'`. Struct and map inputs fail 
analysis with
   `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (requiredType `("STRING" or 
"BINARY" or "ARRAY")` in 4.2). The
   expression is null-intolerant, so a NULL input gives NULL.
   
   The string form exists since Spark 1.5.0, the array form since 2.4.0, and 
the binary form since 4.2.0. There are
   no aliases. The function is a single overloaded name in Spark, so the 
datafusion-spark UDF must accept all three
   input kinds.
   
   ### Examples
   
   ```sql
   SELECT reverse('Spark SQL');
   -- LQS krapS
   SELECT reverse(array(2, 1, 4, 3));
   -- [3,4,1,2]
   SELECT hex(reverse(x'CAFE'));
   -- FECA          (Spark 4.2.0+)
   SELECT reverse(3215);
   -- 5123
   SELECT reverse(null);
   -- NULL
   SELECT reverse(array(array(1, 2), array(3, 4)));
   -- [[3,4],[1,2]]
   SELECT reverse(struct(1, 'a'));
   -- error DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE
   ```
   
   Spark's test suites also cover these array cases 
(`CollectionExpressionsSuite`):
   - `[1, null, 2, null]` becomes `[null, 2, null, 1]`.
   - `[null, null, null]` stays the same.
   - `[]` stays `[]`.
   
   ### Edge cases to test
   
   - NULL string, binary and array inputs, plus an empty string, an empty 
binary and an empty array.
   - Unicode:
     - Spark's `UTF8String.reverse` copies whole UTF-8 sequences (code points), 
so multi-byte characters and
       4-byte characters (emoji, supplementary-plane CJK) stay intact.
     - Grapheme clusters are not preserved: a base letter followed by a 
combining mark comes out with the mark
       before the letter, and ZWJ emoji sequences are split.
     - Test a decomposed `e` + U+0301 and a flag emoji.
   - Binary input, from `DataFrameFunctionsSuite` / 
`CollectionExpressionsSuite`:
     - `x'CAFE'` becomes `x'FECA'`, `x''` stays `x''`, and `x'01'` stays 
`x'01'`.
     - `x'E687'` becomes `x'87E6'`. That input is a valid UTF-8 prefix, but the 
bytes are reversed raw.
     - The result type must stay BINARY.
   - Arrays:
     - NULL elements, nested arrays (only the outer level is reversed), and 
struct, map and binary elements.
     - `containsNull = false` arrays.
   - Non-string scalar inputs (INT, DOUBLE, DATE, TIMESTAMP, DECIMAL) are cast 
to STRING with Spark's cast
     formatting before reversing. The cast output depends on the session time 
zone for TIMESTAMP.
   - Collations (4.0+): the result keeps the input's collation, and a trim 
collation is accepted. The reversal
     itself ignores the collation.
   - Scalar and column inputs, including sliced arrays.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5: the input types are `TypeCollection(StringType, ArrayType)`. 
BINARY is not a member, so a BINARY
     argument is implicitly cast to STRING (both TypeCoercion and 
AnsiTypeCoercion allow atomic-to-string), and the
     result is a STRING. The binary bytes are interpreted as UTF-8 and reversed 
by sequence.
   - 4.0 and 4.1: the same, plus collation-aware string input types.
   - 4.2: `BinaryType` is added to the input types, so `reverse(binary)` 
returns BINARY with the bytes reversed.
     Engines that target several Spark versions need a way to choose between 
these behaviours, or the planner
     must insert the cast for older versions.
   
   ### Implementation notes
   
   - Spark implements this in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
     (`Reverse`). The string path is `UTF8String.reverse()` in
     
`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`.
   - DataFusion core splits this into two functions:
     - `reverse` (`datafusion/functions/src/unicode/reverse.rs`) reverses by 
`char` (code point), which matches
       Spark for valid UTF-8. It only handles string types and implicitly 
coerces any input to string, so arrays
       and binary are not supported.
     - `array_reverse` / `list_reverse` 
(`datafusion/functions-nested/src/reverse.rs`) handles `List`,
       `LargeList`, `FixedSizeList` and `ListView`, keeps the type, and matches 
Spark's element semantics.
   - The datafusion-spark UDF can dispatch on the input type:
     - strings go to DataFusion's string `reverse`,
     - lists go to `array_reverse`,
     - binary needs a new byte-reverse kernel for `Binary`, `LargeBinary` and 
`BinaryView`.
     - In `coerce_types`, cast other scalar types to `Utf8` the way Spark does, 
using Spark cast formatting
       (for example `2.0` for DOUBLE).
   - There is already a placeholder 
`crates/sqllogictest/test_files/collection/reverse.slt`, ported from Sail with
     its queries commented out. Turn it into real tests, put the UDF in 
`crates/spark/src/function/collection/`
     (next to `size`), and document it with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to