andygrove opened a new pull request, #328:
URL: https://github.com/apache/datafusion-spark/pull/328

   Closes #135.
   
   ## Summary
   
   Adds a Spark-compatible `contains(left, right)` UDF, ported from DataFusion 
Comet (`native/spark-expr/src/string_funcs/contains.rs`).
   
   - Returns true if `right` occurs in `left`. The match is case-sensitive. An 
empty `right` is found in every value, including `''`. Returns NULL if either 
input is NULL.
   - Accepts `Utf8`, `LargeUtf8`, `Utf8View`, `Binary`, `LargeBinary`, 
`BinaryView` (plus `FixedSizeBinary` and dictionaries, which are coerced to 
their value type).
   - `coerce_types` gives both arguments the same type, because Arrow's 
`contains` kernel needs that:
     - When both arguments are strings, the wider type wins (LargeUtf8 > 
Utf8View > Utf8).
     - When either argument is binary, both are compared as binary. Spark 
matches bytes when both are binary. With one string and one binary it casts the 
binary to a string, but a `UTF8_BINARY` string compares as bytes, so the answer 
is the same. Comparing as binary also means bytes that are not valid UTF-8 
cannot make the cast fail.
     - Integer, boolean and date arguments are cast to `Utf8`, as Spark's 
implicit cast does. In these cases Arrow prints the value the same way Spark 
does.
     - A NULL literal is coerced to the other side's type.
   - `invoke_with_args` handles all four array/scalar combinations directly. A 
scalar side stays an Arrow `Scalar`, so the kernel's scalar fast path is used. 
With two scalars the result is a scalar, whatever `number_rows` is.
   - Documented with `#[user_doc]` and registered in `function/string/mod.rs` 
(including `expr_fn::contains`).
   
   Spark's `startswith`/`endswith` share Spark's builder, but Comet's file only 
covers `contains`, so they are not part of this PR.
   
   ## Differences from Comet's version
   
   - **Binary support.** Comet's scalar paths accept only string scalars. A 
binary scalar fails with "requires string type", so `contains(binary_col, 
x'..')` errors. This version supports binary in every combination, including a 
string paired with binary.
   - **Type coercion.** Comet uses `Signature::variadic_any` and has no 
`coerce_types`, so it relies on Comet's planner to send matching types. This 
version uses a `user_defined` signature with `coerce_types`, which also checks 
the argument count and rejects types it does not support at plan time.
   - **Scalar-scalar.** Comet uses `str::contains` on two string scalars. This 
version runs the Arrow kernel on two length-1 scalars, so the same code handles 
strings and binary.
   - **Errors.** This version uses `take_function_args` for the argument count. 
An unsupported type gives an `Unsupported Data Type: Spark \`contains\` 
function expects STRING or BINARY arguments` error.
   - Comet's unit tests are ported. They now go through `invoke_with_args`, and 
new tests cover array-array, scalar-scalar with `number_rows > 1`, the binary 
variants and `coerce_types`.
   
   ### Known gaps compared with Spark
   
   - Spark implicitly casts any atomic type to a string. This version rejects 
floating-point, decimal, timestamp and other types at plan time rather than 
cast them, because Arrow's string form can differ from Spark's. For example, 
Spark renders `1.0E10` and Arrow renders `10000000000`.
   - Spark 4 collation-aware `contains` is out of scope; it belongs to the 
collation epic.
   
   ## Testing
   
   - The expected values in 
`crates/sqllogictest/test_files/string/contains.slt` come from PySpark 4.2.0 
(ANSI on, which is the default). I ran ad-hoc queries and the PySpark SLT 
runner: `pyspark_slt_runner.py 
crates/sqllogictest/test_files/string/contains.slt` returned 21 of 21 records 
passing and none skipped. The `arrow_cast` records were translated to Spark 
types and passed. Cases covered: the doc examples, NULLs on either or both 
sides, empty strings or binary, case sensitivity, multi-byte characters, binary 
with invalid UTF-8 bytes, mixed string/binary, integer/boolean/date inputs, 
column/literal combinations, the LargeUtf8/Utf8View/LargeBinary/BinaryView 
variants, and the wrong-arity and wrong-type errors.
   - `cargo fmt --all`: clean
   - `cargo clippy --all-targets --all-features -- -D warnings`: clean
   - `cargo test -p datafusion-spark`: 350 passed
   - `cargo test -p datafusion-spark-sqllogictest`: 249 of 249 files passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to