jonasdedden opened a new pull request, #50989:
URL: https://github.com/apache/arrow/pull/50989
### Rationale for this change
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping,
so any expression using them fails to serialize:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
# ArrowNotImplementedError: No conversion function exists to convert the
# Arrow function starts_with to a Substrait call
```
Comparisons, `isin` and arithmetic serialize fine, so this is a per-function
gap. It matters for engines that ingest a PyArrow filter through Substrait,
where an unmappable function becomes a hard failure rather than a fallback.
See #50988.
### What changes are included in this PR?
Map the three kernels onto `starts_with`, `ends_with` and `contains` from
Substrait's `functions_string.yaml`, in both directions.
The signatures do not line up. Substrait passes the pattern as a second
argument, while the Arrow kernels are unary and carry it in
`MatchSubstringOptions`. So:
* encoding lifts `MatchSubstringOptions::pattern` out into a literal argument
* decoding requires that argument to be a non-null string literal, and
returns `NotImplemented` otherwise
Substrait's `case_sensitivity` option maps onto `ignore_case`.
`CASE_INSENSITIVE_ASCII` has no Arrow equivalent and returns `NotImplemented`.
Both caveats are documented in `docs/source/cpp/acero/substrait.rst`.
### Are these changes tested?
Yes.
`Substrait.StringMatchExpressionSerialization` round-trips all three
functions with and without `ignore_case`, plus a nested
`invert(starts_with(...))`. `Substrait.StringMatchExpressionDeserialization`
drives hand-written Substrait JSON to cover the default (no option) case and
the two rejection paths.
Both pass locally. The rest of `arrow-substrait-substrait-test` is
unaffected (the only failures in my run were read-relation tests that need the
`parquet-testing` submodule, which I had not checked out).
`test_serializing_string_match_expressions` covers the PyArrow path. I was
not able to build pyarrow locally, so that one has not run outside CI. Its
assertion compares `str()` of the deserialized expression against a normalized
form, following `test_serializing_multiple_expressions`; I checked that
comparison holds for all seven parametrized cases by replicating it in C++
against the built libraries.
### Are there any user-facing changes?
Yes. `pc.starts_with`, `pc.ends_with` and `pc.match_substring` can now be
serialized to Substrait and consumed back, and Acero can consume plans using
Substrait's `starts_with`, `ends_with` and `contains`. No existing behaviour
changes: these previously raised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
