jonasdedden opened a new issue, #50988:
URL: https://github.com/apache/arrow/issues/50988
### Describe the bug, including details regarding any error messages,
version, and platform.
# Describe the enhancement requested
`starts_with`, `ends_with` and `match_substring` have no Substrait mapping,
so expressions using them cannot be serialized.
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)
```
```
pyarrow.lib.ArrowNotImplementedError: No conversion function exists to
convert
the Arrow function starts_with to a Substrait call
```
Same for `pc.ends_with` and `pc.match_substring`. Comparisons, `isin` and
arithmetic all serialize fine, so this looks like a per-function gap rather
than a structural one.
Substrait defines `starts_with`, `ends_with` and `contains` in
`functions_string.yaml`, which cover all three.
The signatures differ: the Substrait functions take the pattern as a second
argument, while the Arrow kernels are unary and carry it in
`MatchSubstringOptions`. So the mapping needs the second argument to be a
string literal. Substrait's `case_sensitivity` option maps onto `ignore_case`,
except for `CASE_INSENSITIVE_ASCII` which has no Arrow equivalent.
This matters for engines that ingest a PyArrow filter through Substrait,
where an unmappable function turns into a hard failure rather than a fallback.
# Component(s)
C++, Python
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
