adriangb commented on issue #11083:
URL: https://github.com/apache/arrow-rs/issues/11083#issuecomment-5689869638
A note on an adjacent cost in the same module, in case it is useful context
for this work. It is not a request to widen the scope of this issue.
The scalar path avoids the per-row work described here, but it compiles the
pattern once **per call**. A caller that applies one literal pattern to a long
scan calls the kernel once per batch, so the same pattern is compiled again for
every batch.
I measured the `regex` crate directly (release build, arm64 macOS, 8192 rows
of ~60 characters, 2000 compiles per pattern), to compare compilation with
matching for one batch:
| pattern | compile | match 8192 rows | compile share |
|---|---:|---:|---:|
| `abc` | 1.55 µs | 90.6 µs | 1.7% |
| `^[a-z]+@[a-z]+\.[a-z]{2,}$` | 6.58 µs | 98.9 µs | 6.2% |
| `(?i)error\|warn(ing)?` | 62.0 µs | 232.5 µs | 21.1% |
| `\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?Z?` | 249.5 µs | 103.7 µs |
70.7% |
For the last pattern, compilation is more expensive than the matching it
enables.
The interesting part is that `arrow-string` already contains the object that
solves this. `Predicate` in `predicate.rs` analyses a pattern once, holds the
compiled `Regex`, and evaluates a whole array (`evaluate_array`). The LIKE
family uses it. The `regexp` module has no equivalent, and `Predicate` is
`pub(crate)`.
A public prepared type for the regexp kernels would let a caller compile
once and reuse across batches:
```rust
pub struct RegexPredicate { /* opaque */ }
impl RegexPredicate {
pub fn try_new(pattern: &str, flags: Option<&str>) -> Result<Self,
ArrowError>;
pub fn is_match(&self, array: &dyn Array) -> Result<BooleanArray,
ArrowError>;
pub fn matches(&self, array: &dyn Array) -> Result<ArrayRef, ArrowError>;
}
```
`regexp_is_match_scalar` and `regexp_match` then become thin wrappers, so
existing callers do not change. `regex` stays a private dependency, because the
compiled value is hidden inside an arrow type. `RowConverter` is a precedent
for a public, reusable object of this kind.
It also improves error reporting: the compile failure occurs in `try_new`,
at a time the caller chooses, so a caller with a literal pattern can report an
invalid pattern before execution starts instead of during it.
Happy to open a separate issue for that if it is wanted, and to keep it out
of the way of this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
