david-mollitor-db opened a new pull request, #58663:
URL: https://github.com/apache/spark/pull/58663
### What changes were proposed in this pull request?
`LikeSimplification` rewrites a `LIKE 'prefix%suffix'` pattern (the
`startsAndEndsWith`
shape, e.g. `'a%b'`) into
```
lengthGuard(input) && StartsWith(input, prefix) && EndsWith(input, postfix)
```
which references the child `input` three times. The single-`Like` branch of
`LikeSimplification.apply` applied this rewrite unconditionally, unlike the
`LikeAll`/`NotLikeAll`/`LikeAny`/`NotLikeAny` branches, which only fire when
`CollapseProject.isCheap(child)` (SPARK-40228).
This PR gates the `startsAndEndsWith` case on
`CollapseProject.isCheap(input)`. For a
non-cheap child the pattern is left as a plain `Like`. The single-reference
shapes
(`startsWith`, `endsWith`, `contains`, `equalTo`) reference the child once
and remain
enabled for any child.
### Why are the changes needed?
Duplicating a non-cheap child is both a correctness and a performance
problem:
- **Correctness** — a nondeterministic child (e.g. `uuid()`, `cast(rand() as
string)`) is
evaluated independently for each reference, so the three copies can
produce different
values and the rewritten predicate no longer matches the semantics of the
original
`LIKE`. Subexpression elimination does not help: it deliberately never
deduplicates
nondeterministic expressions.
- **Performance** — an expensive deterministic child (e.g. `sha2(col)`) is
written into
the plan three times; the logical plan should not rely on subexpression
elimination
collapsing the repeats during codegen.
This is the same duplication class that SPARK-40228 fixed for the
multi-`LIKE` rules.
### Does this PR introduce _any_ user-facing change?
Yes. For a `LIKE 'prefix%suffix'` pattern over a nondeterministic child, the
child is now
evaluated once instead of once per reference. For example, `uuid() LIKE
'a%b'`:
- **Before** — rewritten to
`length(uuid()) >= 2 AND startswith(uuid(), 'a') AND endswith(uuid(),
'b')`, drawing
three independent UUIDs and testing the length, prefix, and suffix of
different strings.
- **After** — left as `uuid() LIKE 'a%b'`, drawing a single UUID and
matching it against
the whole pattern.
The new behavior matches evaluating the `LIKE` directly (a single evaluation
of the
child), which is what users expect. Queries with a nondeterministic argument
to such a
`LIKE` can return different rows than before. Because the
`startsAndEndsWith` rewrite is
long-standing, this is a user-facing change relative to released Spark
versions as well as
`master`. Behavior is unchanged for cheap children (attributes, foldables,
etc.), the
common case.
### How was this patch tested?
Added two unit tests to `LikeSimplificationSuite`:
- `SPARK-59371: do not simplify startsAndEndsWith LIKE for a non-cheap
child` — asserts
`$"a".substring(1, 5) like "a%b"` is left as `Like` (fails before this
change).
- `SPARK-59371: still simplify single-reference LIKE shapes for a non-cheap
child` —
asserts the `startsWith`/`endsWith`/`contains`/`equalTo` shapes still
simplify for a
non-cheap child (guards against over-gating).
`build/sbt 'catalyst/testOnly *LikeSimplificationSuite'` passes (22 tests);
scalastyle
clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
