David Mollitor created SPARK-59371:
--------------------------------------
Summary: LikeSimplification duplicates a non-cheap child in the
`prefix%suffix` (startsAndEndsWith) rewrite
Key: SPARK-59371
URL: https://issues.apache.org/jira/browse/SPARK-59371
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Problem
{{LikeSimplification}} rewrites a {{LIKE 'prefix%suffix'}} pattern – the
{{startsAndEndsWith}} shape, e.g. {{'a%b'}} – into a length guard plus
{{StartsWith}}
plus {{{}EndsWith{}}}:
{code:none}
lengthGuard(input) && StartsWith(input, prefix) && EndsWith(input, postfix)
{code}
This references {{input}} three times. The single-{{{}Like{}}} branch of
{{LikeSimplification.apply}} applies the rewrite unconditionally, unlike the
{{{}LikeAll{}}}/{{{}NotLikeAll{}}}/{{{}LikeAny{}}}/{{{}NotLikeAny{}}} branches,
which since SPARK-40228
only fire when {{{}CollapseProject.isCheap(child){}}}. So a non-cheap child
gets duplicated:
* *Correctness* – a nondeterministic child (e.g. {{{}uuid(){}}},
{{{}cast(rand() as string){}}}) is evaluated independently for each reference,
so the three copies can produce different values and the rewritten predicate no
longer matches the semantics of the original {{{}LIKE{}}}. Subexpression
elimination does not help: it deliberately never deduplicates nondeterministic
expressions.
* *Performance* – an expensive deterministic child (e.g. {{{}sha2(col){}}}) is
written into
the plan three times. Subexpression elimination usually collapses the repeats
during codegen, but that is a physical optimization the logical plan should not
rely on.
The single-reference shapes ({{{}startsWith{}}}, {{{}endsWith{}}},
{{{}contains{}}}, {{{}equalTo{}}})
reference {{input}} once and are not affected.
h2. Example
{code:sql}
-- uuid() is nondeterministic and returns a string
SELECT * FROM t WHERE uuid() LIKE 'a%b'
{code}
is rewritten to roughly
{code:sql}
SELECT * FROM t
WHERE length(uuid()) >= 2 AND startswith(uuid(), 'a') AND endswith(uuid(), 'b')
{code}
where each {{uuid()}} call draws a different value, so the length, prefix, and
suffix
checks are applied to three different strings – a result that can differ from
the
original single {{{}LIKE{}}}.
h2. Fix
Gate the {{startsAndEndsWith}} case on {{{}CollapseProject.isCheap(input){}}}.
For a non-cheap child the pattern is left as a plain {{{}Like{}}}. The
single-reference shapes evaluate the child exactly once, just like the original
{{{}LIKE{}}}, so they stay enabled for any child.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
