James Xu created SPARK-60844:
--------------------------------
Summary: Rewrite literal RLIKE alternations into string
containment checks
Key: SPARK-60844
URL: https://issues.apache.org/jira/browse/SPARK-60844
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: James Xu
Problem:
Spark evaluates RLIKE with java.util.regex.Matcher.find. A pattern that is only
an alternation of literal substrings, often wrapped in redundant .* or a plain
outer group, is a containment test. The regex engine still walks each
alternative on every row.
A production scan of about 8 billion rows filters with this shape. The pattern
is a membership test over a fixed set of literals, and the per-row regex walk
dominates the scan.
SELECT request_id
FROM serving_log
WHERE prompt RLIKE '.*(alpha|beta|gamma).*'
LikeSimplification already rewrites LIKE '%alpha%' to a containment check.
RLIKE has no counterpart, so the same literal shape stays on the regex path.
Root Cause:
RLike.matches compiles the pattern and calls Matcher.find(0). Under find(),
.*alpha.*, alpha.*, .*alpha, and (alpha|beta) match exactly the strings that
contain one of those literals. Anchors, escapes, empty alternatives, and other
metacharacters do not have that equivalence, and LikeSimplification does not
visit RLike.
Proposal:
Add an optimizer rule, disabled by default, that rewrites an eligible constant
RLIKE into a balanced OR of containment checks. The input collation must be
UTF8_BINARY. After stripping redundant leading and trailing .* and plain outer
groups, every alternative must be a non-empty literal with no regex
metacharacter. A non-deterministic input is not duplicated across alternatives.
Anchored patterns, escapes, empty alternatives, and non-binary collations stay
on the regex path.
The rule is gated by spark.sql.optimizer.rlikeSimplification.enabled (default
false), with ConfigBindingPolicy.NOT_APPLICABLE and version 4.4.0. Disabled
mode leaves the plan unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
