[
https://issues.apache.org/jira/browse/SPARK-60844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60844:
-----------------------------------
Labels: pull-request-available (was: )
> Rewrite literal RLIKE alternations into string containment checks
> -----------------------------------------------------------------
>
> Key: SPARK-60844
> URL: https://issues.apache.org/jira/browse/SPARK-60844
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: James Xu
> Priority: Major
> Labels: pull-request-available
>
> Problem:
> Spark evaluates RLIKE with java.util.regex.Matcher.find. A pattern that is
> only an alternation of literal substrings, often wrapped in redundant .* or a
> plain outer group, is a containment test. The regex engine still walks each
> alternative on every row.
> A production scan of about 8 billion rows filters with this shape. The
> pattern is a membership test over a fixed set of literals, and the per-row
> regex walk dominates the scan.
> SELECT request_id
> FROM serving_log
> WHERE prompt RLIKE '.*(alpha|beta|gamma).*'
> LikeSimplification already rewrites LIKE '%alpha%' to a containment check.
> RLIKE has no counterpart, so the same literal shape stays on the regex path.
> Root Cause:
> RLike.matches compiles the pattern and calls Matcher.find(0). Under find(),
> .*alpha.*, alpha.*, .*alpha, and (alpha|beta) match exactly the strings that
> contain one of those literals. Anchors, escapes, empty alternatives, and
> other metacharacters do not have that equivalence, and LikeSimplification
> does not visit RLike.
> Proposal:
> Add an optimizer rule, disabled by default, that rewrites an eligible
> constant RLIKE into a balanced OR of containment checks. The input collation
> must be UTF8_BINARY. After stripping redundant leading and trailing .* and
> plain outer groups, every alternative must be a non-empty literal with no
> regex metacharacter. A non-deterministic input is not duplicated across
> alternatives. Anchored patterns, escapes, empty alternatives, and non-binary
> collations stay on the regex path.
> The rule is gated by spark.sql.optimizer.rlikeSimplification.enabled (default
> false), with ConfigBindingPolicy.NOT_APPLICABLE and version 4.4.0. Disabled
> mode leaves the plan unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
