David Mollitor created SPARK-59968:
--------------------------------------
Summary: Reuse the regex Matcher across rows in LIKE and RLIKE
codegen
Key: SPARK-59968
URL: https://issues.apache.org/jira/browse/SPARK-59968
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: David Mollitor
h3. Problem
When the pattern of {{RLIKE}} or {{LIKE}} is foldable, the generated code
compiles the {{java.util.regex.Pattern}} once and keeps it in mutable state.
However, it still calls {{pattern.matcher(input)}} on every row:
{code:java}
value = patternRLike.matcher(input.toString()).find(0);
{code}
Each call allocates a new {{java.util.regex.Matcher}} along with its internal
{{{}groups{}}}, {{locals}} and {{localsPos}} arrays. The JIT cannot
scalar-replace this allocation because the Matcher escapes into the regex
engine's recursive {{Node.match}} graph. Allocation profiles of a filter using
a token-boundary {{RLIKE}} show {{Matcher.<init>}} <- {{Pattern.matcher}} <-
generated {{processNext}} at roughly 2-5% of allocated bytes.
{{LIKE}} has the same shape (using {{{}matches(){}}}) for patterns that
{{LikeSimplification}} does not rewrite into {{StartsWith}} / {{EndsWith}} /
{{{}Contains{}}}, such as {{{}a%b%c{}}}.
h3. Proposal
In the foldable branch of {{RLike.doGenCode}} and {{{}Like.doGenCode{}}}, build
the {{Matcher}} once in the mutable state's init code, and reset it per row
instead of allocating a new one:
{code:java}
// init
matcherRLike = java.util.regex.Pattern.compile("<regex>", <flags>).matcher("");
// per row
value = matcherRLike.reset(input.toString()).find(0);
{code}
* The Matcher replaces the Pattern field (the Matcher holds its Pattern), so
the number of mutable states does not change.
* Codegen mutable state is per generated-object instance, so the Matcher is
never shared across threads.
* Results are unchanged; {{find(0)}} / {{matches()}} keep the same semantics
as the interpreted path.
Out of scope: the interpreted path (RLike/Like are not {{{}stateful{}}}),
non-foldable patterns, LIKE ANY / LIKE ALL, and the per-row
{{UTF8String.toString()}} conversion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
