[
https://issues.apache.org/jira/browse/SPARK-59961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59961:
-----------------------------------
Labels: pull-request-available (was: )
> Cache the compiled regex in StringSplit
> ---------------------------------------
>
> Key: SPARK-59961
> URL: https://issues.apache.org/jira/browse/SPARK-59961
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem
> {{StringSplit}} (the {{split}} function) compiles its regex on every row.
> Both the interpreted and the codegen path call {{UTF8String.split(UTF8String,
> int)}}, which decodes the regex to a Java String and calls {{String.split}}.
> {{String.split}} runs {{Pattern.compile(regex)}} on every call unless the
> regex is a single literal character, which it handles with an {{indexOf}}
> fast path.
> So each of these compiles a new {{Pattern}} for every row, even though the
> regex is a literal:
> {code:sql}
> SELECT split(s, '[,;]') FROM t;
> SELECT split(s, '\\s+') FROM t;
> SELECT split(s, '|') FROM t;
> SELECT split(s COLLATE UTF8_LCASE, ',') FROM t; -- the regex gets a (?ui)
> prefix
> {code}
> The other regex expressions in the same file ({{RegExpReplace}},
> {{RegExpExtractBase}}) already keep the last regex and its compiled
> {{Pattern}}. {{StringSplit}} never got that cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
