David Mollitor created SPARK-59961:
--------------------------------------
Summary: Cache the compiled regex in StringSplit
Key: SPARK-59961
URL: https://issues.apache.org/jira/browse/SPARK-59961
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: David Mollitor
h3. Problem
{{StringSplit}} (the {{split}} function) compiles its regex on every row. Both
the interpreted and the codegen path call {{UTF8String.split(UTF8String,
int)}}, which decodes the regex to a Java String and calls {{String.split}}.
{{String.split}} runs {{Pattern.compile(regex)}} on every call unless the regex
is a single literal character, which it handles with an {{indexOf}} fast path.
So each of these compiles a new {{Pattern}} for every row, even though the
regex is a literal:
{code:sql}
SELECT split(s, '[,;]') FROM t;
SELECT split(s, '\\s+') FROM t;
SELECT split(s, '|') FROM t;
SELECT split(s COLLATE UTF8_LCASE, ',') FROM t; -- the regex gets a (?ui)
prefix
{code}
The other regex expressions in the same file ({{RegExpReplace}},
{{RegExpExtractBase}}) already keep the last regex and its compiled
{{Pattern}}. {{StringSplit}} never got that cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
