Max Gekk created SPARK-57758:
--------------------------------
Summary: Built-in function resolution is no longer O(1) after
SPARK-54807, regressing Spark Connect AnalyzePlan
Key: SPARK-57758
URL: https://issues.apache.org/jira/browse/SPARK-57758
Project: Spark
Issue Type: Bug
Components: Connect, SQL
Affects Versions: 4.2.0
Reporter: Max Gekk
SPARK-54807 (#53570) added qualified function names and a configurable
resolution search path ({{spark.sql.functionResolution.sessionOrder}}). As a
side effect it changed how
*unqualified* function names are resolved, introducing a performance
regression in the analyzer.
Previously, an unqualified built-in function (e.g. {{count}}, {{coalesce}},
{{sum}}) resolved with a single in-memory registry lookup. After SPARK-54807,
{{FunctionResolution.resolveFunction}} (and {{resolveTableFunction}}) build
an ordered candidate search path for *every* {{UnresolvedFunction}}. For each
function node, per call,
it now:
* reads the {{AnalysisContext}} thread-local and {{CatalogManager}}
({{currentCatalogPath}}),
* reads the {{spark.sql.functionResolution.sessionOrder}} conf and allocates
the search-path {{Seq}}s ({{resolutionSearchPath}}),
* allocates the candidate list ({{searchPath.map(_ ++ nameParts)}}), and
* iterates candidates, each doing a name-kind parse plus a registry lookup.
None of this is memoized across an analysis pass, so it is recomputed for
every function node.
For plans with many built-in function references this adds substantial
per-function overhead. The impact is amplified under Spark Connect, which
re-analyzes the entire (growing)
plan on every {{AnalyzePlan}} call: the per-function overhead is paid
repeatedly, scaling roughly with plan size x number of analyze calls, and
produces a multi-fold regression
in analysis time. Execution time is unaffected -- the regression is isolated
to the analysis phase.
*Reproduction:* a Spark Connect session that incrementally builds a wide plan
containing many built-in function calls, comparing {{AnalyzePlan}} latency
against a pre-SPARK-54807
build.
h3. Proposed fix
# *Built-in fast-path for single-part names.* In
{{resolveFunction}}/{{resolveTableFunction}}, before constructing the candidate
search path, resolve a single-part name directly
against the in-memory built-in/temp registry
({{v1SessionCatalog.resolveBuiltinOrTempFunction}}) and return immediately on a
hit. {{system.builtin}} is the first candidate in
*all* {{sessionOrder}} modes ({{first}}/{{second}}/{{last}}), so a
*built-in-only* fast-path cannot change resolution precedence.
# *Do not fast-path session/temporary functions.* Under {{sessionOrder=last}}
a persistent function must shadow a session function, so session/temp
resolution must remain in the
ordered search path. Only built-ins are safe to short-circuit.
# *Memoize per pass.* Cache {{currentCatalogPath}} and the computed
{{resolutionSearchPath}} for the duration of one resolution pass (they do not
change mid-pass), eliminating
the repeated thread-local reads and {{Seq}} allocations even for the
temp/persistent fall-through.
# *Minor correctness check (optional).*
{{resolveQualifiedFunction}}/{{resolveQualifiedTableFunction}} catch
{{AnalysisException}} with condition {{FORBIDDEN_OPERATION}} and
return {{None}}, which can surface a genuine permission error as an
"unresolved routine" error. Worth confirming this is intended.
This restores the previous fast resolution for built-in-heavy plans (the
dominant case) while preserving the full qualified-name and configurable-order
semantics SPARK-54807
introduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
