Josh Rosen created SPARK-58507:
----------------------------------
Summary: ClosureCleaner's indylambda path repeats expensive work
on every clean() call
Key: SPARK-58507
URL: https://issues.apache.org/jira/browse/SPARK-58507
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Josh Rosen
ClosureCleaner's indylambda path repeats expensive work on every clean() call:
1. Non-capturing closures are fully analyzed for nothing. SparkContext.runJob
cleans every closure unconditionally; for a closure that captures nothing (e.g.
the "iter => iter.toArray" that RDD.collect passes to runJob) there is nothing
to clean, yet the cleaner still loads the capturing class and runs a full ASM
parse of its bytecode before discarding the result. ~15% of indylambda cleans
in certain CI test workloads are non-capturing.
2. The return-statement fail-fast re-parses the same bytecode over and over.
The verdict of ReturnStatementFinder is a pure function of the capturing
class's immutable bytecode, and capturing classes repeat heavily across jobs
because they are mostly Spark's own classes (SparkContext, RDD, Dataset,
WholeStageCodegenExec): a DataFrameSuite run performed 3,251 full ASM parses
over just 21 distinct classes (155x repeat ratio); an RDDSuite run, 1,306 over
11 (119x).
Measured cost (JFR profiling):
- Cleaning-related frames appear in 9.8-13.2% of test-JVM CPU samples across
six suites, including core-only RDDSuite.
- Per-call parse cost scales with the capturing class's file size: ~46 us for
a 9 KB class, ~340 us for a 113 KB one; RDD.class and SparkContext.class are
~190-205 KB.
- In a loop of minimal collect()/count() jobs, roughly half of per-job driver
time goes to re-parsing SparkContext.class and RDD.class (~5 parses per job).
This overhead is paid on every job submission, so short interactive queries and
streaming microbatches see it most.
Proposed improvement:
- Check SerializedLambda.getCapturedArgCount == 0 (an O(1) read of a field the
cleaner already holds) before loading and parsing the capturing class, skipping
the work entirely for non-capturing closures. This is safe with respect to the
return-statement fail-fast because a non-local return captures its
NonLocalReturnControl key, so such a closure always has at least one captured
argument.
- Memoize the return-statement verdict per class in a java.lang.ClassValue,
using a collecting visitor so a single parse answers for every method of the
class. ClassValue entries are reclaimed with the class, so runtime-generated
closure classes (REPL lines, Ammonite commands) are not pinned.
With the change, cleaning-related frames fall to ~0% of CPU samples
(xbean.asm9: 11.02% -> 0.17% on the profiled suite), getClassReader is invoked
once per distinct class instead of once per job, and RDDSuite wall clock drops
~8.5%.
Behavior is unchanged, including the ReturnStatementInClosureException
fail-fast for every capturing closure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
