Vinod KC created SPARK-58208:
--------------------------------
Summary: Deep-copy stateful expressions in
QueryExecution.optimizedPlan and ConvertToLocalRelation to prevent concurrent
optimizer races
Key: SPARK-58208
URL: https://issues.apache.org/jira/browse/SPARK-58208
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0, 4.0.0, 4.1.0, 3.5.0, 4.3.0
Reporter: Vinod KC
When two DataFrames are derived from the same base (e.g. val t =
base.select(...); val a = t.select(...); val b = t.select(...)), their logical
plans share the same expression objects. If a.collect() and b.collect() run
concurrently, both QueryExecution.optimizedPlan calls enter the optimizer with
the same expression instances. Expressions that hold mutable state (e.g.
NamedLambdaVariable with an AtomicReference) are then written by two threads
simultaneously, silently corrupting results.
ConvertToLocalRelation had the same gap: it evaluated
InterpretedMutableProjection directly over the shared projectList without
making independent copies first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
