Vinod KC created SPARK-58209:
--------------------------------
Summary: Mark XPathExtract, CallMethodViaReflection, and
UserDefinedGenerator stateful to prevent executor-side data races
Key: SPARK-58209
URL: https://issues.apache.org/jira/browse/SPARK-58209
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0, 4.0.0, 4.1.0, 3.5.0, 4.3.0
Reporter: Vinod KC
Three Catalyst expressions `XPathExtract` `CallMethodViaReflection` ,
`UserDefinedGenerator` maintain mutable state internally but do not override
stateful (which defaults to false).
As a result, freshCopyIfContainsStatefulExpression() never makes independent
copies of them. When two executor tasks share the same expression instance —
which can happen when sibling QueryExecutions are derived from the same base
DataFrame — they race on the shared mutable fields and silently produce
incorrect results.
Need to override def stateful: Boolean = true to XPathExtract,
CallMethodViaReflection, and UserDefinedGenerator. This will cause
freshCopyIfContainsStatefulExpression() to return a fresh instance per
concurrent task, giving each its own independent evaluator, buffer, and lazy
vars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
