This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 2c06ef1a49d8 [SPARK-49628][SQL] ConstantFolding should copy stateful
expression before evaluating
2c06ef1a49d8 is described below
commit 2c06ef1a49d8c81bdc1b880d7b0e8319186c2004
Author: Wenchen Fan <[email protected]>
AuthorDate: Fri Sep 13 15:00:53 2024 +0800
[SPARK-49628][SQL] ConstantFolding should copy stateful expression before
evaluating
### What changes were proposed in this pull request?
It's possible that a logical plan instance is being shared by multiple DFs
and these DFs are executed in parallel. Spark always copy stateful expressions
before evaluating them, but one place is missed: `ConstantFolding` can also
execute expressions. This PR fixes it.
### Why are the changes needed?
avoid concurrency issues.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Not able to write a test for it, but this concurrency issue is quite obvious
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #48104 from cloud-fan/constant.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
(cherry picked from commit 319e7cc7d0e7ba9a99f808d51a8d635a6159ce8f)
Signed-off-by: Kent Yao <[email protected]>
---
.../scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 1d756a2dcb74..c0e49d668c24 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -77,7 +77,7 @@ object ConstantFolding extends Rule[LogicalPlan] {
// Fold expressions that are foldable.
case e if e.foldable =>
try {
- Literal.create(e.eval(EmptyRow), e.dataType)
+
Literal.create(e.freshCopyIfContainsStatefulExpression().eval(EmptyRow),
e.dataType)
} catch {
case NonFatal(_) if isConditionalBranch =>
// When doing constant folding inside conditional expressions, we
should not fail
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
