This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1f929c3095f [SPARK-42061][SQL] mark expression InvokeLike and
ExternalMapToCatalyst stateful
1f929c3095f is described below
commit 1f929c3095f7b7918ed02a0d666c0ee469b1ccfc
Author: Fred Liu <[email protected]>
AuthorDate: Wed Jan 18 09:54:14 2023 +0800
[SPARK-42061][SQL] mark expression InvokeLike and ExternalMapToCatalyst
stateful
### What changes were proposed in this pull request?
Those two expressions involves Array/Buffer that are not thread-safe. Need
to mark those stateful so existing Spark infra can copy those properly.
### Why are the changes needed?
Avoid correctness issues.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
This was found in our internal workload which runs queries concurrently.
The problem is gone after the fix.
Closes #39630 from lzlfred/fix_stateful_expression.
Authored-by: Fred Liu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
.../org/apache/spark/sql/catalyst/expressions/objects/objects.scala | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 56facda2af6..299a928f267 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -50,6 +50,9 @@ trait InvokeLike extends Expression with NonSQLExpression
with ImplicitCastInput
def propagateNull: Boolean
+ // InvokeLike is stateful because of the evaluatedArgs Array
+ override def stateful: Boolean = true
+
override def foldable: Boolean =
children.forall(_.foldable) && deterministic &&
trustedSerializable(dataType)
protected lazy val needNullCheck: Boolean =
needNullCheckForIndex.contains(true)
@@ -1400,6 +1403,9 @@ case class ExternalMapToCatalyst private(
override def nullable: Boolean = inputData.nullable
+ // ExternalMapToCatalyst is stateful because of the rowBuffer in
mapCatalystConverter
+ override def stateful: Boolean = true
+
override def children: Seq[Expression] = Seq(
keyLoopVar, keyConverter, valueLoopVar, valueConverter, inputData)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
