HyukjinKwon commented on code in PR #37552:
URL: https://github.com/apache/spark/pull/37552#discussion_r947869640
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonExec.scala:
##########
@@ -116,6 +116,7 @@ trait EvalPythonExec extends UnaryExecNode {
}.toArray
}.toArray
val projection = MutableProjection.create(allInputs.toSeq, child.output)
+ projection.initialize(context.partitionId())
Review Comment:
BTW, this is a bandaid fix. To fundamentally fix this, I think we should do
something like adding a projection when there's a nondeterministic expression.
But this PR doesn't target to fix that. At least this code base is consistent
and does not break the intention of `initialize` since `EvalPythonExec` doesn't
do anything on partitions, and this is what `ProjectExec` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
