HeartSaVioR commented on code in PR #38115:
URL: https://github.com/apache/spark/pull/38115#discussion_r988463233
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/FlatMapGroupsInPandasWithStateExec.scala:
##########
@@ -83,7 +83,17 @@ case class FlatMapGroupsInPandasWithStateExec(
private val chainedFunc = Seq(ChainedPythonFunctions(Seq(pythonFunction)))
private lazy val (dedupAttributes, argOffsets) = resolveArgOffsets(
groupingAttributes ++ child.output, groupingAttributes)
- private lazy val unsafeProj = UnsafeProjection.create(dedupAttributes,
child.output)
+
+ // See processTimedOutState: we create a row which contains the actual
values for grouping key,
+ // but all nulls for value side by intention. This technically changes the
schema of input to
+ // be "nullable", hence the schema information and the internal projection
of row should take
+ // this into consideration. Strictly saying, it's not applied to the part of
grouping key, but
+ // it doesn't hurt much to we apply the same for grouping key as well.
Review Comment:
```suggestion
// it doesn't hurt much even if we apply the same for grouping key as well.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
