ulysses-you commented on code in PR #58339:
URL: https://github.com/apache/spark/pull/58339#discussion_r3921828465
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -96,7 +108,8 @@ case class GroupPartitionsExec(
case None => projectedExpressions
}
KeyedPartitioning(
- effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed)
+ effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed,
+ mayContainUnknownPartitionKeys =
k.mayContainUnknownPartitionKeys)
Review Comment:
thank you @szehon-ho . Fixed in cf354c4153e. The claim is dropped for any
non-identity grouping (reorder, coalesce, or resize) and kept only when output
partition `i` holds exactly input partition `i`. It is given up as
`UnknownPartitioning` at the physical grouped count rather than by clearing the
flag, so the layout still reports that undeclared rows may exist. Added an
end-to-end regression test on your exact example (union `[3, 4, 1, 2]`,
one-side-shuffled key 5, final marked-vs-marked join) that loses the key-5
match without the fix, plus an identity-grouping test showing the claim is
retained there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
