ulysses-you commented on code in PR #58339:
URL: https://github.com/apache/spark/pull/58339#discussion_r3921828465
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -96,7 +108,8 @@ case class GroupPartitionsExec(
case None => projectedExpressions
}
KeyedPartitioning(
- effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed)
+ effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed,
+ mayContainUnknownPartitionKeys =
k.mayContainUnknownPartitionKeys)
Review Comment:
Fixed in cf354c4153e. The claim is dropped for any non-identity grouping
(reorder, coalesce, or resize) and kept only when output partition `i` holds
exactly input partition `i`. It is given up as `UnknownPartitioning` at the
physical grouped count rather than by clearing the flag, so the layout still
reports that undeclared rows may exist. Added an end-to-end regression test on
your exact example (union `[3, 4, 1, 2]`, one-side-shuffled key 5, final
marked-vs-marked join) that loses the key-5 match without the fix, plus an
identity-grouping test showing the claim is retained there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
