szehon-ho commented on code in PR #58339:
URL: https://github.com/apache/spark/pull/58339#discussion_r3920336519
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:
##########
@@ -96,7 +108,8 @@ case class GroupPartitionsExec(
case None => projectedExpressions
}
KeyedPartitioning(
- effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed)
+ effectiveExpressions, partitionKeys, grouping.isGrouped,
grouping.isCollapsed,
+ mayContainUnknownPartitionKeys =
k.mayContainUnknownPartitionKeys)
Review Comment:
**[P1] Preserve the marker only across an identity partition mapping**
The marked-layout contract relies on every undeclared key being at
`hash(key) % numPartitions`. This copy preserves
`mayContainUnknownPartitionKeys` after any reducer-free `GroupPartitionsExec`,
but this node can reorder or coalesce the physical partitions, invalidating
that index relationship.
For an end-to-end example, start with a union whose declared key order is
`[3, 4, 1, 2]`, then one-side-shuffle a row with undeclared key 5 onto it. The
row lands at `h = hash(5) % 4`. A subsequent SPJ against an accurate `[1, 2, 3,
4]` layout is admitted by the subset check and makes this node sort the marked
side. Its old partitions `0, 1, 2, 3` move to `2, 3, 0, 1`, so key 5 moves to
`perm(h)`, while the marker is retained. An independently one-side-shuffled
marked layout with declared order `[1, 2, 3, 4]` puts key 5 at `h`. A final
marked-vs-marked join accepts the identical declared keys/order without a
shuffle, but the two key-5 rows are in `perm(h)` and `h`, so their match is
lost.
Please drop the keyed claim whenever grouping changes the physical partition
mapping (order, count, or membership), unless the fallback routing permutation
is explicitly retained. Clearing only the boolean would also be unsound,
because undeclared rows still exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
