szehon-ho commented on code in PR #42306:
URL: https://github.com/apache/spark/pull/42306#discussion_r1304978430
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -344,7 +344,11 @@ case class KeyGroupedPartitioning(
} else {
// We'll need to find leaf attributes from the partition
expressions first.
val attributes = expressions.flatMap(_.collectLeaves())
- attributes.forall(x =>
requiredClustering.exists(_.semanticEquals(x)))
+
+ // Support only when all cluster key have an associated partition
expression key
+ requiredClustering.exists(x =>
attributes.exists(_.semanticEquals(x))) &&
+ // and if all partition expression contain only a single
partition key.
+ expressions.forall(_.collectLeaves().size == 1)
Review Comment:
This was to fix a test, I couldn't find it back to be honest. There was a
test somewhere that was trying this case (which isnt actually supported in the
code currently), and I think asserting the right exception, which I think would
break if SPJ is activated. I could revert this and see again to find the test,
if you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
