ulysses-you commented on PR #58044: URL: https://github.com/apache/spark/pull/58044#issuecomment-5350578626
Thanks for the thorough re-check, @peter-toth! Both findings addressed. **Finding 4 (root cause).** You're right that the qualifier is what the SQL test actually discriminates on — I've updated the framing to "compare children in the union's attribute space" across the title, PR description, the `prepareOutputPartitioning` comment, and the test comments. One nuance: nullability is still one of the fields `AttributeReference.equals` compares, so it remains a legitimate cause in its own right — the fix just normalizes all of them (name, nullability, metadata, qualifier) at once by remapping every child, including the first, to the union's output attributes. To keep both causes covered I added a dedicated DataFrame test that isolates the nullability case. Constructing it required disabling `PushDownPredicates`: otherwise `PushPredicateThroughNonJoin` relocates `IsNotNull` below the shuffle and child 0 becomes the `ShuffleExchangeExec` (no nullability mismatch), which matches your observation that a nullability-only plan isn't reachable through the no rmal optimizer path. **Finding 5 (attribute space).** Added `assert(hashPartitioning.expressions == Seq(unionExec.head.output.head))` to both tests, pinning that the propagated partitioning is expressed in the union's own output attributes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
