peter-toth commented on PR #58501: URL: https://github.com/apache/spark/pull/58501#issuecomment-5561177660
@dongjoon-hyun thank you for the review, and sorry for the churn. **The goal of this PR changed**, so your two regressions no longer apply to the code, and your latent `concat` finding turned out to be the more serious of the two bugs it fixes. **What it is now: a bugfix only.** Partition key rows compare at types with the naming erased, which closes two defects measured on master. - A join between two keyed sides whose struct key fields are named differently throws `STORAGE_PARTITION_JOIN_INCOMPATIBLE_REDUCED_TYPES`, for a join that reduced nothing. - A union over a key that two children hold under two namings **silently drops rows**. This is the `isGrouped` miscount from your `concat` comment, and it is not latent. An inner join over such a union returns 2 rows where 3 are correct. The JIRA is retyped to a Bug and rewritten, and it affects 4.2.0 onwards. **What moved out: carrying the key data types on the partitioning**, which was the original goal and the old title. Your review is why. An independent field has to be decided at four sites that put a partitioning's expressions over keys they did not build, and two of them mix members, so the field needs to be *shared* rather than per-member. That is [SPARK-59285](https://issues.apache.org/jira/browse/SPARK-59285), as a `KeyLayout` the collection's members hold by reference, and it will be a separate PR stacked on this one. The `SPARK-59176` guard in `EnsureRequirements` therefore stays here for now. I answered each of your comments in its thread, and said where each one landed in SPARK-59285 rather than here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
