jbewing opened a new pull request, #13534: URL: https://github.com/apache/iceberg/pull/13534
This PR fixes a case where tables with current identical partitioning structures where one table was created directly w/ this structure and the other was created, had it's partitions evolved in a way that changed their order to match the partitioning of the other table. Currently, partition groupings are reported to Spark with partition fields sorted by fieldId and if two tables have identical partitions but report them in different orders then the storage partitioned join feature is disabled. This PR fixes this issue by changing the partition grouping key reported to Spark (and really just used everywhere) to calculate the grouping key sorted by the order of partition fields in the most recent spec ascending rather than by fieldId ascending. This change ends up changing how the grouping key is calculated for all of Iceberg by default, so I'm open to feedback on potentially scoping this down to just Spark if you'd like and/or investigating whether a fix is possible on the Spark end of things rather than in Iceberg. ### Testing I've added some unit tests to verify that an SPJ plan is generated for this case which fail on the current main branch but pass on this feature branch. Fixes: https://github.com/apache/iceberg/issues/13530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
