pan3793 commented on PR #59323: URL: https://github.com/apache/spark/pull/59323#issuecomment-6092970475
Could we add a V2-specific override for `spark.sql.requireAllClusterKeysForCoPartition`, shared by the SPJ and one-side shuffle paths? For example, a large Iceberg order-items table bucketed by `user_id` could join a much smaller, non-broadcastable table on `(user_id, item_id)`. If each grouped bucket is only 32-128 MB, preserving the large table's layout and shuffling only the smaller side may be preferable. Disabling the global config also relaxes the protection for ordinary `HashPartitioning` from intermediate joins. A V2-specific override would let users opt into storage-layout reuse while keeping that protection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
