naveenp2708 commented on PR #59317: URL: https://github.com/apache/spark/pull/59317#issuecomment-6160441057
Thanks @HyukjinKwon, good catch. You're right that on the current planner a SPLIT_FOR_JOIN overflow fails the committed describesSameKeys gate in EnsureRequirements and falls back to a shuffle, so it never reaches fromPartitionings, and SPLIT_FOR_ORDERING pairs with nothing. Took your suggested comment, and reworded the PR description so it says the guard trades the silent shuffle fallback (and the duplicate-keys-as-grouped report on the ordering path) for a planning-time error. Makes the fail-fast vs derive-isGrouped question clearer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
