peter-toth commented on PR #59599: URL: https://github.com/apache/spark/pull/59599#issuecomment-6606950423
Thanks @ammarchalifah, merged to branch-4.1. Could you open the backport PR for branch-4.0? The cherry-pick applies cleanly but doesn't compile there. On 4.0 `StoragePartitionJoinParams` is still in `org.apache.spark.sql.execution.datasources.v2` (SPARK-53272 moved it to `execution.joins` in 4.1). So only that import in `EnsureRequirements` needs to change. With it, `KeyGroupedPartitioningSuite` and `EnsureRequirementsSuite` pass on the current branch-4.0, including all 30 new tests. After that I'd go for SPARK-60460 rather than SPARK-59121: - SPARK-60460 still reproduces on branch-4.1 with this PR in. The reorder shape returns no rows, and the others fail at planning or execution. - SPARK-59121's shapes seem to be covered on 4.1 by this PR's reduced-scan check. I ran the 4.1 tests we had for it (the double reduce and the shuffle onto reduced keys) and the two-key-space shape from #58447. All of them returned wrong rows before this PR and right rows after it, with a shuffle instead of SPJ. The `days`/`years` shape needs a cross-type reducer, which 4.1 doesn't have. For SPARK-60460, keep in mind that 4.2+ gets these shapes right with `GroupPartitionsExec`, which won't come to 4.1 and 4.0. So the fix should either do the same thing in 4.1's terms or fall back to a shuffle. My two comments on the ticket have the shapes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
