peter-toth commented on PR #59599: URL: https://github.com/apache/spark/pull/59599#issuecomment-6339569304
@szehon-ho @uros-b IMO this is a real correctness problem, but it isn't a regression since the last 4.1 and 4.0 releases. So it isn't necessarily a release blocker. These SPJ bugs are open on 4.1 and 4.0 too, and fixed on `branch-4.2` and later: - SPARK-59121: wrong results when a storage-partitioned join reduces the partition keys of both sides. - SPARK-59887: wrong results when a storage-partitioned join pairs a transform of a join-key expression with one of a column. - SPARK-59901: an `INTERNAL_ERROR` when a one-side shuffle's join key references two columns. - SPARK-59688: wrong results when a storage-partitioned join skips the reduce on coinciding partition keys. - SPARK-58996: a partially clustered storage-partitioned join duplicates rows when `EnsureRequirements` re-runs. On 4.1 and 4.0 this needs `spark.sql.adaptive.forceOptimizeSkewedJoin`. Two smaller ones are open on every branch, master included: - SPARK-60027: a partition transform skips its input casts, so the query fails with a `ClassCastException`. - SPARK-60260: wrong results when `spark.sql.sources.v2.bucketing.enabled` is changed after a storage-partitioned join is planned. One more is open on 4.1 and 4.0 only: - SPARK-60460: a storage-partitioned join over a nested join that shuffles one side fails with `Can't zip RDDs`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
