sunchao opened a new pull request, #58685: URL: https://github.com/apache/spark/pull/58685
Related JIRA: [SPARK-55411](https://issues.apache.org/jira/browse/SPARK-55411), which introduced the affected behavior. A separate follow-up JIRA for this specific bug is pending creation; this draft currently uses the originating issue key. Draft: standalone and full-build validation remain pending. ### What changes were proposed in this pull request? Stop reporting the original scan ordering when storage-partitioned join planning drops partition-key positions or applies partition-key reducers. These operations can concatenate separately ordered input splits. `EnsureRequirements` can then insert the sort required by a downstream sort-merge join. Full-key identity projections retain the existing ordering check. This is a focused branch-4.0 follow-up. The newer grouping implementation introduced by [#54330](https://github.com/apache/spark/pull/54330) already checks ordering against the final groups; its optional sorted-merge optimization in [#55116](https://github.com/apache/spark/pull/55116) is not backported here. ### Why are the changes needed? `DataSourceV2ScanExecBase.outputOrdering` checks only the original groups. `BatchScanExec.inputRDD` can subsequently project or reduce the keys and concatenate several of those groups into one partition. For example, two singleton splits ordered by `(k, id)` contain `(1, 5)` and `(1, 1)` under different discarded partition keys. Grouping them by `k` produces ids `5,1`, but the scan still advertises ascending `(k,id)` ordering. A sort-merge join can skip a matching row because its input is not sorted. ### Does this PR introduce _any_ user-facing change? Yes. Joins over scans whose input partitions are regrouped produce correct results by sorting when necessary. The check is conservative: a key-dropping projection or reducer can discard ordering even when its particular partition values do not merge. Full-key identity projections preserve ordering. ### How was this patch tested? Added native `KeyGroupedPartitioningSuite` regressions using the existing in-memory catalog's `SupportsReportOrdering` scan. Singleton splits make the reported input ordering truthful. The tests cover subset projection, full-key identity projection, and bucket reduction, checking complete join results and the required sort while preserving the storage-partitioned join. `git diff --check` passed. Both baseline regrouping cases lost one sort-merge join match. The full-key identity-projection control passed on both baseline and candidate. Fresh selected-source local validation of a combined branch-4.0 tree containing the separate window, key-routing, scan-ordering and NULL-fixture proposals passed all 11 focused tests. The broader run completed all 244 exact test identities: 237 passed and seven function-based ordering cases in WriteDistributionAndOrderingSuite failed. All seven also failed on the Apache production baseline with matching exception types, messages and first eight stack frames; they remain a limitation of this local setup, not a passing full-suite result. No tests were skipped and no suite aborted; fresh-class origin checks passed. The separate baseline control reproduced six targeted production regressions, while five expected controls passed. Removing only the two NULL-fixture guards separately reproduced the insertion exception and stale-value result. Candidate tests cover their complete loops; baseline failures stop at the first failing iteration and do not establish later iterations. All nine changed files in the combined 4.0 tree passed Scalastyle with zero errors or warnings. JDK 17 / Scala 2.13.16 freshly compiled 40 selected Scala production sources, five Java sources and nine test/fixture sources; remaining dependencies were cached and fingerprinted. This is not a complete build, standalone validation of this PR, a whole-source-equivalent Apache runtime, or CI success. Earlier failed setup and audit attempts are retained separately and are not counted as passes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
