parthchandra opened a new pull request, #7043: URL: https://github.com/apache/datafusion-comet/pull/7043
## Which issue does this PR close? Related to #5337 (see https://github.com/apache/datafusion-comet/pull/5331#pullrequestreview-5523559500). ## Rationale for this change Once an Iceberg release reports output orderings (apache/iceberg#16750), Spark will drop the Sort above an Iceberg scan when `preserve-data-ordering` is on. Comet's native Iceberg scan has no k-way merge, so it does not keep that order. The result is unsorted rows that Spark thinks are sorted, which means wrong results. As suggested in the review on #5331, this guard is split out from the sort-aware read work so it can ship to release branches without it. ## What changes are included in this PR? - `CometScanRule`: fall back to Spark when `scanExec.ordering` is non-empty. The check runs before any Iceberg metadata reads. - Tests in `CometIcebergNativeSuite`: - Falls back when Iceberg reports an ordering. - Stays native when Iceberg cannot report one, even with the `preserve-data-grouping` and `preserve-data-ordering` configs on. - Test helpers in `CometIcebergTestBase`: `icebergReportsOrdering` and `setIcebergSortOrderAsc`. ## How are these changes tested? The new tests in `CometIcebergNativeSuite`. The fallback test is skipped on Iceberg runtimes that don't report an ordering (no `SortOrderAnalyzer`). The "stay native" test runs on every profile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
