andygrove opened a new pull request, #6879: URL: https://github.com/apache/datafusion-comet/pull/6879
## Which issue does this PR close? Backport of #5957 to branch-1.1. Related to #5719 and #6152. ## Rationale for this change Reverting a transition-heavy native write stage can erase or duplicate the write node because the write execs identify their child as the original Spark plan. That can leave Iceberg without commit-message rows or report a successful Parquet write without producing output. ## What changes are included in this PR? This adapts #5957 to branch-1.1. Native Iceberg and Parquet writes retain the Spark write exec they replaced, `CometExec.sparkFallback` owns operator-specific restoration, invalid aliases skip whole-stage reversion, stacked transitions are fully removed without crossing exchange boundaries, result-stage output format is restored, and reverted native shuffle stages retain Arrow-backed input. It includes the applicable writer, planner, scan, shuffle, documentation, and regression-test updates. The post-1.1 fused LocalTopK fallback implementation and its tests are omitted because that operator is not present on branch-1.1. ## How are these changes tested? - Maven compilation, Scalastyle, and Spotless checks - `RevertNativeForTransitionHeavyStagesSuite`: 33 tests - Focused Iceberg write fallback tests: 4 tests with AQE on and off - Focused Parquet writer fallback tests: 6 tests with AQE on and off - `git diff --cached --check` The `run-all-spark-profiles` and `run-iceberg-tests` labels request the broader profile and Iceberg coverage warranted by these planner and writer changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
