andygrove opened a new pull request, #7650: URL: https://github.com/apache/datafusion-comet/pull/7650
## Which issue does this PR close? Backport of #6852 to `branch-1.2`. Closes #6687 on this branch. ## Rationale for this change `RevertNativeForTransitionHeavyStages` assumed a result stage is always collected as rows. On Spark 4.0+, a cached query whose serializer accepts columnar input, as Comet's `ArrowCachedBatchSerializer` does, gets a columnar result stage, and once the rule reverted that stage to a row-based root nothing restored the columnar output. Building the cache then failed with `FilterExec has column support mismatch` (#6687). `branch-1.2` has the same rule and the same call sites. Transition reversion is off by default, so only users who enable `spark.comet.exec.transitionRevert.enabled` are affected. See #6852 for the full analysis. ## What changes are included in this PR? Cherry-pick of d7f4fcaa58 (`git cherry-pick -x`) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes. Both result-stage call sites in `RevertNativeForTransitionHeavyStages` go through a new `revertResultStageIfNeeded`, which reads the requested output format off the stage root, and `RevertNativeForTransitionHeavyStagesSuite` gets the source PR's tests. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw test-compile` succeeds, and `RevertNativeForTransitionHeavyStagesSuite` passes (43 tests, 0 failures). No native source changed. With `spark/src/main` reverted to `branch-1.2`'s version, five of the new tests fail: `transition-heavy revert restores columnar output for a cached AQE result stage`, the two `AQE=true` cases of `transition-heavy revert of a result root that supports both formats`, and both `transition-heavy revert keeps the output format of the result root` cases. So the branch has the bug and the tests catch it. I did not run the other Spark profiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
