andygrove opened a new pull request, #7650:
URL: https://github.com/apache/datafusion-comet/pull/7650

   ## Which issue does this PR close?
   
   Backport of #6852 to `branch-1.2`. Closes #6687 on this branch.
   
   ## Rationale for this change
   
   `RevertNativeForTransitionHeavyStages` assumed a result stage is always 
collected as rows. On Spark 4.0+, a cached query whose serializer accepts 
columnar input, as Comet's `ArrowCachedBatchSerializer` does, gets a columnar 
result stage, and once the rule reverted that stage to a row-based root nothing 
restored the columnar output. Building the cache then failed with `FilterExec 
has column support mismatch` (#6687). `branch-1.2` has the same rule and the 
same call sites. Transition reversion is off by default, so only users who 
enable `spark.comet.exec.transitionRevert.enabled` are affected. See #6852 for 
the full analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of d7f4fcaa58 (`git cherry-pick -x`) with no conflicts and no 
adaptations. The diff against the source commit is identical apart from line 
numbers and blob hashes. Both result-stage call sites in 
`RevertNativeForTransitionHeavyStages` go through a new 
`revertResultStageIfNeeded`, which reads the requested output format off the 
stage root, and `RevertNativeForTransitionHeavyStagesSuite` gets the source 
PR's tests.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to. Whether `branch-1.1` needs a backport is not part 
of this PR.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw 
test-compile` succeeds, and `RevertNativeForTransitionHeavyStagesSuite` passes 
(43 tests, 0 failures). No native source changed. With `spark/src/main` 
reverted to `branch-1.2`'s version, five of the new tests fail: 
`transition-heavy revert restores columnar output for a cached AQE result 
stage`, the two `AQE=true` cases of `transition-heavy revert of a result root 
that supports both formats`, and both `transition-heavy revert keeps the output 
format of the result root` cases. So the branch has the bug and the tests catch 
it. I did not run the other Spark profiles.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to