andygrove opened a new pull request, #7645:
URL: https://github.com/apache/datafusion-comet/pull/7645

   ## Which issue does this PR close?
   
   Backport of #7526 to `branch-1.2`. Closes #7303 on this branch.
   
   ## Rationale for this change
   
   JVM columnar shuffle writers read their Comet settings from the executor 
task thread. In a map stage launched outside a SQL execution (for example 
`df.queryExecution.toRdd`), Spark does not propagate the session's SQL 
configuration to the task, so the writers ran with Comet defaults instead of 
the session's settings (#7303). `branch-1.2` has the same 
`CometShuffleManager.getWriter` path. See #7526 and #7303 for the full analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of 55ea0d4f77 (`git cherry-pick -x`) with no conflicts and no 
adaptations. The shuffle dependency captures the session's Comet settings 
(reusing the native shuffle specification's capture when there is one), and 
`CometShuffleManager` applies them when constructing, writing and stopping the 
bypass-merge-sort and serialized JVM shuffle writers. The source PR's 
`CometColumnarShuffleSuite` test is included.
   
   This does not depend on #7348: the new test passes on this branch without it.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to. Whether `branch-1.1` needs a backport is not part 
of this PR.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw 
test-compile` passes, and the new test `JVM shuffle writers see session configs 
outside SQL execution` passes in both `CometShuffleSuite` (AQE on) and 
`DisableAQECometShuffleSuite` against a native library built from `branch-1.2`. 
With `spark/src/main` reverted to `branch-1.2`'s version, the test fails in 
both suites, so the branch has the bug and the test catches it. I ran the 
default Spark profile only.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to