andygrove opened a new pull request, #7645: URL: https://github.com/apache/datafusion-comet/pull/7645
## Which issue does this PR close? Backport of #7526 to `branch-1.2`. Closes #7303 on this branch. ## Rationale for this change JVM columnar shuffle writers read their Comet settings from the executor task thread. In a map stage launched outside a SQL execution (for example `df.queryExecution.toRdd`), Spark does not propagate the session's SQL configuration to the task, so the writers ran with Comet defaults instead of the session's settings (#7303). `branch-1.2` has the same `CometShuffleManager.getWriter` path. See #7526 and #7303 for the full analysis. ## What changes are included in this PR? Cherry-pick of 55ea0d4f77 (`git cherry-pick -x`) with no conflicts and no adaptations. The shuffle dependency captures the session's Comet settings (reusing the native shuffle specification's capture when there is one), and `CometShuffleManager` applies them when constructing, writing and stopping the bypass-merge-sort and serialized JVM shuffle writers. The source PR's `CometColumnarShuffleSuite` test is included. This does not depend on #7348: the new test passes on this branch without it. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw test-compile` passes, and the new test `JVM shuffle writers see session configs outside SQL execution` passes in both `CometShuffleSuite` (AQE on) and `DisableAQECometShuffleSuite` against a native library built from `branch-1.2`. With `spark/src/main` reverted to `branch-1.2`'s version, the test fails in both suites, so the branch has the bug and the test catches it. I ran the default Spark profile only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
