andygrove opened a new issue, #7519: URL: https://github.com/apache/datafusion-comet/issues/7519
## Summary Spark 4.3.0 is released (tag `v4.3.0`). This tracks bringing up Comet support for it, following the staged recipe in the contributor guide, "Adding Support for a New Spark Version" (`docs/source/contributor-guide/adding_a_new_spark_version.md`). Spark 4.3.0 publishes the same Java 17, Scala 2.13.18, Parquet 1.17.0, slf4j 2.0.17, Hadoop 3.5.0 and Arrow 19.0.0 versions as 4.2.0, so the build profile is a copy of `spark-4.2`. ## Stages - [ ] **Stage 1**: Maven profile, shims, `isSpark43Plus`, compile-only `lint-java` CI row. Draft PR: #7518 - [ ] **Stage 2**: enable Comet's own JVM test suites under `-Pspark-4.3`, skipping failures with `assume()` and a linked issue each. - [ ] **Stage 3**: Spark SQL tests (`dev/diffs/4.3.0.diff`, a `spark_4_3` job in `ci.yml`, `compute-changes.py` and `check-ci-config.py`) and plan stability golden files. - [ ] **Stage 4**: add 4.3 to the experimental table and jar list in `installation.md`. - [ ] **Stage 5**: fix the skipped tests, one issue or cluster per PR. - [ ] **Stage 6**: promote from experimental to supported. ## Known Spark 4.3 changes to plan for Found while doing Stage 1: - `spark.shuffle.manager` must be a `BlockingShuffleManager` (new in 4.3), and `shuffleBlockResolver` moved onto it. Comet's shuffle managers need the shim from the Stage 1 PR; the Celeborn integration needs a Celeborn build that targets 4.3, so it should be checked at Stage 2. - `SpecializedGetters` gained `getTimestampNTZNanos` and `getTimestampLTZNanos`, for new nanosecond timestamp types. Comet currently throws for them; decide per type whether to fall back or support them. - `BasePythonRunner` passes an `Option[PythonWorkerHandle]` instead of a pid. - `SparkEnv.shuffleManager` is deprecated in favour of `shuffleManagerFor(dependency)`; there is also a new pipelined shuffle manager kind, which Comet's shuffle planning should not mistake for a blocking one. - `SimpleCostEvaluator` counts local sorts (`spark.sql.adaptive.costEvaluator.countLocalSort.enabled`), which can change AQE plan choices over Comet plans. - An opt-in Arrow cache serializer: tracked separately in #7517. ## Not yet decided - Which Delta and Iceberg versions to pair with the 4.3 profile. Delta has no 4.3 release; the profile reuses 4.3.1 for compilation only, as `spark-4.2` does. Iceberg tests reuse the 4.0 runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
