andygrove opened a new pull request, #7518:
URL: https://github.com/apache/datafusion-comet/pull/7518
## Which issue does this PR close?
There is no tracking issue for Spark 4.3 support yet. This is Stage 1 of the
bring-up described in the "Adding Support for a New Spark Version" contributor
guide, so it does not close anything. Related: the Spark 4.3 Arrow cache
research issue #7517.
## Rationale for this change
Spark 4.3.0 is released. Following the staged recipe in the contributor
guide, the first PR adds a build profile that compiles, with no tests enabled
for it.
## What changes are included in this PR?
- `spark-4.3` Maven profile in `pom.xml` and `spark/pom.xml` (Spark 4.3.0,
Scala 2.13.18, Parquet 1.17.0, slf4j 2.0.17, Hadoop 3.5.0, all the same as
4.2.0, so the profile is a copy of `spark-4.2`). Delta and Iceberg test
dependencies follow `spark-4.2`; Delta has no 4.3 release yet.
- `spark/src/{main,test}/spark-4.3/`, seeded from `spark-4.2` with one
change, in `CometInternalRowShim`, which gains the new `getTimestampNTZNanos`
and `getTimestampLTZNanos` getters.
- `isSpark43Plus` in `CometSparkSessionExtensions`.
- A "Spark 4.3, JDK 17" row in the `lint-java` matrix, which compiles main
and test sources and runs scalafix. The new version is not added to any test
matrix.
- `!spark/src/main/spark-4.3/**` exclusions in the other versions' `FILTERS`
in `dev/ci/compute-changes.py`, so a change confined to the 4.3 shims does not
trigger their Spark SQL jobs.
Spark 4.3 changes that needed code outside the new directory:
- **Shuffle managers.** `SparkEnv` now throws unless `spark.shuffle.manager`
is a `BlockingShuffleManager`, and `shuffleBlockResolver` moved from
`ShuffleManager` onto it. A new `ShimBlockingShuffleManager` (one copy for 3.x,
4.0, 4.1 and 4.2, one for 4.3) is the base of `CometShuffleManager` and
`CometCelebornShuffleManager`, with a `blockResolver` helper for the places
that read the resolver off a manager. Without this the Comet shuffle manager
compiles but fails at startup on 4.3.
- **Python runner.** `BasePythonRunner` passes an
`Option[PythonWorkerHandle]` where it passed an `Option[Int]` pid, so
`CometArrowPythonRunnerBase` uses a `CometPythonWorkerShim.WorkerHandle` alias.
- **Decimal overflow error.** `ShimSparkErrorConverter` calls
`DataTypeErrors.cannotChangeDecimalPrecisionError`, the only place it exists in
4.3. It has the same signature on every 4.x release.
- **Test code.** `TaskContextImpl` takes `cpuAmount: BigDecimal` instead of
`cpus: Int` (passed positionally so both compile), `SimpleCostEvaluator` and
`SimpleCost` gained a field (built through their constructors in
`CometExecRuleSuite`), and `createSparkSession` returns a
`classic.SparkSession` on 4.3, so `CometVariantShreddingSuite` drops the
explicit return type. The Celeborn test shuffle managers extend the new shim.
Known gap: `CometCelebornShuffleManager` reads the block resolver of the
Celeborn shuffle manager it wraps. That manager must be a
`BlockingShuffleManager` on 4.3, so a Celeborn build compiled against an older
Spark will fail with a clear `IllegalStateException`. Not testable here, since
no Celeborn release targets 4.3.
## How are these changes tested?
Main and test sources compile (`./mvnw -DskipTests test-compile`, JDK 17)
under `-Pspark-4.3`, and still compile under `-Pspark-4.2`, `-Pspark-4.1`
(default), `-Pspark-4.0`, `-Pspark-3.5 -Pscala-2.12` and `-Pspark-3.4
-Pscala-2.12`. `dev/ci/check-ci-config.py` passes. No tests were run on 4.3;
per the guide that starts in Stage 2, which will also exercise the shuffle
manager change at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
