comphead commented on code in PR #7521:
URL: https://github.com/apache/datafusion-comet/pull/7521#discussion_r4485578710
##########
dev/regenerate-golden-files.sh:
##########
@@ -129,7 +130,7 @@ main() {
if [ -n "$target_version" ]; then
if [[ ! "$target_version" =~ ^(3\.4|3\.5|4\.0|4\.1|4\.2)$ ]]; then
Review Comment:
The accepted-version regex still includes `3\.4`, while the error message
below it and the default `versions` list no longer do. `--spark-version 3.4`
passes validation and then runs Maven with `-Pspark-3.4`, which does not exist
any more. Could `3\.4|` be dropped from the pattern?
##########
docs/source/user-guide/latest/installation.md:
##########
@@ -47,15 +47,8 @@ Other versions may work well enough for development and
evaluation purposes.
Comet requires JDK 17 or later. JDK 11 is no longer supported as of the 1.1.0
release.
-```{warning}
-Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in
a future release.
-Apache Spark's own SQL test suite no longer runs against Spark 3.4
automatically; it runs only on
-demand. We recommend moving to Spark 3.5 or later.
-```
-
| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL
Tests in CI |
| ------------- | ------------ | ------------- | ----------------- |
--------------------- |
-| 3.4.3 | 17 | 2.12/2.13 | Nightly | On demand
|
| 3.5.9 | 17 | 2.12/2.13 | Nightly | Nightly
|
Review Comment:
The 3.4 table row, warning and download links are gone, but I could not find
a place that tells an existing 3.4 user that the next release no longer builds
it. `migration-guide.md` has no section for the version `main` is heading to
(`1.3.0-SNAPSHOT`), and `versioning_policy.md` tells users on an unsupported
Spark to stay on an earlier Comet release. Could this PR add a short `Upgrading
to Comet 1.3.0` entry saying Spark 3.4 is no longer built or published and that
users on 3.4 should stay on the last release that has `comet-spark-spark3.4_*`?
The JDK 11 removal has the same kind of note in the 1.1.0 section.
##########
dev/ci/linux-test-profiles.py:
##########
@@ -46,7 +46,6 @@
# `run-all-spark-profiles` label). The goal of the list is coverage of every
# Java, Scala and Spark version without testing every combination.
PROFILES = [
- {"name": "Spark 3.4, JDK 17, Scala 2.12", "java_version": "17",
"maven_opts": "-Pspark-3.4 -Pscala-2.12", "tier": "nightly"},
{"name": "Spark 3.5, JDK 17, Scala 2.13", "java_version": "17",
"maven_opts": "-Pspark-3.5 -Pscala-2.13", "tier": "nightly"},
Review Comment:
The removed `Spark 3.4, JDK 17, Scala 2.12` row was the only place the Comet
suites ran on Scala 2.12 (the Spark 3.5 row here is `-Pscala-2.13`). After this
change Scala 2.12 only gets compiled by `lint-java` and runs
`CelebornReflectionCompatibilitySuite`, plus Spark's own SQL suite for 3.5 when
it runs against a 2.12 build of Comet. We still publish
`comet-spark-spark3.5_2.12`, and the comment above this list says the goal is
coverage of every Java, Scala and Spark version. Should the 3.5 nightly row
move to `-Pscala-2.12`, or should a `Spark 3.5, JDK 17, Scala 2.12` nightly row
be added next to it? I have not run the matrix. This is from grepping
`scala-2.12` over `.github` and `dev/ci` at this head and at `upstream/main`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
