ytoprakc commented on code in PR #71650: URL: https://github.com/apache/airflow/pull/71650#discussion_r3918131759
########## airflow-e2e-tests/docker/Dockerfile.java: ########## @@ -19,12 +19,15 @@ # can spawn JVM subprocesses for @task.stub tasks. # # Pin Java 17 (rather than default-jre-headless): the Scala Spark example runs -# Apache Spark 3.5.x, which supports Java 8/11/17 but not Java 21. +# Apache Spark 3.5.x, which supports Java 8/11/17 but not Java 21. Debian Trixie Review Comment: Thanks for pointing this out @raphaelauv. I agree with the general direction for the Spark provider. Since `apache-airflow-providers-apache-spark` 6.x now uses Spark dependencies `>=4.0.0`, Java 21 is a valid baseline for Spark 4.x. For this specific file, I think there is a small distinction worth keeping. `airflow-e2e-tests/docker/Dockerfile.java` is used by the Java SDK e2e tests, not directly by the Spark provider package. That path currently includes `java-sdk/scala_spark_example`, which still pins `org.apache.spark:spark-sql_2.13:3.5.9` directly and has its runtime/module-opening setup written around Spark 3.5.x. So for this PR, I kept that Java SDK e2e image aligned with the existing Spark 3.5.x example and only changed how Java 17 is supplied on Trixie. Moving the Java SDK Scala Spark example to Spark 4.x / Java 21 sounds like a good follow-up, but I think it deserves its own PR because it would touch the Gradle dependency, Java toolchain, e2e build image, JVM module options, docs, and e2e validation together. Spark 4.2 adds Java 25 compatibility, but provider 6.x only gives us a Spark `>=4.0.0` baseline, so Java 25 feels ahead of the follow-up. This PR is already fairly large for a base image migration, so I would rather keep this part scoped to making the existing Java 17-based e2e setup work on Trixie instead of also changing the Spark version and Java SDK test baseline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
