This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5fd608b82509 [SPARK-57411][INFRA] Add a daily build with the JDK
codegen backend
5fd608b82509 is described below
commit 5fd608b825090e9da3b06107dc0756d114fb815b
Author: YangJie <[email protected]>
AuthorDate: Tue Aug 18 20:57:17 2026 +0800
[SPARK-57411][INFRA] Add a daily build with the JDK codegen backend
### What changes were proposed in this pull request?
Adds `build_codegen_jdk.yml`, a scheduled workflow that runs the test suite
with `spark.sql.codegen.compiler=jdk` every other day. The change has two parts:
- `SPARK_CODEGEN_COMPILER` overrides the `spark.sql.codegen.compiler`
default, so the workflow can select the backend through `envs`. This follows
the shape `SPARK_ANSI_SQL_MODE` already uses for `spark.sql.ansi.enabled`, and
the four TPC-DS steps forward the variable explicitly the same way they forward
that one. An empty value counts as unset, for the reason in the testing section.
- The workflow raises three job timeouts to 200, 130 and 140, because javac
pushes those jobs past their defaults. The inputs it passes came from #58833,
split out of this PR per review feedback and now merged. Numbers are in the
testing section.
The workflow keeps only the jobs that compile generated code: build,
pyspark, pyspark-pandas, sparkr, tpcds-1g, docker-integration-tests, yarn.
Docs, lint, buf and ui are skipped because they never reach
`CodeGenerator.compile`. It runs on `*/2` days, matching the cadence of the
other heavy scheduled builds; daily would have made a non-default backend
roughly twice the compute of the next-heaviest scheduled workflow.
### Why are the changes needed?
The default backend is Janino, so `CodeCompilerSuite` is currently the only
suite that exercises `JdkCodeCompiler`. Every other generated unit could take
the javac path and none of them is tested on it. The SPARK-57403 umbrella aims
at `spark.sql.codegen.compiler=jdk` actually meaning javac, and there is no
signal today on whether that holds beyond one suite.
This has already cost us. The four silent-wrong-answer bugs fixed in #57768
(a shadowed field, a hidden static method, an instance method clashing with a
static, and an array type) were all found by hand-probing reflection semantics
rather than by a failing test. Each sits on a path the encoder,
`ObjectExpressions` and `Dataset` suites already cover under Janino, so running
those suites on the JDK backend is what turns the next such bug into a test
failure instead of a wrong query result.
### Does this PR introduce _any_ user-facing change?
No. The config default is unchanged unless `SPARK_CODEGEN_COMPILER` is set
to `janino` or `jdk`, which only the new workflow does.
### How was this patch tested?
Three probe runs of the full suite on the JDK backend, driven by a branch
commit that flipped the `build_and_test.yml` default `envs` to `jdk`. That
commit is reverted here; it existed only to measure the cost and the failure
count.
The last of them, run 31593826890, was green across all 33 jobs: 79201
tests, 0 failures. Every job outside `CodeCompilerSuite` passes on javac.
Durations for the jobs that ran near their limit, javac against Janino:
| job | javac | Janino | old cap | new cap |
| --- | --- | --- | --- | --- |
| `sql - other tests` | 171m | 127m | 150 | 200 |
| `pyspark-sql` and the rest of its matrix entry | 109m | 73m | 120 | 130 |
| `docker-integration-tests` | 116m | 102m | 120 | 140 |
Each new cap is the javac figure plus about 20%. The Janino column is the
slowest observed across recent runs of `build_main`, `build_java21`,
`build_java25`, `build_non_ansi` and `build_uds`, five to twelve samples
depending on the job. The 171m is a single measurement: the first two probe
runs cancelled that job at the 150 default, which bounds it from below and says
nothing more. Next closest after these three was `sql - slow tests` at 124m
against a 150 cap, so it needed no change [...]
On test failures, the first run reported 2 out of 76519, in
`StateDataSourceTransformWithStateSuiteCheckpointV2`, on leaked file streams.
Neither later run reproduced it, and that suite has failed before on a commit
using the Janino default, so it is not a javac problem. Its logs have since
expired, so run 31593826890 and its 0 failures is the part of this that can
still be re-checked.
Reverting the probe then turned up a bug in the env override itself, which
is worth recording because it is the one place this change could have broken
other callers. `tpcds-1g` failed with:
```
[INVALID_CONF_VALUE.OUT_OF_RANGE_OF_OPTIONS] The value '' in the config
"spark.sql.codegen.compiler" is invalid. It should be one of 'janino, jdk'.
```
The TPC-DS steps read `${{ fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER
}}`, and only this PR's new workflow puts that key in its `envs`. For every
other caller the expression resolves to nothing, and the runner exports the
variable with an empty value rather than omitting it. `System.getenv` reports
it as present with a zero-length string, so `getOrElse` returned `""` and
`checkValues` rejected it on the first read of the config. The two existing
env-driven confs in `SQLConf` avoid t [...]
The conf now filters the empty string out, which keeps the variable's three
states distinct: unset and empty both mean use the default, anything else is
validated as before. Probed all of them against a forked JVM: unset gives
janino, empty gives janino, `jdk` gives jdk, `JDK` gives jdk (the `transform`
still lowercases), and `ecj` is still rejected. The empty case fails without
the filter and passes with it.
One coverage gap came out of the same review. The fourth TPC-DS step, "Run
TPC-DS queries on collated data", had no `env:` block at all, so
`TPCDSCollationQueryTestSuite` was still running on Janino inside this very
workflow. It now gets the variable like the other three. That step also still
lacks `SPARK_ANSI_SQL_MODE`, which `build_non_ansi.yml` would presumably want;
left alone here as a separate question.
Locally, `CodeCompilerSuite` passes (60 tests), along with
`catalyst/compile` and `catalyst/scalastyle`. `CodeCompilerSuite` and
`CodeGenerationSuite` also pass with the default flipped
(`SPARK_CODEGEN_COMPILER=jdk build/sbt "catalyst/testOnly *CodeCompilerSuite
*CodeGenerationSuite"`, 97 tests), which confirms no test depends on the
default being `janino`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #57925 from LuciferYang/spark-codegen-jdk-daily.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.github/workflows/build_and_test.yml | 5 ++
.github/workflows/build_codegen_jdk.yml | 65 ++++++++++++++++++++++
README.md | 1 +
.../org/apache/spark/sql/internal/SQLConf.scala | 3 +-
4 files changed, 73 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 15b55bf9aa49..78dbfc5c2521 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1348,6 +1348,7 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly
org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{
fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=-1
spark.sql.join.preferSortMergeJoin=true
@@ -1356,6 +1357,7 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly
org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{
fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=10485760
- name: Run TPC-DS queries (Shuffled hash join)
@@ -1363,12 +1365,15 @@ jobs:
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly
org.apache.spark.sql.TPCDSQueryTestSuite"
env:
SPARK_ANSI_SQL_MODE: ${{ fromJSON(inputs.envs).SPARK_ANSI_SQL_MODE }}
+ SPARK_CODEGEN_COMPILER: ${{
fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
SPARK_TPCDS_JOIN_CONF: |
spark.sql.autoBroadcastJoinThreshold=-1
spark.sql.join.forceApplyShuffledHashJoin=true
- name: Run TPC-DS queries on collated data
run: |
SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 build/sbt "sql/testOnly
org.apache.spark.sql.TPCDSCollationQueryTestSuite"
+ env:
+ SPARK_CODEGEN_COMPILER: ${{
fromJSON(inputs.envs).SPARK_CODEGEN_COMPILER }}
- name: Upload test results to report
if: always()
uses: actions/upload-artifact@v7
diff --git a/.github/workflows/build_codegen_jdk.yml
b/.github/workflows/build_codegen_jdk.yml
new file mode 100644
index 000000000000..3df5f94ce484
--- /dev/null
+++ b/.github/workflows/build_codegen_jdk.yml
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Runs the test suite with `spark.sql.codegen.compiler=jdk`, so that codegen
+# reaching the JDK backend is covered by more than `CodeCompilerSuite`. The
+# default backend is Janino, which leaves the javac path (SPARK-57403)
exercised
+# only by its own suite even though every generated unit could take it.
+#
+# javac is slower than Janino, so this runs on its own schedule rather than on
+# every push, skips the jobs that do not compile generated code (docs, lint,
buf,
+# ui), and raises the timeout of the three jobs that no longer fit their
default.
+name: "Build / Codegen JDK backend (master, Scala 2.13, JDK 17)"
+
+on:
+ schedule:
+ - cron: '37 8 */2 * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: master
+ hadoop: hadoop3
+ # See SPARK-57411 for the measurements behind these three.
+ build_timeout_minutes: 200
+ pyspark_timeout_minutes: 130
+ docker_integration_tests_timeout_minutes: 140
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-312",
+ "PYTHON_TO_TEST": "python3.12",
+ "SPARK_CODEGEN_COMPILER": "jdk"
+ }
+ jobs: >-
+ {
+ "build": "true",
+ "pyspark": "true",
+ "pyspark-pandas": "true",
+ "sparkr": "true",
+ "tpcds-1g": "true",
+ "docker-integration-tests": "true",
+ "yarn": "true"
+ }
diff --git a/README.md b/README.md
index a8349b9ba5e7..b57b53688f54 100644
--- a/README.md
+++ b/README.md
@@ -34,6 +34,7 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml)
|
+| | [](https://github.com/apache/spark/actions/workflows/build_codegen_jdk.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_uds.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_rockdb_as_ui_backend.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml)
|
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index a10af15b2822..4deaa0bcf7f2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2984,7 +2984,8 @@ object SQLConf {
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
.checkValues(Set("janino", "jdk"))
- .createWithDefault("janino")
+ .createWithDefault(
+
sys.env.get("SPARK_CODEGEN_COMPILER").filter(_.nonEmpty).getOrElse("janino"))
val WHOLESTAGE_HUGE_METHOD_LIMIT =
buildConf("spark.sql.codegen.hugeMethodLimit")
.internal()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
