LuciferYang opened a new pull request, #57925: URL: https://github.com/apache/spark/pull/57925
### What changes were proposed in this pull request? Adds `build_codegen_jdk.yml`, a scheduled workflow that runs the test suite with `spark.sql.codegen.compiler=jdk`, plus a `SPARK_CODEGEN_COMPILER` env override on the config default so the workflow can set it. The override follows the same shape `SPARK_ANSI_SQL_MODE` already uses for `spark.sql.ansi.enabled`. The workflow runs daily and keeps only the jobs that compile generated code: build, pyspark, pyspark-pandas, sparkr, tpcds-1g, docker-integration-tests, yarn. Docs, lint, buf and ui are skipped because they never reach `CodeGenerator.compile`. The second commit on this branch also flips the `build_and_test.yml` default `envs` to `jdk`, which is what makes this PR's own CI run the whole suite on the JDK backend. That commit is a probe and will be dropped before merge; it is here to measure the failure count and the wall-clock cost, which is the open question below. ### Why are the changes needed? The default backend is Janino, so today `CodeCompilerSuite` is the only suite that exercises `JdkCodeCompiler` at all. Every other generated unit could take the javac path and none of them is tested on it. SPARK-57403 aims at `spark.sql.codegen.compiler=jdk` actually meaning javac, and there is currently no signal on whether that holds beyond one suite. The gap is not theoretical. The four silent-wrong-answer bugs fixed in #57768 (a shadowed field, a hidden static method, an instance method clashing with a static, and an array type) were all found by hand-probing reflection semantics rather than by a failing test. Each sits on a path the encoder, `ObjectExpressions` and `Dataset` suites already cover under Janino, so running those suites on the JDK backend is what turns the next such bug into a test failure instead of a wrong query result. ### Does this PR introduce _any_ user-facing change? No. The config default is unchanged unless `SPARK_CODEGEN_COMPILER` is set, which only the new workflow does. ### How was this patch tested? `CodeCompilerSuite` and `CodeGenerationSuite` pass with the default flipped (`SPARK_CODEGEN_COMPILER=jdk build/sbt "catalyst/testOnly *CodeCompilerSuite *CodeGenerationSuite"`, 97 tests), which confirms no test depends on the default being `janino`. The workflow YAML parses and matches the structure of `build_non_ansi.yml`. Beyond that, this PR's own CI run is the test: it exercises every job on the JDK backend. javac is several times slower than Janino on large generated units and up to two orders of magnitude slower on small ones, so the run may well surface both failures and jobs that no longer fit in the time limit. I will report the numbers here and adjust the job list before asking for review. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
