LuciferYang commented on code in PR #56430:
URL: https://github.com/apache/spark/pull/56430#discussion_r3388790887
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2647,6 +2625,32 @@ object SQLConf {
"disable logging or -1 to apply no limit.")
.createWithDefault(1000)
+ val CODEGEN_COMPILER = buildConf("spark.sql.codegen.compiler")
+ .internal()
+ .doc("The compiler used to turn generated Java source into bytecode. " +
+ "Supported values are 'janino' (default) and 'jdk'. " +
+ "'janino' uses the Janino library; it is several times faster but the
project " +
+ "is unmaintained upstream (last release 3.1.12, Feb 2024). " +
+ "'jdk' uses javax.tools.JavaCompiler from the JDK; it is maintained on
the " +
+ "JDK release cadence but adds ~5x cold-start latency for large generated
units " +
+ "and 30-300x for small ones. Switch to 'jdk' only if Janino's
maintenance status " +
+ "is a concern; for most workloads the default remains the better
trade-off. " +
+ "When 'jdk' is requested but javax.tools.JavaCompiler is unavailable " +
+ "(e.g. JRE-only image) Spark falls back to 'janino' with a warning. " +
+ "Regardless of this setting, codegen in REPL / interactive sessions
(spark-shell, " +
+ "Spark Connect session artifacts) and generated code referencing a class
nested in " +
+ "a Scala package object always compile with 'janino', because the JDK
compiler " +
+ "cannot resolve such classes; a one-time INFO log records each such
routing.")
+ .version("5.0.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .stringConf
+ .transform(_.toLowerCase(Locale.ROOT))
+ .checkValues(Set("janino", "jdk"))
+ // TEMP: default flipped to "jdk" so this branch's GHA workflows exercise
the JDK
+ // backend across the full Spark test surface. Revert to "janino" before
the PR
+ // merges to upstream master.
+ .createWithDefault("jdk")
Review Comment:
for test, will revert after test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
