jzhuge opened a new pull request, #58365: URL: https://github.com/apache/spark/pull/58365
### What changes were proposed in this pull request? Add `spark.log.level.<logger>`, a static Spark conf that sets the level of an individual logger on both the driver and the executors. ``` --conf spark.log.level=WARN --conf spark.log.level.org.apache.spark.sql.execution=DEBUG ``` `Utils.setLoggerLevels(conf)` reads every `spark.log.level.<logger>` entry, validates the value against `SparkContext.VALID_LOG_LEVELS`, and applies it with log4j2 `Configurator.setLevel`. It is called from `SparkContext` immediately after the existing `spark.log.level` handling, and from `CoarseGrainedExecutorBackend.run` immediately after the executor applies the root level, so the per-logger level is always applied after the root level and takes precedence for the named logger. ### Why are the changes needed? SPARK-43782 added `spark.log.level`, but it only sets the root logger. Raising the level of a single class still requires shipping a `log4j2.properties` and pointing both driver and executor JVMs at it with `-Dlog4j.configurationFile`, which is awkward in cluster mode and not possible from a Spark Connect client that has no `SparkContext` and cannot change the log4j configuration of the platform it connects to. That is the same motivation that was accepted for SPARK-43782. ### Does this PR introduce _any_ user-facing change? Yes, a new configuration property, documented in `docs/configuration.md`. There is no behavior change when it is not set. ### How was this patch tested? New tests in `SparkContextSuite`: one asserts the named logger ends at `DEBUG` while the root logger stays at the `ERROR` set by `spark.log.level`; one asserts an invalid level is rejected with a message naming the offending config key. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
