John Zhuge created SPARK-59059:
----------------------------------
Summary: Support per-logger log level configuration with static
Spark conf
Key: SPARK-59059
URL: https://issues.apache.org/jira/browse/SPARK-59059
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.2.0
Reporter: John Zhuge
SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger
at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There
is still no static-conf way to set the level of an individual logger.
Today, turning on DEBUG for one class means shipping a `log4j2.properties` and
pointing both the driver and the executors at it:
{code:java}
spark-submit \
--files /tmp/log4j2.properties#log4j2.properties \
--conf
spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
--conf
spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
...
{code}
That is awkward in cluster mode and impossible for a Spark Connect client,
which has no `SparkContext` and often cannot change the log4j configuration of
the platform it connects to – the same motivation that was accepted for
SPARK-43782.
Proposal: support `spark.log.level.<logger>`, applied on the driver and on the
executors after the root level from `spark.log.level`, so it takes precedence
for the named logger only.
{code:java}
--conf spark.log.level=WARN
--conf spark.log.level.org.apache.spark.sql.execution=DEBUG
{code}
Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete
in 2021 without a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
