Amiya Chakraborty created BIGTOP-3093:
-----------------------------------------
Summary: Modifying the way Spark's log4j.properties file is being
generated
Key: BIGTOP-3093
URL: https://issues.apache.org/jira/browse/BIGTOP-3093
Project: Bigtop
Issue Type: Improvement
Components: spark
Reporter: Amiya Chakraborty
Attachments: SAMPLE
patch-Modify_Spark_log4j.properties_file_generation.diff
*WHY ?*
Currently, when Bigtop's spark puppet module is deployed, it takes
/etc/spark/conf/log4j.properties.template (which comes from the Spark
distribution) and generates /etc/spark/conf/log4j.properties.
However, there is a disadvantage to this - any custom configuration overrides
(originating from .yaml files) containing developer-friendly comments are not
successfully parsed and subsequently, can't make to
/etc/spark/conf/log4j.properties file.As a result, from
troubleshooting/debugging viewpoint; down the line any support eng./developer
would find it difficult to interpret significance of these settings.
For example, the following log4j rolling settings would get added to generated
/etc/spark/conf/log4j.properties file, but without the associated comments:
_# Set the default spark streaming app log level to INFO. When running spark
streaming app, # the logs generated by this class will be rolled on hourly
basis to avoid deletion_
_# of stdout and stderr logs by LogPusher on crossing the 4 hour retention
period log4j.logger.org.apache.spark.streaming=INFO, DRFA-stderr, DRFA-stdout
log4j.appender.DRFA-stderr=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA-stderr.file=${spark.yarn.app.container.log.dir}/stderr
log4j.appender.DRFA-stderr.DatePattern='.'yyyy-MM-dd-HH_
_log4j.appender.DRFA-stderr.layout=org.apache.log4j.PatternLayout_
_log4j.appender.DRFA-stderr.layout.ConversionPattern=%d\{ISO8601} %p [%t]
%c:%m%n_
_#Roll stdout logs_
_log4j.appender.DRFA-stdout=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA-stdout.file=${spark.yarn.app.container.log.dir}/stdout
log4j.appender.DRFA-stdout.DatePattern='.'yyyy-MM-dd-HH_
_log4j.appender.DRFA-stdout.layout=org.apache.log4j.PatternLayout_
_log4j.appender.DRFA-stdout.layout.ConversionPattern=%d\{ISO8601} %p [%t]
%c:%m%n
_Gradually, as the Spark's log4j.properties file will grow with numerous
configurations; absence of comments will render log4j.properties file difficult
to maintain. Besides, given that many Cloud vendors override Spark's
log4j.properties configurations; this will be beneficial to substantial number
of users. Please find enclosed sample patch file providing glimpse of how the
"overrides" attribute is being used.
OBJECTIVE ?
This Jira intends to expedite the process and get Spark's log4j.properties file
be sourced on newly added template rather than being sourced on open source
Spark distribution file log4j.properties.template.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
