Vicky Kak created GOBBLIN-452:
---------------------------------
Summary: Logging related Improvement in Gobblin Cluster
Key: GOBBLIN-452
URL: https://issues.apache.org/jira/browse/GOBBLIN-452
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Vicky Kak
Assignee: Vicky Kak
Following are the logging related improvement that need to be done in Gobblin
Cluster Implementation
1) I have running the GAAS and Standalone cluster on the same machine and from
the same distribution, this will be typically needed for quick setup. Since I
have been starting the GAAS and Standalone master on same distribution, they
both are directing the logs to the same master.out file leading to overlap of
the logging details from the GAAS and standalone master. I have changed the
logging file from master.out to clustermaster.out on my local set up by
changing the $GOBBLIN_HOME/bin/gobblin-cluster-master.sh as
nohup $COMMAND >clustermaster.out 2>&1 & echo $! > $PID
We better make the changes in the distribution.
2) The log4j logging configuration is dynamically controlled in the
standalone/worker implementation, it does not work by default.I looked at how
the log4j configurations are being controlled in other modes, it is done via
the bootstrap scripts e.g gobblin-aws.sh as
LOG4J_PATH=file://${FWDIR_CONF}/log4j-aws.properties
COMMAND="$JAVA_HOME/bin/java -cp $CLASSPATH $JVM_FLAGS
gobblin.aws.GobblinAWSClusterLauncher -D log4j.configuration=$LOG4J_PATH" I see
the log4j configurations similarly being configured in gobblin-standalone.sh
too COMMAND+="-Dlog4j.configuration=file://$FWDIR_CONF/log4j-standalone.xml " I
did made the similar changes for the gobblin-service.sh as
LOG4J_PATH=file://${FWDIR_CONF}/log4j-cluster.properties
COMMAND="$JAVA_HOME/bin/java -Dlog4j.debug -Dlog4j.configuration=$LOG4J_PATH
-cp $CLASSPATH $JVM_FLAGS gobblin.service.modules.core.GobblinServiceManager
--service_name $SERVICE_NAME $LOG_ARGS" This was done because the log4j
configuration for the GAAS which should have been taken from
$GOBBLIN_HOME/conf/service/log4j-cluster.properties was not being taken from
there, it was taken from the $GOBBLIN_HOME/lib/generator-2.6.0.jar. We should
keep the consistent model of loading the log4j, for the standalone cluster the
log4j configurations are being loaded via code and for the other gobblin
components(modes) it is via the configuration in the bootstrap scripts. We
should have it consistent and I think having it in the bootstrap scripts via
-Dlog4j.configuration is good option. I have to copy the
log4j-cluster.properties into the GOBBLIN_HOME/bin for running the Standalone
cluster master/worker node. We need to fix these log4j configrations issues.
This is already discussed here
[https://mail-archives.apache.org/mod_mbox/incubator-gobblin-user/201801.mbox/%3CCAPaCpY-QT5e%3DJm-94k2aZOLWq5crOLbPYK4KusAmEgUvQYHa0w%40mail.gmail.com%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
