-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25265/
-----------------------------------------------------------
(Updated Sept. 2, 2014, 7:07 p.m.)
Review request for Ambari and Dmitro Lisnichenko.
Bugs: AMBARI-7119
https://issues.apache.org/jira/browse/AMBARI-7119
Repository: ambari
Description
-------
PROBLEM: log4j settings made via Ambari update the log4j file but do not take
any affect when restarting HDFS. It seems there are hardcoded settings in
/usr/lib/hadoop/libexec/hadoop-config.sh such as this at line 221:
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.root.logger=$
{HADOOP_ROOT_LOGGER:-INFO,console}
"
BUSINESS IMPACT: Customers have to change core files or set environment
variables explicitly by setting up a profile script
STEPS TO REPRODUCE: Log in to Ambari and change log4j properties such that
hadoop.root.logger=INFO,DRFA. The log4j file is updated in /etc/hadoop/conf.
Restart the HDFS service. Do a ps -ef | grep <PID> for the namenode. The
process shows duplicate entries for several properties and does not show the
logging change. Here is the duplication and incorrect root logger setting seen
locally in testing:
hdfs 4304 1 14 07:26 ? 00:00:10 /usr/jdk64/jdk1.7.0_45/bin/java
-Dproc_namenode -Xmx1024m -Djava.net.preferIPv4Stack=true
-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
-Dhadoop.root.logger=INFO,console
-Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-
amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml
-Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hdfs
-Dhadoop.log.file=hadoop-hdfs-namenode-sandbox.hortonworks.com.log
-Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
-Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/lib/hadoop/lib/native
/Linux-amd64-64:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/native/Linux-
amd64-64:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server
-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
-XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:NewSize=100m
-XX:MaxNewSize=50m -Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS
-Dhdfs.audit.logger=INFO,DRFAAUDIT -server -XX:ParallelGCThreads=8
-XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log
-XX:NewSize=100m -XX:MaxNewSize=50m
-Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726 -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m
-Xmx1024m -Dhadoop.security.logger=INFO,DRFAS
-Dhdfs.audit.logger=INFO,DRFAAUDIT -server -XX:ParallelGCThreads=8
-XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log
-XX:NewSize=100m -XX:MaxNewSize=50m
-Xloggc:/var/log/hadoop/hdfs/gc.log-201407140726 -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms1024m
-Xmx1024m -Dhadoop.security.logger=INFO,DRFA
ACTUAL BEHAVIOR: log4j changes made in Ambari do not persist in process. It
seems there are values set in /usr/lib/hadoop/libexec/hadoop-config.sh that
override no matter what. There is also duplication of settings, assuming this
is from hadoop-config.sh as well.
EXPECTED BEHAVIOR: Settings made in Ambari should be persisted and used by the
process.
SUPPORT ANALYSIS: Support made changes to log4j in Ambari on a test cluster
and they were not used
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog170.java
c2dec57
ambari-server/src/main/resources/stacks/HDP/1.3.2/hooks/after-INSTALL/scripts/params.py
a4fb172
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/configuration/hadoop-env.xml
a8e6243
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/configuration/hdfs-log4j.xml
1366c5d
ambari-server/src/main/resources/stacks/HDP/1.3.2/services/HDFS/package/scripts/params.py
7d23680
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
3fe5905
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/configuration/hadoop-env.xml
817be91
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/configuration/hdfs-log4j.xml
976a3c2
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/package/scripts/params.py
ee6b64f
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog170Test.java
cc97114
ambari-web/app/data/HDP2/site_properties.js 3ba0246
ambari-web/app/data/site_properties.js 1f6bb63
Diff: https://reviews.apache.org/r/25265/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
