Enable rotateable JVM garbage collection logs for Hadoop daemons
----------------------------------------------------------------
Key: HADOOP-6858
URL: https://issues.apache.org/jira/browse/HADOOP-6858
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 0.22.0
Reporter: Andrew Ryan
The purpose of this enhancement is to make it easier to collect garbage
collection logs and insure that they persist across restarts in the same way
that the standard output files of Hadoop daemon JVM's currently does.
Garbage collection logs are a vital debugging tool for administrators and
developers. In our production environments, at some point or another, every
single type of Hadoop daemon has OOM'ed or experienced other significant issues
related to GC and/or lack of heap memory. For the longest time, we have put in
garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS,
etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails
-Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
Unfortunately, these logs don't survive a restart of the node, so if a node
OOM's and then is restarted automatically, or manually by someone who is
unaware, we lose the GC logs forever. We also have to manually add GC log
options to each daemon. This patch:
1) Creates a single, optional, off by default, parameter for specifying GC
logging.
2) If that parameter is set, automatically enables GC logging for all daemons
in the cluster. The parameter is flexible enough to allow for the different
ways various vendor's JVM's require garbage collection logging to be specified.
3) If GC logging is on, insures that the GC log files for each daemon are
rotated with up to 5 copies kept, same as the .out files currently.
We are currently running a variation of this patch in our 0.20 install. This
patch actually includes changes to common, mapred, and hdfs, so it obviously
cannot be applied as-is, but is included here for review and comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
