----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53248/#review154185 -----------------------------------------------------------
Ship it! Ship It! - Jaimin Jetly On Oct. 28, 2016, 8:15 p.m., XIAOBING ZHOU wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53248/ > ----------------------------------------------------------- > > (Updated Oct. 28, 2016, 8:15 p.m.) > > > Review request for Ambari and Jaimin Jetly. > > > Repository: ambari > > > Description > ------- > > As HDFS-11047 reported, DirectoryScanner does scan by deep copying > FinalizedReplica. In a deployment with 500,000+ blocks, we've seen the DN > heap usage being accumulated to high peaks very quickly. Deep copies of > FinalizedReplica will make DN heap usage even worse if directory scans are > scheduled more frequently. > > Another factor is that huge number of ScanInfo instances corresponding to > HDFS blocks are lingering in garbage to eat many heap memories until a full > GC takes place. > > This proposes adding JVM settings to force GC more frequently to release > DataNode heap consumed as a result of two aforementioned reasons, i.e. add > the options to HADOOP_DATANODE_OPTS > {noformat} > -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly > -XX:ConcGCThreads=8 -XX:+UseConcMarkSweepGC > {noformat} > > > Diffs > ----- > > > ambari-funtest/src/test/resources/stacks/HDP/0.2/services/HDFS/configuration/hadoop-env.xml > 2f88e06 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml > 32442d9 > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/configuration/hadoop-env.xml > 5188913 > > ambari-server/src/main/resources/stacks/HDP/2.0.6.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml > 5c1c441 > > ambari-server/src/main/resources/stacks/HDP/2.1.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml > 95fc12a > > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml > 7b92a6c > > ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/GLUSTERFS/configuration/hadoop-env.xml > f005a75 > > ambari-server/src/main/resources/stacks/HDPWIN/2.1/services/HDFS/configuration/hadoop-env.xml > 0371194 > > ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/HDFS/configuration/hadoop-env.xml > 08a1847 > > ambari-server/src/test/resources/stacks/HDP/0.2/services/HDFS/configuration/hadoop-env.xml > 228b2e0 > > ambari-server/src/test/resources/stacks_with_common_services/HDP/0.2/services/HDFS/configuration/hadoop-env.xml > b8b5f8c > > ambari-server/src/test/resources/stacks_with_extensions/HDP/0.2/services/HDFS/configuration/hadoop-env.xml > 2f9b95f > > Diff: https://reviews.apache.org/r/53248/diff/ > > > Testing > ------- > > > File Attachments > ---------------- > > AMBARI-18694.001.patch > > https://reviews.apache.org/media/uploaded/files/2016/10/28/14865347-d776-45e6-8f96-7cb6b8d071e0__AMBARI-18694.001.patch > > > Thanks, > > XIAOBING ZHOU > >
