Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/390237 )
Change subject: hadoop: raise jvm heap sizes for HDFS datanode and Yarn daemons ...................................................................... hadoop: raise jvm heap sizes for HDFS datanode and Yarn daemons Bug: T178876 Change-Id: I3ba5d1bf42521b1b05ab5e94cad58d0adbd7ad56 --- M hieradata/common/cdh/hadoop.yaml D hieradata/hosts/analytics1030.yaml 2 files changed, 10 insertions(+), 33 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/37/390237/1 diff --git a/hieradata/common/cdh/hadoop.yaml b/hieradata/common/cdh/hadoop.yaml index 0f2275a..ec08241 100644 --- a/hieradata/common/cdh/hadoop.yaml +++ b/hieradata/common/cdh/hadoop.yaml @@ -65,17 +65,17 @@ cdh::hadoop::mapreduce_reduce_memory_mb: 4096 # 2 * 2G cdh::hadoop::mapreduce_reduce_java_opts: -Xmx3276m # 0.8 * 2 * 2G -# This is the heap zize of YARN daemons ResourceManager and NodeManagers. +# This is the heap size of YARN daemons ResourceManager and NodeManagers. # This setting is used to configure the max heap size for both. -# The default is 1000m, we increase it to 2048m in production. -cdh::hadoop::yarn_heapsize: 2048 -cdh::hadoop::yarn_nodemanager_opts: "-Xms2048m" -cdh::hadoop::yarn_resourcemanager_opts: "-Xms2048m" +# The default is 1000m, we increase it to 4096m in production. +cdh::hadoop::yarn_heapsize: 4096 +cdh::hadoop::yarn_nodemanager_opts: "-Xms4096m" +cdh::hadoop::yarn_resourcemanager_opts: "-Xms4096m" # Heap size for HDFS datanode. -# The default is 1000m, we increase it to 2048m in production. -cdh::hadoop::hadoop_heapsize: 2048 -cdh::hadoop::hadoop_datanode_opts: "-Xms2048m" +# The default is 1000m, we increase it to 4096m in production. +cdh::hadoop::hadoop_heapsize: 4096 +cdh::hadoop::hadoop_datanode_opts: "-Xms4096m" # Yarn ApplicationMaster container size and max heap size (-Xmx) cdh::hadoop::yarn_app_mapreduce_am_resource_mb: 4096 # 2 * 2G @@ -91,12 +91,12 @@ # This is the total amount of memory that NodeManagers # will use for allocation to containers. -cdh::hadoop::yarn_nodemanager_resource_memory_mb: 57344 # 64G - 8G +cdh::hadoop::yarn_nodemanager_resource_memory_mb: 53248 # 64G - 12G # Setting _minimum_allocation_mb to 0 to allow Impala to submit small # reservation requests. cdh::hadoop::yarn_scheduler_minimum_allocation_mb: 0 -cdh::hadoop::yarn_scheduler_maximum_allocation_mb: 57344 +cdh::hadoop::yarn_scheduler_maximum_allocation_mb: 53248 # Setting minimum_allocation_vcores to 0 to allow Impala to submit small # reservation requests. cdh::hadoop::yarn_scheduler_minimum_allocation_vcores: 0 diff --git a/hieradata/hosts/analytics1030.yaml b/hieradata/hosts/analytics1030.yaml deleted file mode 100644 index 0f56f51..0000000 --- a/hieradata/hosts/analytics1030.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# Experiment to test increased Heap sizes for Datanode/Nodemanager -# Hadoop daemons from 2G to 4G -# https://phabricator.wikimedia.org/T178876 - -cdh::hadoop::yarn_heapsize: 4096 -cdh::hadoop::yarn_nodemanager_opts: "-Xms4096m" - -# Heap size for HDFS datanode. -# The default is 1000m, we increase it to 2048m in production. -cdh::hadoop::hadoop_heapsize: 4096 -cdh::hadoop::hadoop_datanode_opts: "-Xms4096m" - -# Save 12G for OS and other processes. -# Memory available for use by Hadoop jobs: 64G - 12G == 52G. -# 52G at 2G per container gives nodes with 64G RAM space for -# 26 containers. - -# This is the total amount of memory that NodeManagers -# will use for allocation to containers. -cdh::hadoop::yarn_nodemanager_resource_memory_mb: 53248 # 64G - 12G - -cdh::hadoop::yarn_scheduler_maximum_allocation_mb: 53248 - -- To view, visit https://gerrit.wikimedia.org/r/390237 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3ba5d1bf42521b1b05ab5e94cad58d0adbd7ad56 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
