Sam, I think your cluster is too small for any meaningful conclusions to be made.
Sent from a remote device. Please excuse any typos... Mike Segel On Jun 18, 2013, at 3:58 AM, sam liu <[email protected]> wrote: > Hi Harsh, > > Thanks for your detailed response! Now, the efficiency of my Yarn cluster > improved a lot after increasing the reducer number(mapreduce.job.reduces) in > mapred-site.xml. But I still have some questions about the way of Yarn to > execute MRv1 job: > > 1.In Hadoop 1.x, a job will be executed by map task and reduce task together, > with a typical process(map > shuffle > reduce). In Yarn, as I know, a MRv1 > job will be executed only by ApplicationMaster. > - Yarn could run multiple kinds of jobs(MR, MPI, ...), but, MRv1 job has > special execution process(map > shuffle > reduce) in Hadoop 1.x, and how Yarn > execute a MRv1 job? still include some special MR steps in Hadoop 1.x, like > map, sort, merge, combine and shuffle? > - Do the MRv1 parameters still work for Yarn? Like mapreduce.task.io.sort.mb > and mapreduce.map.sort.spill.percent? > - What's the general process for ApplicationMaster of Yarn to execute a job? > > 2. In Hadoop 1.x, we can set the map/reduce slots by setting > 'mapred.tasktracker.map.tasks.maximum' and > 'mapred.tasktracker.reduce.tasks.maximum' > - For Yarn, above tow parameter do not work any more, as yarn uses container > instead, right? > - For Yarn, we can set the whole physical mem for a NodeManager using > 'yarn.nodemanager.resource.memory-mb'. But how to set the default size of > physical mem of a container? > - How to set the maximum size of physical mem of a container? By the > parameter of 'mapred.child.java.opts'? > > Thanks as always! > > 2013/6/9 Harsh J <[email protected]> >> Hi Sam, >> >> > - How to know the container number? Why you say it will be 22 containers >> > due to a 22 GB memory? >> >> The MR2's default configuration requests 1 GB resource each for Map >> and Reduce containers. It requests 1.5 GB for the AM container that >> runs the job, additionally. This is tunable using the properties >> Sandy's mentioned in his post. >> >> > - My machine has 32 GB memory, how many memory is proper to be assigned to >> > containers? >> >> This is a general question. You may use the same process you took to >> decide optimal number of slots in MR1 to decide this here. Every >> container is a new JVM, and you're limited by the CPUs you have there >> (if not the memory). Either increase memory requests from jobs, to >> lower # of concurrent containers at a given time (runtime change), or >> lower NM's published memory resources to control the same (config >> change). >> >> > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', >> > will other parameters for mapred-site.xml still work in yarn framework? >> > Like 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent' >> >> Yes, all of these properties will still work. Old properties specific >> to JobTracker or TaskTracker (usually found as a keyword in the config >> name) will not apply anymore. >> >> On Sun, Jun 9, 2013 at 2:21 PM, sam liu <[email protected]> wrote: >> > Hi Harsh, >> > >> > According to above suggestions, I removed the duplication of setting, and >> > reduce the value of 'yarn.nodemanager.resource.cpu-cores', >> > 'yarn.nodemanager.vcores-pcores-ratio' and >> > 'yarn.nodemanager.resource.memory-mb' to 16, 8 and 12000. Ant then, the >> > efficiency improved about 18%. I have questions: >> > >> > - How to know the container number? Why you say it will be 22 containers >> > due >> > to a 22 GB memory? >> > - My machine has 32 GB memory, how many memory is proper to be assigned to >> > containers? >> > - In mapred-site.xml, if I set 'mapreduce.framework.name' to be 'yarn', >> > will >> > other parameters for mapred-site.xml still work in yarn framework? Like >> > 'mapreduce.task.io.sort.mb' and 'mapreduce.map.sort.spill.percent' >> > >> > Thanks! >> > >> > >> > >> > 2013/6/8 Harsh J <[email protected]> >> >> >> >> Hey Sam, >> >> >> >> Did you get a chance to retry with Sandy's suggestions? The config >> >> appears to be asking NMs to use roughly 22 total containers (as >> >> opposed to 12 total tasks in MR1 config) due to a 22 GB memory >> >> resource. This could impact much, given the CPU is still the same for >> >> both test runs. >> >> >> >> On Fri, Jun 7, 2013 at 12:23 PM, Sandy Ryza <[email protected]> >> >> wrote: >> >> > Hey Sam, >> >> > >> >> > Thanks for sharing your results. I'm definitely curious about what's >> >> > causing the difference. >> >> > >> >> > A couple observations: >> >> > It looks like you've got yarn.nodemanager.resource.memory-mb in there >> >> > twice >> >> > with two different values. >> >> > >> >> > Your max JVM memory of 1000 MB is (dangerously?) close to the default >> >> > mapreduce.map/reduce.memory.mb of 1024 MB. Are any of your tasks getting >> >> > killed for running over resource limits? >> >> > >> >> > -Sandy >> >> > >> >> > >> >> > On Thu, Jun 6, 2013 at 10:21 PM, sam liu <[email protected]> wrote: >> >> >> >> >> >> The terasort execution log shows that reduce spent about 5.5 mins from >> >> >> 33% >> >> >> to 35% as below. >> >> >> 13/06/10 08:02:22 INFO mapreduce.Job: map 100% reduce 31% >> >> >> 13/06/10 08:02:25 INFO mapreduce.Job: map 100% reduce 32% >> >> >> 13/06/10 08:02:46 INFO mapreduce.Job: map 100% reduce 33% >> >> >> 13/06/10 08:08:16 INFO mapreduce.Job: map 100% reduce 35% >> >> >> 13/06/10 08:08:19 INFO mapreduce.Job: map 100% reduce 40% >> >> >> 13/06/10 08:08:22 INFO mapreduce.Job: map 100% reduce 43% >> >> >> >> >> >> Any way, below are my configurations for your reference. Thanks! >> >> >> (A) core-site.xml >> >> >> only define 'fs.default.name' and 'hadoop.tmp.dir' >> >> >> >> >> >> (B) hdfs-site.xml >> >> >> <property> >> >> >> <name>dfs.replication</name> >> >> >> <value>1</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>dfs.name.dir</name> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_name_dir</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>dfs.data.dir</name> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/dfs_data_dir</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>dfs.block.size</name> >> >> >> <value>134217728</value><!-- 128MB --> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>dfs.namenode.handler.count</name> >> >> >> <value>64</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>dfs.datanode.handler.count</name> >> >> >> <value>10</value> >> >> >> </property> >> >> >> >> >> >> (C) mapred-site.xml >> >> >> <property> >> >> >> <name>mapreduce.cluster.temp.dir</name> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_temp</value> >> >> >> <description>No description</description> >> >> >> <final>true</final> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.cluster.local.dir</name> >> >> >> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/mapreduce_local_dir</value> >> >> >> <description>No description</description> >> >> >> <final>true</final> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.child.java.opts</name> >> >> >> <value>-Xmx1000m</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.framework.name</name> >> >> >> <value>yarn</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.tasktracker.map.tasks.maximum</name> >> >> >> <value>8</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.tasktracker.reduce.tasks.maximum</name> >> >> >> <value>4</value> >> >> >> </property> >> >> >> >> >> >> >> >> >> <property> >> >> >> <name>mapreduce.tasktracker.outofband.heartbeat</name> >> >> >> <value>true</value> >> >> >> </property> >> >> >> >> >> >> (D) yarn-site.xml >> >> >> <property> >> >> >> <name>yarn.resourcemanager.resource-tracker.address</name> >> >> >> <value>node1:18025</value> >> >> >> <description>host is the hostname of the resource manager and >> >> >> port is the port on which the NodeManagers contact the Resource >> >> >> Manager. >> >> >> </description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <description>The address of the RM web application.</description> >> >> >> <name>yarn.resourcemanager.webapp.address</name> >> >> >> <value>node1:18088</value> >> >> >> </property> >> >> >> >> >> >> >> >> >> <property> >> >> >> <name>yarn.resourcemanager.scheduler.address</name> >> >> >> <value>node1:18030</value> >> >> >> <description>host is the hostname of the resourcemanager and port >> >> >> is >> >> >> the port >> >> >> on which the Applications in the cluster talk to the Resource >> >> >> Manager. >> >> >> </description> >> >> >> </property> >> >> >> >> >> >> >> >> >> <property> >> >> >> <name>yarn.resourcemanager.address</name> >> >> >> <value>node1:18040</value> >> >> >> <description>the host is the hostname of the ResourceManager and >> >> >> the >> >> >> port is the port on >> >> >> which the clients can talk to the Resource Manager. </description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.local-dirs</name> >> >> >> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_local_dir</value> >> >> >> <description>the local directories used by the >> >> >> nodemanager</description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.address</name> >> >> >> <value>0.0.0.0:18050</value> >> >> >> <description>the nodemanagers bind to this port</description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.resource.memory-mb</name> >> >> >> <value>10240</value> >> >> >> <description>the amount of memory on the NodeManager in >> >> >> GB</description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.remote-app-log-dir</name> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_app-logs</value> >> >> >> <description>directory on hdfs where the application logs are moved >> >> >> to >> >> >> </description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.log-dirs</name> >> >> >> <value>/opt/hadoop-2.0.4-alpha/temp/hadoop/yarn_nm_log</value> >> >> >> <description>the directories used by Nodemanagers as log >> >> >> directories</description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.aux-services</name> >> >> >> <value>mapreduce.shuffle</value> >> >> >> <description>shuffle service that needs to be set for Map Reduce to >> >> >> run </description> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.resourcemanager.client.thread-count</name> >> >> >> <value>64</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.resource.cpu-cores</name> >> >> >> <value>24</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.vcores-pcores-ratio</name> >> >> >> <value>3</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.resource.memory-mb</name> >> >> >> <value>22000</value> >> >> >> </property> >> >> >> >> >> >> <property> >> >> >> <name>yarn.nodemanager.vmem-pmem-ratio</name> >> >> >> <value>2.1</value> >> >> >> </property> >> >> >> >> >> >> >> >> >> >> >> >> 2013/6/7 Harsh J <[email protected]> >> >> >>> >> >> >>> Not tuning configurations at all is wrong. YARN uses memory resource >> >> >>> based scheduling and hence MR2 would be requesting 1 GB minimum by >> >> >>> default, causing, on base configs, to max out at 8 (due to 8 GB NM >> >> >>> memory resource config) total containers. Do share your configs as at >> >> >>> this point none of us can tell what it is. >> >> >>> >> >> >>> Obviously, it isn't our goal to make MR2 slower for users and to not >> >> >>> care about such things :) >> >> >>> >> >> >>> On Fri, Jun 7, 2013 at 8:45 AM, sam liu <[email protected]> >> >> >>> wrote: >> >> >>> > At the begining, I just want to do a fast comparision of MRv1 and >> >> >>> > Yarn. >> >> >>> > But >> >> >>> > they have many differences, and to be fair for comparison I did not >> >> >>> > tune >> >> >>> > their configurations at all. So I got above test results. After >> >> >>> > analyzing >> >> >>> > the test result, no doubt, I will configure them and do comparison >> >> >>> > again. >> >> >>> > >> >> >>> > Do you have any idea on current test result? I think, to compare >> >> >>> > with >> >> >>> > MRv1, >> >> >>> > Yarn is better on Map phase(teragen test), but worse on Reduce >> >> >>> > phase(terasort test). >> >> >>> > And any detailed suggestions/comments/materials on Yarn performance >> >> >>> > tunning? >> >> >>> > >> >> >>> > Thanks! >> >> >>> > >> >> >>> > >> >> >>> > 2013/6/7 Marcos Luis Ortiz Valmaseda <[email protected]> >> >> >>> >> >> >> >>> >> Why not to tune the configurations? >> >> >>> >> Both frameworks have many areas to tune: >> >> >>> >> - Combiners, Shuffle optimization, Block size, etc >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> 2013/6/6 sam liu <[email protected]> >> >> >>> >>> >> >> >>> >>> Hi Experts, >> >> >>> >>> >> >> >>> >>> We are thinking about whether to use Yarn or not in the near >> >> >>> >>> future, >> >> >>> >>> and >> >> >>> >>> I ran teragen/terasort on Yarn and MRv1 for comprison. >> >> >>> >>> >> >> >>> >>> My env is three nodes cluster, and each node has similar hardware: >> >> >>> >>> 2 >> >> >>> >>> cpu(4 core), 32 mem. Both Yarn and MRv1 cluster are set on the >> >> >>> >>> same >> >> >>> >>> env. To >> >> >>> >>> be fair, I did not make any performance tuning on their >> >> >>> >>> configurations, but >> >> >>> >>> use the default configuration values. >> >> >>> >>> >> >> >>> >>> Before testing, I think Yarn will be much better than MRv1, if >> >> >>> >>> they >> >> >>> >>> all >> >> >>> >>> use default configuration, because Yarn is a better framework than >> >> >>> >>> MRv1. >> >> >>> >>> However, the test result shows some differences: >> >> >>> >>> >> >> >>> >>> MRv1: Hadoop-1.1.1 >> >> >>> >>> Yarn: Hadoop-2.0.4 >> >> >>> >>> >> >> >>> >>> (A) Teragen: generate 10 GB data: >> >> >>> >>> - MRv1: 193 sec >> >> >>> >>> - Yarn: 69 sec >> >> >>> >>> Yarn is 2.8 times better than MRv1 >> >> >>> >>> >> >> >>> >>> (B) Terasort: sort 10 GB data: >> >> >>> >>> - MRv1: 451 sec >> >> >>> >>> - Yarn: 1136 sec >> >> >>> >>> Yarn is 2.5 times worse than MRv1 >> >> >>> >>> >> >> >>> >>> After a fast analysis, I think the direct cause might be that Yarn >> >> >>> >>> is >> >> >>> >>> much faster than MRv1 on Map phase, but much worse on Reduce >> >> >>> >>> phase. >> >> >>> >>> >> >> >>> >>> Here I have two questions: >> >> >>> >>> - Why my tests shows Yarn is worse than MRv1 for terasort? >> >> >>> >>> - What's the stratage for tuning Yarn performance? Is any >> >> >>> >>> materials? >> >> >>> >>> >> >> >>> >>> Thanks! >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> Marcos Ortiz Valmaseda >> >> >>> >> Product Manager at PDVSA >> >> >>> >> http://about.me/marcosortiz >> >> >>> >> >> >> >>> > >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Harsh J >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J >
