then you're mostly likely swapping, reason why your tasks are not responding. you're dedicating 6GB of memory to each of the 7 tasks on a 16GB machine. You can do the math. If you have control over your cluster configuration, I suggest you set max 1 (one) mapper per machine, then assign ~ 16GB to each mapper, and use 8 compute threads.
On Fri, Mar 7, 2014 at 5:16 PM, Suijian Zhou <[email protected]> wrote: > 7, each node has a datanode and a tasktracker running on it. I attach the > full file here: > > 2014.03.07|10:13:17~/HadoopSetupTest/hadoop-1.2.1/conf>cat mapred-site.xml > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > > <property> > <name>mapred.job.tracker</name> > <value>compute-1-23:50331</value> > <description>The host and port at which the MapReduce job tracker runs. > If "local", then jobs are run in-process as a single map > and reduce task. > </description> > </property> > > <property> > <name>mapred.job.tracker.http.address</name> > <value>0.0.0.0:50332</value> > <description>The port at which the MapReduce task tracker runs. > </description> > </property> > > <property> > <name>mapred.task.tracker.http.address</name> > <value>0.0.0.0:50333</value> > <description>The port at which the MapReduce task tracker runs. > </description> > </property> > > <property> > <name>mapred.tasktracker.map.tasks.maximum</name> > <value>7</value> > <description>The maximum number of map tasks that will run > simultaneously by a task tracker. > </description> > </property> > > <property> > <name>mapred.tasktracker.reduce.tasks.maximum</name> > <value>7</value> > <description>The maximum number of reduce tasks that will run > simultaneously by a task tracker. > </description> > </property> > > <property> > <name>mapred.jobtracker.taskScheduler</name> > <value>org.apache.hadoop.mapred.FairScheduler</value> > </property> > > <property> > <name>mapred.fairscheduler.poolnameproperty</name> > <value>pool.name</value> > <description>pool name property can be specified in jobconf</description> > </property> > > <property> > <name>mapred.local.dir</name> > <value>${hadoop.tmp.dir}/mapred/local</value> > <description>The local directory where MapReduce stores intermediate > data files. May be a comma-separated list of > directories on different devices in order to spread disk I/O. > Directories that do not exist are ignored. > </description> > </property> > > <property> > <name>mapred.system.dir</name> > <value>${hadoop.tmp.dir}/system/mapred</value> > <description>The shared directory where MapReduce stores control files. > </description> > </property> > > <property> > <name>mapred.tasktracker.dns.interface</name> > <value>default</value> > <description>The name of the Network Interface from which a task > tracker should report its IP address. (e.g. eth0) > </description> > </property> > > <property> > <name>mapred.child.java.opts</name> > <value>-Xmx3600m -XX:+UseParallelGC -mx1024m -XX:MaxHeapFreeRatio=10 > -XX:MinHeapFreeRatio=10</value> > <description>Java opts for the task tracker child processes. > The following symbol, if present, will be interpolated: @taskid@ is > replaced > by current TaskID. Any other occurrences of '@' will go unchanged. > For example, to enable verbose gc logging to a file named for the taskid > in > /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: > -Xmx1024m -verbose:gc -Xloggc:/tmp/@[email protected] > The configuration variable mapred.child.ulimit can be used to control the > maximum virtual memory of the child processes. > </description> > </property> > > <property> > <name>mapred.job.reuse.jvm.num.tasks</name> > <value>-1</value> > <description>How many tasks to run per jvm. If set to -1, there is no > limit.</description> > </property> > > <property> > <name>mapred.job.tracker.handler.count</name> > <value>40</value> > <description>The number of server threads for the JobTracker. This > should be roughly 4% of the number of tasktracker nodes.</description> > </property> > > <property> > <name>mapred.jobtracker.maxtasks.per.job</name> > <value>-1</value> > <description>The maximum number of tasks for a single job. A value of > -1 indicates that there is no maximum.</description> > </property> > > <property> > <name>mapred.tasktracker.expiry.interval</name> > <value>600000</value> > <description>Time to wait get progress report from a task tracker so > that jobtracker decides the task is in progress. default is 1000*60*10 i.e. > 10 minutes</description> > </property> > > <property> > <name>mapred.task.timeout</name> > <value>0</value> > <description>Time to wait get progress report from a task tracker so > that jobtracker decides the task is in progress. default is 1000*60*10 i.e. > 10 minutes</description> > </property> > > <property> > <name>mapred.map.tasks.speculative.execution</name> > <value>false</value> > <description>set the speculative execution for map tasks</description> > </property> > > <property> > <name>mapred.reduce.tasks.speculative.execution</name> > <value>false</value> > <description>set the speculative execution for reduce > tasks</description> > </property> > > <property> > <name>mapred.hosts.exclude</name> > <value>conf/excludes</value> > </property> > > <property> > <name>mapred.job.tracker.handler.count</name> > <value>40</value> > </property> > > </configuration> > > > > 2014-03-07 9:59 GMT-06:00 Claudio Martella <[email protected]>: > > that depends on your cluster configuration. what is the maximum number of >> mappers you can have concurrently on each node? >> >> >> On Fri, Mar 7, 2014 at 4:42 PM, Suijian Zhou <[email protected]>wrote: >> >>> The current setting is: >>> <name>mapred.child.java.opts</name> >>> <value>-Xmx6144m -XX:+UseParallelGC -mx1024m -XX:MaxHeapFreeRatio=10 >>> -XX:MinHeapFreeRatio=10</value> >>> >>> Is 6144MB enough( for each task tracker)? I.e: I have 39 nodes to >>> process the 8*2GB input files. >>> >>> Best Regards, >>> Suijian >>> >>> >>> >>> 2014-03-07 9:21 GMT-06:00 Claudio Martella <[email protected]>: >>> >>> this setting won't be used by Giraph (or by any mapreduce application), >>>> but by the hadoop infrastructure itself. >>>> you should use mapred.child.java.opts instead. >>>> >>>> >>>> On Fri, Mar 7, 2014 at 4:19 PM, Suijian Zhou <[email protected]>wrote: >>>> >>>>> Hi, Claudio, >>>>> I have set the following when ran the program: >>>>> export HADOOP_DATANODE_OPTS="-Xmx10g" >>>>> and >>>>> export HADOOP_HEAPSIZE=30000 >>>>> >>>>> in hadoop-env.sh and restarted hadoop. >>>>> >>>>> Best Regards, >>>>> Suijian >>>>> >>>>> >>>>> >>>>> 2014-03-06 17:29 GMT-06:00 Claudio Martella < >>>>> [email protected]>: >>>>> >>>>> did you actually increase the heap? >>>>>> >>>>>> >>>>>> On Thu, Mar 6, 2014 at 11:43 PM, Suijian Zhou <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> I tried to process only 2 of the input files, i.e, 2GB + 2GB >>>>>>> input, the program finished successfully in 6 minutes. But as I have 39 >>>>>>> nodes, they should be enough to load and process the 8*2GB=16GB size >>>>>>> graph? Can somebody help to give some hints( Will all the nodes >>>>>>> participate >>>>>>> in graph loading from HDFS or only master node load the graph?)? Thanks! >>>>>>> >>>>>>> Best Regards, >>>>>>> Suijian >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2014-03-06 16:24 GMT-06:00 Suijian Zhou <[email protected]>: >>>>>>> >>>>>>> Hi, Experts, >>>>>>>> I'm trying to process a graph by pagerank in giraph, but the >>>>>>>> program always stucks there. >>>>>>>> There are 8 input files, each one is with size ~2GB and all copied >>>>>>>> onto HDFS. I use 39 nodes and each node has 16GB Mem and 8 cores. It >>>>>>>> keeps >>>>>>>> printing the same info(as the following) on the screen after 2 hours, >>>>>>>> looks >>>>>>>> no progress at all. What are the possible reasons? Testing small >>>>>>>> example >>>>>>>> files run without problems. Thanks! >>>>>>>> >>>>>>>> 14/03/06 16:17:42 INFO job.JobProgressTracker: Data from 39 workers >>>>>>>> - Compute superstep 0: 5854829 out of 49200000 vertices computed; 181 >>>>>>>> out >>>>>>>> of 1521 partitions computed >>>>>>>> 14/03/06 16:17:47 INFO job.JobProgressTracker: Data from 39 workers >>>>>>>> - Compute superstep 0: 5854829 out of 49200000 vertices computed; 181 >>>>>>>> out >>>>>>>> of 1521 partitions computed >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Suijian >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claudio Martella >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Claudio Martella >>>> >>>> >>> >>> >> >> >> -- >> Claudio Martella >> >> > > -- Claudio Martella
