I checked my MR admin page running on <host>:50030:jobtracker.jsp this is what
I am learning.
I run 2-core processor, so the admin page told me that my max map/reduce slot
capacity was 14 or so I assume 7 nodes x 2slots.
I did not touch the property of .map.tasks. It seemed that MR set it nicely to
14 automatically, processing 45 of my maps. However, I increased reduce task
from the default 1 to 14 in the mapred-site.xml of the JobTracker.
<property>
<name>mapred.reduce.tasks</name>
<value>14</value>
</property>
The job worked really fast.
Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map
SlotsOccupied Reduce SlotsReserved Map SlotsReserved Reduce SlotsMap Task
CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted NodesExcluded
Nodes14017<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=active>1400014144.000<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=blacklisted>1<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=excluded>
From: kapil bhosale [mailto:[email protected]]
Sent: Thursday, November 01, 2012 3:51 AM
To: [email protected]
Subject: Re: Map Reduce slot
Hi Uddipan,
In my Opinion
Hadoop Jobtracker Interface provides every Details Including
# number of Map slots
# number of Reducer slots
and other Infromation can also be found in Log files generated.
Which can also be explored through the same Interface.
Jobtracker address : Masternode:your_port/jobtracker.jsp<http://>
Other useful details can also be found.
-
kapil
On Thu, Nov 1, 2012 at 1:11 PM, Uddipan Mukherjee
<[email protected]<mailto:[email protected]>> wrote:
Hi Hadoop Gurus.
What is the good way to know following information in my Hadoop cluster.
# number of Map slots
# number of Reducer slots
JVM Heapsize for Map task
JVM heapsize for reduce task
reuse JVM flag
Thanks and Regards
Uddipan
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are
not
to copy, disclose, or distribute this e-mail or its contents to any other
person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has
taken
every reasonable precaution to minimize this risk, but is not liable for any
damage
you may sustain as a result of any virus in this e-mail. You should carry out
your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this
e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
--
kap's
NOTICE: This e-mail message and any attachments are confidential, subject to
copyright and may be privileged. Any unauthorized use, copying or disclosure is
prohibited. If you are not the intended recipient, please delete and contact
the sender immediately. Please consider the environment before printing this
e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont
confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le
secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est
interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel,
supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ?
l'environnement avant d'imprimer le pr?sent courriel
