On Jun 14, 2011, at 6:31 PM, Praveen Sripati wrote:
Hi,
I have gone through MapReduce NextGen Blog entries and JIRA and have
the
following queries
There is a single API between the Scheduler and the
ApplicationMaster:
(List <Container> newContainers, List <ContainerStatus>
containerStatuses) allocate (List <ResourceRequest> ask,
List<Container>
release)
The AM ask for specific resources via a list of ResourceRequests
(ask)
and releases unnecessary Containers which were allocated by the
Scheduler.
The response contains a list of newly allocated Containers and the
statuses of application-specific Containers that completed since the
previous interaction between the AM and the RM.
Q) If split-0 is is available in host1, host2 and host3, can
ApplicationMaster request a scheduler for a container on host1 or
host2 or
host3? This way the scheduler can allocate the resources more
effectively.
Yes, absolutely.
Q) In a cluster there might be nodes of different capacities, how
will the
scheduler know that a particular node has 4 GB and another has 16 GB
RAM
before allocating the resources to the ApplicationMaster?
The NodeManager informs the RM about its capabilities on registration.
The RM allocates appropriate resources to the AM(s).
Q) Are the unnecessary containers (List<Container> release) in the
request
released by the ApplicationMaster the ones rejected by the
ApplicationMaster
or those on which the map/reduce tasks have been completed?
Only unused ones.
Q) What does the following in the response contain - "List
<ContainerStatus>
containerStatuses"?
Status for completed completed containers.
Q) Once the ApplicationMaster gets the list of the new containers
from the
Scheduler, what is the interaction between the ApplicationMaster and
the
Node Manager? Will the ApplicationMaster ask the Node Manager on the
different nodes to launch/monitor the map/reduce tasks in those
containers?
No, the AM directly monitors the containers via an application-
specific protocol.
For MR applications we use TaskUmbilicalProtocol.
The NM just monitors the unix process and informs the RM on exit of
the unix process.
Q) Does the Scheduler ask the Node Manager to create the containers
on the
different nodes?
No, the Scheduler allocates them to the respective AMs who then launch
the container by talking to the NM.
The NM can securely verify the authenticity of the 'container launch'
request, including the resources allocated to the container.
The resource requests are also aggregated by racks and then by the
special any (*) for all containers. All resource requests are
subject to
change via the delta protocol.
Q) Does (*) mean that the ApplicationMaster is OK with a container
in any
rack/host? This might be applicable for Reduce tasks.
Yes.
Hope this helps.
Arun
