On Tue, Oct 11, 2011 at 2:05 PM, <[email protected]> wrote: > As part of MPI implementation in Yarn (aka Hamster), I was looking at > refactoring some of the functionality into node manager services, so that > it can be reused by other frameworks too. (Based on the discussion with > some folks led me to believe that suffle etc is also being planned as NM > service, separating it from deep integration with node manager.)
A general-purpose transfer protocol for intermediate data was designed and scrapped for time. So was a local service allocated and shared by resident containers. Instead, the NM supports loading modules (like ShuffleHandler) at startup that implement auxiliary (or its misspelling; my bad), ops-approved services, particularly shared protocols independent of container and application allocation. > I could not find the documentation for building NM services, in the MR-279 > architecture PDF. Does the service need to reside inside the NM jvm ? Yes. > How > does NM talk to the service ? The NM loads and starts a list of services from the config by classname. Each of these services follows the same lifecycle as the other NM components. Each service is given a name in the config, which is also its address for containers launched on that machine. Each container can pass an opaque blob to the service to initialize it with container-specific data. For the MR shuffle, these are the job credentials. Don't bother listing everything that's wrong with this model. Its principal virtue was speed of implementation. > How do containers discover the service ? They don't. They need to know the name of the service a priori. -C > (I > know I can look at the code, but if there is already some docs (or even > philosophical musings :-), it will make my life easier. > > Thanks, > > - milind > > --- > Milind Bhandarkar > Greenplum Labs, EMC > (Disclaimer: Opinions expressed in this email are those of the author, and > do not necessarily represent the views of any organization, past or > present, the author might be affiliated with.) > > > >
