This is cool. Exactly what I was looking for ! Thanks Chris & Mahadev. - Milind
On 10/11/11 2:33 PM, "Chris Douglas" <[email protected]> wrote: >On Tue, Oct 11, 2011 at 2:05 PM, <[email protected]> wrote: >> As part of MPI implementation in Yarn (aka Hamster), I was looking at >> refactoring some of the functionality into node manager services, so >>that >> it can be reused by other frameworks too. (Based on the discussion with >> some folks led me to believe that suffle etc is also being planned as NM >> service, separating it from deep integration with node manager.) > >A general-purpose transfer protocol for intermediate data was designed >and scrapped for time. So was a local service allocated and shared by >resident containers. Instead, the NM supports loading modules (like >ShuffleHandler) at startup that implement auxiliary (or its >misspelling; my bad), ops-approved services, particularly shared >protocols independent of container and application allocation. > >> I could not find the documentation for building NM services, in the >>MR-279 >> architecture PDF. Does the service need to reside inside the NM jvm ? > >Yes. > >> How >> does NM talk to the service ? > >The NM loads and starts a list of services from the config by >classname. Each of these services follows the same lifecycle as the >other NM components. Each service is given a name in the config, which >is also its address for containers launched on that machine. Each >container can pass an opaque blob to the service to initialize it with >container-specific data. For the MR shuffle, these are the job >credentials. > >Don't bother listing everything that's wrong with this model. Its >principal virtue was speed of implementation. > >> How do containers discover the service ? > >They don't. They need to know the name of the service a priori. -C > >> (I >> know I can look at the code, but if there is already some docs (or even >> philosophical musings :-), it will make my life easier. >> >> Thanks, >> >> - milind >> >> --- >> Milind Bhandarkar >> Greenplum Labs, EMC >> (Disclaimer: Opinions expressed in this email are those of the author, >>and >> do not necessarily represent the views of any organization, past or >> present, the author might be affiliated with.) >> >> >> >> >
