Thanks Ted. Basically what we are doing is to ask HPC workload and BigData workload share the same infrastructure by Slider. The current way to fit Hadoop into HPC community is either on demand provisioning Hadoop cluster by HPC scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC workload scheduler directly (quite a lot papers).
Besides them, we actually could ask Yarn/Slider as resource manager to manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice however I prefer Yarn/Slider due to its BigData DNA. Looking forward to getting more support from Slider and Yarn team. Thanks, Yong On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu <[email protected]> wrote: > Yong: > Let me discuss with slider / YARN developers and get back to you. > > Happy new year. > > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng <[email protected]> wrote: > > > Jon, Ted > > > > Happy new year. > > > > After read more, I understand knowing the host list or single host of > > master role in Slider script or template is not supported by either > Slider > > or YARN. > > > > So what I can do is > > > > a) pre-define a group of hosts with label of master > > b) specify label of master for master role in app resource specfication > > c) get host list with label of master in template/scripts when deploying > > slave role > > > > However I did not find doc how to get host list with label of master in > > Slider template/scripts like rs_hosts as follows > > > > {% for host in rs_hosts %}{{host}} > > {% endfor %} > > > > Any suggestion? > > > > Thanks, > > > > Yong > > > > > > > > > > > > > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng <[email protected]> > wrote: > > > > > Hi John > > > > > > I do expect the hosts allocated to master role of my slider application > > > are labeled as appname_rolename, but the question is who will add such > > > label and how? and how to get the hosts with the label? > > > > > > Thanks, > > > > > > Yong > > > > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron <[email protected]> > > wrote: > > > > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t you > > know > > >> a priori the list of master hosts? > > >> > > >> > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng <[email protected]> > wrote: > > >> > > >> > Thanks Ted. > > >> > > > >> >>> knowing the host list of master role in Slider script or template > is > > >> not > > >> > supported by either Slider or YARN. > > >> > [Yong] If there is only one instance of master role, at least I > could > > >> know > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I > could > > >> work > > >> > it around to just configure one master. > > >> > > > >> > Yong > > >> > > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu <[email protected]> > wrote: > > >> > > > >> >> What's the timeframe for this migration ? > > >> >> > > >> >> If time permits, you should consider rewriting master failover > logic > > in > > >> >> your application. > > >> >> Using zookeeper is a common technique. HBase, e.g., registers > active > > >> master > > >> >> under a well known zookeeper znode. > > >> >> > > >> >> This is much more flexible than the way you described. > > >> >> > > >> >> Again, knowing the host list of master role in Slider script or > > >> template is > > >> >> not supported by either Slider or YARN. > > >> >> > > >> >> Cheers > > >> >> > > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng <[email protected]> > > >> wrote: > > >> >> > > >> >>> ah, rs_hosts means all the nodes managed by YARN. > > >> >>> > > >> >>> I know a little about labels feature of YARN which adds a tag to > > host > > >> and > > >> >>> allow yarn API client to filter hosts according to the tag. I > don't > > >> think > > >> >>> it helps my case. > > >> >>> > > >> >>> Taking a typical master/slave cluster as an example. To support > > >> >>> active-passive fail over, we need define multiple instances for > > master > > >> >> role > > >> >>> (unfortunately it does not use Zookeeper as the cluster is > > implemented > > >> >> 10+ > > >> >>> years ago before Zookeeper). In slave node, a list of mast host > are > > >> >>> specified in conf file, so that slave node could contact master. > > >> >>> > > >> >>> To deploy such cluster, we need know host list of master and > > configure > > >> >> them > > >> >>> in the conf file of slave. Changing membership of master need > > restart > > >> >> whole > > >> >>> cluster so we will not support flex up/down role of master. > > >> >>> > > >> >>> How can I know the host list of master role in Slider script or > > >> template? > > >> >>> > > >> >>> Thanks, > > >> >>> > > >> >>> Yong > > >> >>> > > >> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu <[email protected]> > > wrote: > > >> >>> > > >> >>>> bq. host list of all the instances of a component > > >> >>>> > > >> >>>> Are you using node labels feature of YARN ? > > >> >>>> If not, the list would comprise of all the nodes in your YARN > > >> cluster. > > >> >>>> > > >> >>>> Cheers > > >> >>>> > > >> >>>> On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng < > [email protected]> > > >> >>> wrote: > > >> >>>> > > >> >>>>> Hi Team, > > >> >>>>> > > >> >>>>> The template of regionserver of HBase application uses following > > >> >>> scripts > > >> >>>> to > > >> >>>>> configure configuraiton file of regionserver of HBase. I did not > > >> find > > >> >>>> where > > >> >>>>> the rs_hosts is defined in any file in Slider HBase package. > How a > > >> >>>> template > > >> >>>>> is designed is mentioned a little at the end of > > >> >>>>> > > >> >>>>> > > >> >>>> > > >> >>> > > >> >> > > >> > > > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > > >> >>>>> , > > >> >>>>> but cannot help much. > > >> >>>>> > > >> >>>>> {% for host in rs_hosts %}{{host}} > > >> >>>>> {% endfor %} > > >> >>>>> > > >> >>>>> Does anyone know where rs_hosts is defined? I am now writing a > > >> >> package > > >> >>>> for > > >> >>>>> a HPC applicaiton and need to know host list of all the > instances > > of > > >> >> a > > >> >>>>> component to prepare configuration file of the application. > > >> >>>>> > > >> >>>>> Thanks, > > >> >>>>> > > >> >>>>> Yong > > >> >>>>> > > >> >>>> > > >> >>> > > >> >> > > >> > > >> > > >> -- > > >> CONFIDENTIALITY NOTICE > > >> NOTICE: This message is intended for the use of the individual or > entity > > >> to > > >> which it is addressed and may contain information that is > confidential, > > >> privileged and exempt from disclosure under applicable law. If the > > reader > > >> of this message is not the intended recipient, you are hereby notified > > >> that > > >> any printing, copying, dissemination, distribution, disclosure or > > >> forwarding of this communication is strictly prohibited. If you have > > >> received this communication in error, please contact the sender > > >> immediately > > >> and delete it from your system. Thank You. > > >> > > > > > > > > >
