Thanks Ted. I will check if I could get some resources to contribute.
Yong On Sat, Jan 3, 2015 at 2:13 PM, Ted Yu <[email protected]> wrote: > YARN-3003 was logged. > But it is not known when it will be implemented. > > Yong: > Please consider using zookeeper for recoding master nodes. This makes your > solution workable on Slider 0.60 > > Cheers > > On Fri, Jan 2, 2015 at 10:33 AM, Wangda Tan <[email protected]> wrote: > > > Currently, YARN doesn't have direct method to do that, but you can get a > > label-to-nodes mapping using YarnClient.getNodeToLabels(). > > > > We can consider to add a method label-to-nodes mapping in future release > > if necessary. > > > > Thanks, > > Wangda > > > > On Fri, Jan 2, 2015 at 9:11 AM, Ted Yu <[email protected]> wrote: > > > >> Thanks for the background information. > >> > >> Adding Wangda who worked on node labels feature. > >> > >> On Fri, Jan 2, 2015 at 9:04 AM, Yong Feng <[email protected]> > wrote: > >> > >>> Thanks Ted. > >>> > >>> Basically what we are doing is to ask HPC workload and BigData workload > >>> share the same infrastructure by Slider. The current way to fit Hadoop > >>> into > >>> HPC community is either on demand provisioning Hadoop cluster by HPC > >>> scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC > >>> workload > >>> scheduler directly (quite a lot papers). > >>> > >>> Besides them, we actually could ask Yarn/Slider as resource manager to > >>> manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice > >>> however I prefer Yarn/Slider due to its BigData DNA. > >>> > >>> Looking forward to getting more support from Slider and Yarn team. > >>> > >>> Thanks, > >>> > >>> Yong > >>> > >>> > >>> On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu <[email protected]> wrote: > >>> > >>> > Yong: > >>> > Let me discuss with slider / YARN developers and get back to you. > >>> > > >>> > Happy new year. > >>> > > >>> > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng <[email protected]> > >>> wrote: > >>> > > >>> > > Jon, Ted > >>> > > > >>> > > Happy new year. > >>> > > > >>> > > After read more, I understand knowing the host list or single host > of > >>> > > master role in Slider script or template is not supported by either > >>> > Slider > >>> > > or YARN. > >>> > > > >>> > > So what I can do is > >>> > > > >>> > > a) pre-define a group of hosts with label of master > >>> > > b) specify label of master for master role in app resource > >>> specfication > >>> > > c) get host list with label of master in template/scripts when > >>> deploying > >>> > > slave role > >>> > > > >>> > > However I did not find doc how to get host list with label of > master > >>> in > >>> > > Slider template/scripts like rs_hosts as follows > >>> > > > >>> > > {% for host in rs_hosts %}{{host}} > >>> > > {% endfor %} > >>> > > > >>> > > Any suggestion? > >>> > > > >>> > > Thanks, > >>> > > > >>> > > Yong > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng <[email protected]> > >>> > wrote: > >>> > > > >>> > > > Hi John > >>> > > > > >>> > > > I do expect the hosts allocated to master role of my slider > >>> application > >>> > > > are labeled as appname_rolename, but the question is who will add > >>> such > >>> > > > label and how? and how to get the hosts with the label? > >>> > > > > >>> > > > Thanks, > >>> > > > > >>> > > > Yong > >>> > > > > >>> > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron < > [email protected] > >>> > > >>> > > wrote: > >>> > > > > >>> > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t > >>> you > >>> > > know > >>> > > >> a priori the list of master hosts? > >>> > > >> > >>> > > >> > >>> > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng <[email protected]> > >>> > wrote: > >>> > > >> > >>> > > >> > Thanks Ted. > >>> > > >> > > >>> > > >> >>> knowing the host list of master role in Slider script or > >>> template > >>> > is > >>> > > >> not > >>> > > >> > supported by either Slider or YARN. > >>> > > >> > [Yong] If there is only one instance of master role, at least > I > >>> > could > >>> > > >> know > >>> > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, > I > >>> > could > >>> > > >> work > >>> > > >> > it around to just configure one master. > >>> > > >> > > >>> > > >> > Yong > >>> > > >> > > >>> > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu <[email protected] > > > >>> > wrote: > >>> > > >> > > >>> > > >> >> What's the timeframe for this migration ? > >>> > > >> >> > >>> > > >> >> If time permits, you should consider rewriting master > failover > >>> > logic > >>> > > in > >>> > > >> >> your application. > >>> > > >> >> Using zookeeper is a common technique. HBase, e.g., registers > >>> > active > >>> > > >> master > >>> > > >> >> under a well known zookeeper znode. > >>> > > >> >> > >>> > > >> >> This is much more flexible than the way you described. > >>> > > >> >> > >>> > > >> >> Again, knowing the host list of master role in Slider script > or > >>> > > >> template is > >>> > > >> >> not supported by either Slider or YARN. > >>> > > >> >> > >>> > > >> >> Cheers > >>> > > >> >> > >>> > > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng < > >>> [email protected]> > >>> > > >> wrote: > >>> > > >> >> > >>> > > >> >>> ah, rs_hosts means all the nodes managed by YARN. > >>> > > >> >>> > >>> > > >> >>> I know a little about labels feature of YARN which adds a > tag > >>> to > >>> > > host > >>> > > >> and > >>> > > >> >>> allow yarn API client to filter hosts according to the tag. > I > >>> > don't > >>> > > >> think > >>> > > >> >>> it helps my case. > >>> > > >> >>> > >>> > > >> >>> Taking a typical master/slave cluster as an example. To > >>> support > >>> > > >> >>> active-passive fail over, we need define multiple instances > >>> for > >>> > > master > >>> > > >> >> role > >>> > > >> >>> (unfortunately it does not use Zookeeper as the cluster is > >>> > > implemented > >>> > > >> >> 10+ > >>> > > >> >>> years ago before Zookeeper). In slave node, a list of mast > >>> host > >>> > are > >>> > > >> >>> specified in conf file, so that slave node could contact > >>> master. > >>> > > >> >>> > >>> > > >> >>> To deploy such cluster, we need know host list of master and > >>> > > configure > >>> > > >> >> them > >>> > > >> >>> in the conf file of slave. Changing membership of master > need > >>> > > restart > >>> > > >> >> whole > >>> > > >> >>> cluster so we will not support flex up/down role of master. > >>> > > >> >>> > >>> > > >> >>> How can I know the host list of master role in Slider script > >>> or > >>> > > >> template? > >>> > > >> >>> > >>> > > >> >>> Thanks, > >>> > > >> >>> > >>> > > >> >>> Yong > >>> > > >> >>> > >>> > > >> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu < > [email protected]> > >>> > > wrote: > >>> > > >> >>> > >>> > > >> >>>> bq. host list of all the instances of a component > >>> > > >> >>>> > >>> > > >> >>>> Are you using node labels feature of YARN ? > >>> > > >> >>>> If not, the list would comprise of all the nodes in your > YARN > >>> > > >> cluster. > >>> > > >> >>>> > >>> > > >> >>>> Cheers > >>> > > >> >>>> > >>> > > >> >>>> On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng < > >>> > [email protected]> > >>> > > >> >>> wrote: > >>> > > >> >>>> > >>> > > >> >>>>> Hi Team, > >>> > > >> >>>>> > >>> > > >> >>>>> The template of regionserver of HBase application uses > >>> following > >>> > > >> >>> scripts > >>> > > >> >>>> to > >>> > > >> >>>>> configure configuraiton file of regionserver of HBase. I > >>> did not > >>> > > >> find > >>> > > >> >>>> where > >>> > > >> >>>>> the rs_hosts is defined in any file in Slider HBase > package. > >>> > How a > >>> > > >> >>>> template > >>> > > >> >>>>> is designed is mentioned a little at the end of > >>> > > >> >>>>> > >>> > > >> >>>>> > >>> > > >> >>>> > >>> > > >> >>> > >>> > > >> >> > >>> > > >> > >>> > > > >>> > > >>> > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > >>> > > >> >>>>> , > >>> > > >> >>>>> but cannot help much. > >>> > > >> >>>>> > >>> > > >> >>>>> {% for host in rs_hosts %}{{host}} > >>> > > >> >>>>> {% endfor %} > >>> > > >> >>>>> > >>> > > >> >>>>> Does anyone know where rs_hosts is defined? I am now > >>> writing a > >>> > > >> >> package > >>> > > >> >>>> for > >>> > > >> >>>>> a HPC applicaiton and need to know host list of all the > >>> > instances > >>> > > of > >>> > > >> >> a > >>> > > >> >>>>> component to prepare configuration file of the > application. > >>> > > >> >>>>> > >>> > > >> >>>>> Thanks, > >>> > > >> >>>>> > >>> > > >> >>>>> Yong > >>> > > >> >>>>> > >>> > > >> >>>> > >>> > > >> >>> > >>> > > >> >> > >>> > > >> > >>> > > >> > >>> > > >> -- > >>> > > >> CONFIDENTIALITY NOTICE > >>> > > >> NOTICE: This message is intended for the use of the individual > or > >>> > entity > >>> > > >> to > >>> > > >> which it is addressed and may contain information that is > >>> > confidential, > >>> > > >> privileged and exempt from disclosure under applicable law. If > the > >>> > > reader > >>> > > >> of this message is not the intended recipient, you are hereby > >>> notified > >>> > > >> that > >>> > > >> any printing, copying, dissemination, distribution, disclosure > or > >>> > > >> forwarding of this communication is strictly prohibited. If you > >>> have > >>> > > >> received this communication in error, please contact the sender > >>> > > >> immediately > >>> > > >> and delete it from your system. Thank You. > >>> > > >> > >>> > > > > >>> > > > > >>> > > > >>> > > >>> > >> > >> > > > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > > to which it is addressed and may contain information that is > confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
