Hi, I've made a diagram to represent the new proposed design in order to support Yarn as a service with some of the pros: https://docs.google.com/document/d/15X0-zSu0G0BDpWyndRhbvAJCXLtAbkNA45wQ_xVKOKQ
Thank you for all your comments and help On Tue, Sep 24, 2019 at 8:57 PM Javi Roman <jroman.espi...@gmail.com> wrote: > Honestly your opinion is welcome, this kind of discussions are great > in this small traffic dev list ;-) > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > Apache Id: javiroman > > On Tue, Sep 24, 2019 at 8:55 PM yuliya Feldman > <yufeld...@yahoo.com.invalid> wrote: > > > > I am not saying it's crazy. I was voicing my opinion. Isn't it what was > the purpose of the discussion? > > It's definitely great to have UI that manages all the YARN clusters, but > it's not like UI/Web service has to be coupled/collocated with any of the > Myriad particular YARN version daemons. > > It's great if you would provide write up with pros and cons for your > approach or any alternative approaches. > > > > > > On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman < > jroman.espi...@gmail.com> wrote: > > > > On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman > > <yufeld...@yahoo.com.invalid> wrote: > > > > > > Hello, > > > Again I apologize for the late reply. > > > I think I replied to the thread, but will add more direct notes here > > > What you are proposing is to have yet another daemon that would start > Yarn Clusters on demand within Mesos framework. > > > Meaning - it would be another layer of abstraction. In this case that > new layer would need to behave as second level scheduler and deal with > third level scheduler(s) (RMs) to propagate offers from Mesos and keep > track, etc. > > > I am sure you can somehow use concept of Capacity and/or FairShare > scheduler in your new layer to do the job. I am just not very much > convinced that 3 layers of scheduling will be easy to > maintain/reconcile/etc. > > > Again - if I understand your design correctly. > > > Would be great if you do a small write up with the proposal and have > some simple diagram of services interactions. > > > Just my 2c. > > > Thanks,Yuliya > > > > Great, I wil do a diagram! > > > > Only for clarify: > > > > Myriad is registered as framework in Mesos master. The same thread > > start the API server and the user interface. By means the user > > interface you select the YARN version to run, and the scheduler get > > resources from master for running RM and NMs. So you con manage as > > many YARN schedulers you want. YARN as a Service. > > > > Maybe I am missing the point, bu I don't feel this is something so > > strange, or so crazy! > > > > > > > On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez < > oscarf...@apache.org> wrote: > > > > > > Hi, > > > > > > I've started working on > https://issues.apache.org/jira/browse/MYRIAD-295 - > > > Multiple versions of Apache Hadoop YARN as a Service. > > > > > > In order to implement this, we should avoid starting the Myriad > framework > > > from Yarn and instead starting Yarn(s) from Myriad on demand. > > > > > > I wanted to ask the Myriad community if this design was intended for a > > > reason or if you think it's a good idea to decouple the execution of > Myriad > > > from the Yarn RM. With the new design, the Myriad Framework would > register > > > on Mesos, and then, start on demand the RM and NM that the user wants, > > > allowing several Yarn clusters to run in he same Mesos, even with > different > > > versions. > > > > > > Thank you > > > > > >
