Honestly your opinion is welcome, this kind of discussions are great in this small traffic dev list ;-) -- Javi Roman
Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info Apache Id: javiroman On Tue, Sep 24, 2019 at 8:55 PM yuliya Feldman <yufeld...@yahoo.com.invalid> wrote: > > I am not saying it's crazy. I was voicing my opinion. Isn't it what was the > purpose of the discussion? > It's definitely great to have UI that manages all the YARN clusters, but it's > not like UI/Web service has to be coupled/collocated with any of the Myriad > particular YARN version daemons. > It's great if you would provide write up with pros and cons for your approach > or any alternative approaches. > > > On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman > <jroman.espi...@gmail.com> wrote: > > On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman > <yufeld...@yahoo.com.invalid> wrote: > > > > Hello, > > Again I apologize for the late reply. > > I think I replied to the thread, but will add more direct notes here > > What you are proposing is to have yet another daemon that would start Yarn > > Clusters on demand within Mesos framework. > > Meaning - it would be another layer of abstraction. In this case that new > > layer would need to behave as second level scheduler and deal with third > > level scheduler(s) (RMs) to propagate offers from Mesos and keep track, etc. > > I am sure you can somehow use concept of Capacity and/or FairShare > > scheduler in your new layer to do the job. I am just not very much > > convinced that 3 layers of scheduling will be easy to > > maintain/reconcile/etc. > > Again - if I understand your design correctly. > > Would be great if you do a small write up with the proposal and have some > > simple diagram of services interactions. > > Just my 2c. > > Thanks,Yuliya > > Great, I wil do a diagram! > > Only for clarify: > > Myriad is registered as framework in Mesos master. The same thread > start the API server and the user interface. By means the user > interface you select the YARN version to run, and the scheduler get > resources from master for running RM and NMs. So you con manage as > many YARN schedulers you want. YARN as a Service. > > Maybe I am missing the point, bu I don't feel this is something so > strange, or so crazy! > > > > On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez > > <oscarf...@apache.org> wrote: > > > > Hi, > > > > I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 - > > Multiple versions of Apache Hadoop YARN as a Service. > > > > In order to implement this, we should avoid starting the Myriad framework > > from Yarn and instead starting Yarn(s) from Myriad on demand. > > > > I wanted to ask the Myriad community if this design was intended for a > > reason or if you think it's a good idea to decouple the execution of Myriad > > from the Yarn RM. With the new design, the Myriad Framework would register > > on Mesos, and then, start on demand the RM and NM that the user wants, > > allowing several Yarn clusters to run in he same Mesos, even with different > > versions. > > > > Thank you > > >
