I am not saying it's crazy. I was voicing my opinion. Isn't it what was the
purpose of the discussion?
It's definitely great to have UI that manages all the YARN clusters, but it's
not like UI/Web service has to be coupled/collocated with any of the Myriad
particular YARN version daemons.
It's great if you would provide write up with pros and cons for your approach
or any alternative approaches.
On Tuesday, September 24, 2019, 11:38:13 AM PDT, Javi Roman
<jroman.espi...@gmail.com> wrote:
On Tue, Sep 24, 2019 at 7:58 PM yuliya Feldman
<yufeld...@yahoo.com.invalid> wrote:
>
> Hello,
> Again I apologize for the late reply.
> I think I replied to the thread, but will add more direct notes here
> What you are proposing is to have yet another daemon that would start Yarn
> Clusters on demand within Mesos framework.
> Meaning - it would be another layer of abstraction. In this case that new
> layer would need to behave as second level scheduler and deal with third
> level scheduler(s) (RMs) to propagate offers from Mesos and keep track, etc.
> I am sure you can somehow use concept of Capacity and/or FairShare scheduler
> in your new layer to do the job. I am just not very much convinced that 3
> layers of scheduling will be easy to maintain/reconcile/etc.
> Again - if I understand your design correctly.
> Would be great if you do a small write up with the proposal and have some
> simple diagram of services interactions.
> Just my 2c.
> Thanks,Yuliya
Great, I wil do a diagram!
Only for clarify:
Myriad is registered as framework in Mesos master. The same thread
start the API server and the user interface. By means the user
interface you select the YARN version to run, and the scheduler get
resources from master for running RM and NMs. So you con manage as
many YARN schedulers you want. YARN as a Service.
Maybe I am missing the point, bu I don't feel this is something so
strange, or so crazy!
> On Wednesday, September 11, 2019, 11:55:07 PM PDT, Oscar Fernandez
><oscarf...@apache.org> wrote:
>
> Hi,
>
> I've started working on https://issues.apache.org/jira/browse/MYRIAD-295 -
> Multiple versions of Apache Hadoop YARN as a Service.
>
> In order to implement this, we should avoid starting the Myriad framework
> from Yarn and instead starting Yarn(s) from Myriad on demand.
>
> I wanted to ask the Myriad community if this design was intended for a
> reason or if you think it's a good idea to decouple the execution of Myriad
> from the Yarn RM. With the new design, the Myriad Framework would register
> on Mesos, and then, start on demand the RM and NM that the user wants,
> allowing several Yarn clusters to run in he same Mesos, even with different
> versions.
>
> Thank you
>
