Re: Apache Slider and Apache Myriad
+1 > On Apr 24, 2018, at 11:41 AM, Brandon Gullawrote: > > Possible, maybe, the right thing to do? Probably not. > > There are multiple container orchestration platforms these days, and the > pecking order is k8s -> mesos -> yarn. DC/OS has already implemented > support for kubernetes, but purely because it is seeing k8s as a competitor > and Mesosphere desires to make it as easy as possible to support hybrid > cloud solutions for their customers. While they may one day add k8s > scheduling functionality to Marathon (speculative), I highly doubt that > they would do the same for yarn. > > A lot of Hadoop-ecosystems are now adding support for kubernetes directly, > look at the latest version of spark. You go where the customers are, and > YARN is still in third place and I would be shocked to see that change any > time soon. > > > > On Tue, Apr 24, 2018 at 2:27 PM, Javi Roman > wrote: > >> Probably you are right, however thinking that we have Apache Mesos as >> coarse-grained scheduling, and generally we run at least one >> fine-grained "general purpose" or "meta-scheduler" such as Marathon, >> Aurora or even Kubernetes as framework. Why not apply the same >> rationale for Myriad, YARN as general purpose framework. >> >> I know this is probably crazy, but imagine a kind of DC/OS flavor with >> Apache Mesos + Apache Myriad + Apache Hadoop running as foundation for >> this flavor (without Marathon or kubernetes frameworks), you run >> Mesos, Myriad and Hadoop as system services, and the user can run >> Hadoop applications, Docker applications and whatever application. >> When you click the "Services" tab at DC/OS interface you get the >> Myriad UI for run applications. >> >> It's crazy I know, but it would be possible? I really don't know. >> >> >> -- >> Javi Roman >> >> Twitter: @javiromanrh >> GitHub: github.com/javiroman >> Linkedin: es.linkedin.com/in/javiroman >> Big Data Blog: dataintensive.info >> >> >> On Tue, Apr 24, 2018 at 8:04 PM, yuliya Feldman >> wrote: >>> Are you suggesting using Myriad as meta-scheduler for Mesos? >>> If yes, then:1. Why would one need Mesos or Myriad at all - use YARN for >> everything?2. Mesos scheduler is much more robust then YARN scheduler as >> it's based on the offers - it was done having multiple subsystems in mind >> unlike YARN >>> AFAIK Marathon serves very thin purpose in Myriad - literally just to >> spin up Myriad Master >>> Just my 2c. >>> Thanks,Yuliya >>>On Tuesday, April 24, 2018, 10:56:37 AM PDT, Javi Roman < >> jroman.espi...@gmail.com> wrote: >>> >>> Hi! >>> >>> Thinking a little more deeply this YARN feature (Hadoop YARN Service >>> framework) could affect Apache Myriad regarding new development goals. >>> Apache Myriad based on YARN 3.x could be a possible substitute for >>> general purpose meta-scheduling Mesos frameworks such as Marathon or >>> Apache Aurora. We can rewrite the Myriad UI in order to allow run >>> YARN native applications, Docker containers or regular applications. >>> >>> It's just a crazy idea: DC/OS + Apache Myriad (natively installed as >>> systemd unit) instead of Marathon, working as meta-scheduler for >>> Mesos. >>> >>> What do you think? >>> -- >>> Javi Roman >>> >>> Twitter: @javiromanrh >>> GitHub: github.com/javiroman >>> Linkedin: es.linkedin.com/in/javiroman >>> Big Data Blog: dataintensive.info >>> >>> >>> On Mon, Apr 23, 2018 at 8:02 AM, Javi Roman >> wrote: Hi Darin, You are right Slider is a kind of framework for long running applications into YARN, and it's a way of running applications on YARN without changing the application code, there is no need to develop a custom Application Master or other YARN code. Apparently you can run unmodified applications on YARN via Slider. This shift towards Hadoop code (instead of separate project, incubator slider) doesn't affect Myriad at all. However I was thinking any kind of collaboration with Slider project, and this collaboration is unclear right now, because of fuzzy continuity of Slider. -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson >> wrote: > Javi can you explain how this effects myriad? My understanding is that > slider is similar to marathon but for yarn. > > On Sat, Apr 21, 2018, 2:23 AM Javi Roman >> wrote: > >> FYI >> >> Regarding the recent conversation about YARN applications development >> via Apache Slider, as a different approach for creating data services >> applications on Apache Mesos (via Myriad,) "article 4" here [1], will >> probably have to wait until the resolution of
Re: Apache Slider and Apache Myriad
Possible, maybe, the right thing to do? Probably not. There are multiple container orchestration platforms these days, and the pecking order is k8s -> mesos -> yarn. DC/OS has already implemented support for kubernetes, but purely because it is seeing k8s as a competitor and Mesosphere desires to make it as easy as possible to support hybrid cloud solutions for their customers. While they may one day add k8s scheduling functionality to Marathon (speculative), I highly doubt that they would do the same for yarn. A lot of Hadoop-ecosystems are now adding support for kubernetes directly, look at the latest version of spark. You go where the customers are, and YARN is still in third place and I would be shocked to see that change any time soon. On Tue, Apr 24, 2018 at 2:27 PM, Javi Romanwrote: > Probably you are right, however thinking that we have Apache Mesos as > coarse-grained scheduling, and generally we run at least one > fine-grained "general purpose" or "meta-scheduler" such as Marathon, > Aurora or even Kubernetes as framework. Why not apply the same > rationale for Myriad, YARN as general purpose framework. > > I know this is probably crazy, but imagine a kind of DC/OS flavor with > Apache Mesos + Apache Myriad + Apache Hadoop running as foundation for > this flavor (without Marathon or kubernetes frameworks), you run > Mesos, Myriad and Hadoop as system services, and the user can run > Hadoop applications, Docker applications and whatever application. > When you click the "Services" tab at DC/OS interface you get the > Myriad UI for run applications. > > It's crazy I know, but it would be possible? I really don't know. > > > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > > > On Tue, Apr 24, 2018 at 8:04 PM, yuliya Feldman > wrote: > > Are you suggesting using Myriad as meta-scheduler for Mesos? > > If yes, then:1. Why would one need Mesos or Myriad at all - use YARN for > everything?2. Mesos scheduler is much more robust then YARN scheduler as > it's based on the offers - it was done having multiple subsystems in mind > unlike YARN > > AFAIK Marathon serves very thin purpose in Myriad - literally just to > spin up Myriad Master > > Just my 2c. > > Thanks,Yuliya > > On Tuesday, April 24, 2018, 10:56:37 AM PDT, Javi Roman < > jroman.espi...@gmail.com> wrote: > > > > Hi! > > > > Thinking a little more deeply this YARN feature (Hadoop YARN Service > > framework) could affect Apache Myriad regarding new development goals. > > Apache Myriad based on YARN 3.x could be a possible substitute for > > general purpose meta-scheduling Mesos frameworks such as Marathon or > > Apache Aurora. We can rewrite the Myriad UI in order to allow run > > YARN native applications, Docker containers or regular applications. > > > > It's just a crazy idea: DC/OS + Apache Myriad (natively installed as > > systemd unit) instead of Marathon, working as meta-scheduler for > > Mesos. > > > > What do you think? > > -- > > Javi Roman > > > > Twitter: @javiromanrh > > GitHub: github.com/javiroman > > Linkedin: es.linkedin.com/in/javiroman > > Big Data Blog: dataintensive.info > > > > > > On Mon, Apr 23, 2018 at 8:02 AM, Javi Roman > wrote: > >> Hi Darin, > >> > >> You are right Slider is a kind of framework for long running > >> applications into YARN, and it's a way of running applications on YARN > >> without changing the application code, there is no need to develop a > >> custom Application Master or other YARN code. Apparently you can run > >> unmodified applications on YARN via Slider. > >> > >> This shift towards Hadoop code (instead of separate project, incubator > >> slider) doesn't affect Myriad at all. However I was thinking any kind > >> of collaboration with Slider project, and this collaboration is > >> unclear right now, because of fuzzy continuity of Slider. > >> -- > >> Javi Roman > >> > >> Twitter: @javiromanrh > >> GitHub: github.com/javiroman > >> Linkedin: es.linkedin.com/in/javiroman > >> Big Data Blog: dataintensive.info > >> > >> > >> On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson > wrote: > >>> Javi can you explain how this effects myriad? My understanding is that > >>> slider is similar to marathon but for yarn. > >>> > >>> On Sat, Apr 21, 2018, 2:23 AM Javi Roman > wrote: > >>> > FYI > > Regarding the recent conversation about YARN applications development > via Apache Slider, as a different approach for creating data services > applications on Apache Mesos (via Myriad,) "article 4" here [1], will > probably have to wait until the resolution of this discussion at > Apache Slider list [2]. Basically the Slider community is discussing > about the retirement of the project, because the Slider core is being >
Re: Apache Slider and Apache Myriad
Probably you are right, however thinking that we have Apache Mesos as coarse-grained scheduling, and generally we run at least one fine-grained "general purpose" or "meta-scheduler" such as Marathon, Aurora or even Kubernetes as framework. Why not apply the same rationale for Myriad, YARN as general purpose framework. I know this is probably crazy, but imagine a kind of DC/OS flavor with Apache Mesos + Apache Myriad + Apache Hadoop running as foundation for this flavor (without Marathon or kubernetes frameworks), you run Mesos, Myriad and Hadoop as system services, and the user can run Hadoop applications, Docker applications and whatever application. When you click the "Services" tab at DC/OS interface you get the Myriad UI for run applications. It's crazy I know, but it would be possible? I really don't know. -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Tue, Apr 24, 2018 at 8:04 PM, yuliya Feldmanwrote: > Are you suggesting using Myriad as meta-scheduler for Mesos? > If yes, then:1. Why would one need Mesos or Myriad at all - use YARN for > everything?2. Mesos scheduler is much more robust then YARN scheduler as it's > based on the offers - it was done having multiple subsystems in mind unlike > YARN > AFAIK Marathon serves very thin purpose in Myriad - literally just to spin up > Myriad Master > Just my 2c. > Thanks,Yuliya > On Tuesday, April 24, 2018, 10:56:37 AM PDT, Javi Roman > wrote: > > Hi! > > Thinking a little more deeply this YARN feature (Hadoop YARN Service > framework) could affect Apache Myriad regarding new development goals. > Apache Myriad based on YARN 3.x could be a possible substitute for > general purpose meta-scheduling Mesos frameworks such as Marathon or > Apache Aurora. We can rewrite the Myriad UI in order to allow run > YARN native applications, Docker containers or regular applications. > > It's just a crazy idea: DC/OS + Apache Myriad (natively installed as > systemd unit) instead of Marathon, working as meta-scheduler for > Mesos. > > What do you think? > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > > > On Mon, Apr 23, 2018 at 8:02 AM, Javi Roman wrote: >> Hi Darin, >> >> You are right Slider is a kind of framework for long running >> applications into YARN, and it's a way of running applications on YARN >> without changing the application code, there is no need to develop a >> custom Application Master or other YARN code. Apparently you can run >> unmodified applications on YARN via Slider. >> >> This shift towards Hadoop code (instead of separate project, incubator >> slider) doesn't affect Myriad at all. However I was thinking any kind >> of collaboration with Slider project, and this collaboration is >> unclear right now, because of fuzzy continuity of Slider. >> -- >> Javi Roman >> >> Twitter: @javiromanrh >> GitHub: github.com/javiroman >> Linkedin: es.linkedin.com/in/javiroman >> Big Data Blog: dataintensive.info >> >> >> On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson >> wrote: >>> Javi can you explain how this effects myriad? My understanding is that >>> slider is similar to marathon but for yarn. >>> >>> On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: >>> FYI Regarding the recent conversation about YARN applications development via Apache Slider, as a different approach for creating data services applications on Apache Mesos (via Myriad,) "article 4" here [1], will probably have to wait until the resolution of this discussion at Apache Slider list [2]. Basically the Slider community is discussing about the retirement of the project, because the Slider core is being included in Hadoop 3 [3]. [1] https://goo.gl/RVkFUF [2] https://goo.gl/CPR2TF [3] https://issues.apache.org/jira/browse/YARN-5079 -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info >
Re: Apache Slider and Apache Myriad
Are you suggesting using Myriad as meta-scheduler for Mesos? If yes, then:1. Why would one need Mesos or Myriad at all - use YARN for everything?2. Mesos scheduler is much more robust then YARN scheduler as it's based on the offers - it was done having multiple subsystems in mind unlike YARN AFAIK Marathon serves very thin purpose in Myriad - literally just to spin up Myriad Master Just my 2c. Thanks,Yuliya On Tuesday, April 24, 2018, 10:56:37 AM PDT, Javi Romanwrote: Hi! Thinking a little more deeply this YARN feature (Hadoop YARN Service framework) could affect Apache Myriad regarding new development goals. Apache Myriad based on YARN 3.x could be a possible substitute for general purpose meta-scheduling Mesos frameworks such as Marathon or Apache Aurora. We can rewrite the Myriad UI in order to allow run YARN native applications, Docker containers or regular applications. It's just a crazy idea: DC/OS + Apache Myriad (natively installed as systemd unit) instead of Marathon, working as meta-scheduler for Mesos. What do you think? -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Mon, Apr 23, 2018 at 8:02 AM, Javi Roman wrote: > Hi Darin, > > You are right Slider is a kind of framework for long running > applications into YARN, and it's a way of running applications on YARN > without changing the application code, there is no need to develop a > custom Application Master or other YARN code. Apparently you can run > unmodified applications on YARN via Slider. > > This shift towards Hadoop code (instead of separate project, incubator > slider) doesn't affect Myriad at all. However I was thinking any kind > of collaboration with Slider project, and this collaboration is > unclear right now, because of fuzzy continuity of Slider. > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > > > On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson > wrote: >> Javi can you explain how this effects myriad? My understanding is that >> slider is similar to marathon but for yarn. >> >> On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: >> >>> FYI >>> >>> Regarding the recent conversation about YARN applications development >>> via Apache Slider, as a different approach for creating data services >>> applications on Apache Mesos (via Myriad,) "article 4" here [1], will >>> probably have to wait until the resolution of this discussion at >>> Apache Slider list [2]. Basically the Slider community is discussing >>> about the retirement of the project, because the Slider core is being >>> included in Hadoop 3 [3]. >>> >>> [1] https://goo.gl/RVkFUF >>> [2] https://goo.gl/CPR2TF >>> [3] https://issues.apache.org/jira/browse/YARN-5079 >>> >>> -- >>> Javi Roman >>> >>> Twitter: @javiromanrh >>> GitHub: github.com/javiroman >>> Linkedin: es.linkedin.com/in/javiroman >>> Big Data Blog: dataintensive.info >>>
Re: Apache Slider and Apache Myriad
Hi! Thinking a little more deeply this YARN feature (Hadoop YARN Service framework) could affect Apache Myriad regarding new development goals. Apache Myriad based on YARN 3.x could be a possible substitute for general purpose meta-scheduling Mesos frameworks such as Marathon or Apache Aurora. We can rewrite the Myriad UI in order to allow run YARN native applications, Docker containers or regular applications. It's just a crazy idea: DC/OS + Apache Myriad (natively installed as systemd unit) instead of Marathon, working as meta-scheduler for Mesos. What do you think? -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Mon, Apr 23, 2018 at 8:02 AM, Javi Romanwrote: > Hi Darin, > > You are right Slider is a kind of framework for long running > applications into YARN, and it's a way of running applications on YARN > without changing the application code, there is no need to develop a > custom Application Master or other YARN code. Apparently you can run > unmodified applications on YARN via Slider. > > This shift towards Hadoop code (instead of separate project, incubator > slider) doesn't affect Myriad at all. However I was thinking any kind > of collaboration with Slider project, and this collaboration is > unclear right now, because of fuzzy continuity of Slider. > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > > > On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnson > wrote: >> Javi can you explain how this effects myriad? My understanding is that >> slider is similar to marathon but for yarn. >> >> On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: >> >>> FYI >>> >>> Regarding the recent conversation about YARN applications development >>> via Apache Slider, as a different approach for creating data services >>> applications on Apache Mesos (via Myriad,) "article 4" here [1], will >>> probably have to wait until the resolution of this discussion at >>> Apache Slider list [2]. Basically the Slider community is discussing >>> about the retirement of the project, because the Slider core is being >>> included in Hadoop 3 [3]. >>> >>> [1] https://goo.gl/RVkFUF >>> [2] https://goo.gl/CPR2TF >>> [3] https://issues.apache.org/jira/browse/YARN-5079 >>> >>> -- >>> Javi Roman >>> >>> Twitter: @javiromanrh >>> GitHub: github.com/javiroman >>> Linkedin: es.linkedin.com/in/javiroman >>> Big Data Blog: dataintensive.info >>>
Re: Apache Slider and Apache Myriad
Hi Darin, You are right Slider is a kind of framework for long running applications into YARN, and it's a way of running applications on YARN without changing the application code, there is no need to develop a custom Application Master or other YARN code. Apparently you can run unmodified applications on YARN via Slider. This shift towards Hadoop code (instead of separate project, incubator slider) doesn't affect Myriad at all. However I was thinking any kind of collaboration with Slider project, and this collaboration is unclear right now, because of fuzzy continuity of Slider. -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info On Mon, Apr 23, 2018 at 1:25 AM, Darin Johnsonwrote: > Javi can you explain how this effects myriad? My understanding is that > slider is similar to marathon but for yarn. > > On Sat, Apr 21, 2018, 2:23 AM Javi Roman wrote: > >> FYI >> >> Regarding the recent conversation about YARN applications development >> via Apache Slider, as a different approach for creating data services >> applications on Apache Mesos (via Myriad,) "article 4" here [1], will >> probably have to wait until the resolution of this discussion at >> Apache Slider list [2]. Basically the Slider community is discussing >> about the retirement of the project, because the Slider core is being >> included in Hadoop 3 [3]. >> >> [1] https://goo.gl/RVkFUF >> [2] https://goo.gl/CPR2TF >> [3] https://issues.apache.org/jira/browse/YARN-5079 >> >> -- >> Javi Roman >> >> Twitter: @javiromanrh >> GitHub: github.com/javiroman >> Linkedin: es.linkedin.com/in/javiroman >> Big Data Blog: dataintensive.info >>
Apache Slider and Apache Myriad
FYI Regarding the recent conversation about YARN applications development via Apache Slider, as a different approach for creating data services applications on Apache Mesos (via Myriad,) "article 4" here [1], will probably have to wait until the resolution of this discussion at Apache Slider list [2]. Basically the Slider community is discussing about the retirement of the project, because the Slider core is being included in Hadoop 3 [3]. [1] https://goo.gl/RVkFUF [2] https://goo.gl/CPR2TF [3] https://issues.apache.org/jira/browse/YARN-5079 -- Javi Roman Twitter: @javiromanrh GitHub: github.com/javiroman Linkedin: es.linkedin.com/in/javiroman Big Data Blog: dataintensive.info