It is arguable that "This means that YARN was not designed for long-running services, nor for short-lived interactive queries (like small and fast Spark jobs), and while it’s possible to have it schedule other kinds of workloads, this is not an ideal model".
I agree two-level scheduler is right direction. One is resource manager/scheduler and the other is workload scheduler. However it does not mean YARN is not designed for long-running service. The scheduler decision YARM made is several allocation units for allocation request. However it does not mean each allocation unit must serve one workload. The app master like slider could collect the workload request and then calculate the number of required service instance, after that negotiate with YARN to manage the desired number of service instance. The service could have its own workload scheduler to dispatch workload/request to service instance. It could also be a service on top of slider to adjust the resource request for long running service. Compared with Mesos which simply push available resource to framework, YARN has whole picture of system and is able to support more intelligent placement policy and also is able to make better decision when reclaim resource from obese application. Thanks, Yong On Thu, Mar 26, 2015 at 10:18 PM, 杨浩 <[email protected]> wrote: > Hi guys， I have found an article to support the opinions > > http://www.infoq.com/news/2015/02/Myriad > > 2015-03-03 21:27 GMT+08:00 杨浩 <[email protected]>: > > > Thanks for your sharing, and I will try it after Mesos needed. > > > > 2015-03-01 20:37 GMT+08:00 Yong Feng <[email protected]>: > > > >> A typo > >> > >> >>While Yarn and Tez are meta-framework in Yarn > >> While Slider and Tez are meta-framework in Yarn. > >> > >> Yong > >> > >> On Sun, Mar 1, 2015 at 7:36 AM, Yong Feng <[email protected]> > wrote: > >> > >> > Both Mesos and Yarn are resource manager or resource scheduler. They > use > >> > the different resource allocaiton protocol, push or request. Google > >> "Mesos > >> > Vs Yarn“, you could find more material to compare Mesos and Yarn. > Mesos > >> > argues that Yarn is "BigData Application" and is able to run on top of > >> > Yarn. We actually also could run Mesos on top of Yarn. > >> > > >> > To help people moving applicaiton to Mesos, there are several > >> > meta-framework, such as Marathon and Chronos. While Yarn and Tez are > >> > meta-framework in Yarn. > >> > > >> > I create a vagrant script to set up Mesos (Marathon/Chronos) > >> environment. > >> > If you want to play with Mesos, please have a try. > >> > > >> > https://github.com/luckyfengyong/vagrant-mesos > >> > > >> > Thanks, > >> > > >> > Yong > >> > > >> > On Sun, Mar 1, 2015 at 6:32 AM, 杨浩 <[email protected]> wrote: > >> > > >> >> Mesos can group several frameworks in the same cluster, but unlike > YARN > >> >> divides resource to different queues, it divides resources to each > >> >> framework. When using Slider, other framework or service can be put > on > >> the > >> >> Hadoop cluster as a logic cluster. I think YARN && Slider is similar > to > >> >> MESOS, am I right? > >> >> > >> > > >> > > >> > > > > >
