Lightweight cluster management may be ideal in terms of ease of use, but if it doesn’t fit within the overall cluster/resource management infrastructure of where Kylin is deployed, you would end up in creating small islands of partitioned resources…
My personal preference would be (1)Mesos/Marathon which has larger installed base (2)Kubernetes based on the momentum behind it and should be optional for small scale deployments... On 11/3/15, 5:39 AM, "Adunuthula, Seshu" <[email protected]> wrote: >What about Kubernetes? More heavy weight, but eBay overall is putting all >its eggs into that basket :-) >http://googlecloudplatform.blogspot.com/2015/07/Kubernetes-V1-Released.htm >l > > >On 11/3/15, 5:26 AM, "Ted Dunning" <[email protected]> wrote: > >>You need to also decide whether Helix has momentum. The low traffic on >>the >>mailing list may be because the project is mature, but may also mean >>small >>user base. >> >>That is a very hard call to make. >> >> >>On Tue, Nov 3, 2015 at 2:00 AM, Li Yang <[email protected]> wrote: >> >>> One important goal is the cluster management should be as simple as >>> possible and introduces minimal dependency. This will allow Kylin to be >>> self-contained, easy to deploy, and is able to run on different >>>computing >>> platform. Especially we don't want to be locked-in by any platform. >>> >>> Helix is good as it's just a lightweight 700 KB lib and the only >>>dependency >>> is zookeeper (which everyone depends on nowadays). >>> >>> YARN is great but implies Hadoop, meaning Kylin may have trouble >>>running on >>> Spark alone. >>> >>> Marathon and Mesos is super great but seems a overkill. All Kylin needs >>>at >>> the moment is leader selection and task dispatching. >>> >>> >>> On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <[email protected]> >>>wrote: >>> >>> > True, but it is one of the few options for running outside of a >>>cluster. >>> > >>> > >>> > >>> > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <[email protected]> wrote: >>> > >>> > > Marathon need Mesos as extra dependency. For those who use hdp or >>> > cloudera >>> > > as sandbox for POC or some production env, Mesos does not seem to >>>be >>> the >>> > > standard component. I think it might be a concern for the users. >>> > > >>> > > Luke Han <[email protected]>于2015年11月3日周二 下午1:31写道： >>> > > >>> > > > Very interesting one, Marathon can run on Mesos which also could >>> manage >>> > > > containers... >>> > > > Worth to have a try... >>> > > > >>> > > > https://github.com/mesosphere/marathon >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > Best Regards! >>> > > > --------------------- >>> > > > >>> > > > Luke Han >>> > > > >>> > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning >>><[email protected]> >>> > > wrote: >>> > > > >>> > > > > Try Marathon then. >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <[email protected]> >>> wrote: >>> > > > > >>> > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster. >>> > > > > > >>> > > > > > Based on today's practices, Kylin's services always sit >>>beside >>> > > clusters >>> > > > > not >>> > > > > > inside to have capability to interactive with more than 2 >>> clusters >>> > > with >>> > > > > > just one Kylin deployment. >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > Best Regards! >>> > > > > > --------------------- >>> > > > > > >>> > > > > > Luke Han >>> > > > > > >>> > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra < >>> > > > [email protected]> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Why dont we just leverage Hadoop YARN? >>> > > > > > > >>> > > > > > > We could use Apache Twill [1] to wrap it as YARN >>>application. >>> > > > > > > >>> > > > > > > Helix is great but one of the limitations it could not >>> provision >>> > > new >>> > > > > > > node when not already available. >>> > > > > > > >>> > > > > > > [1] http://twill.incubator.apache.org/ >>> > > > > > > >>> > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <[email protected]> >>>wrote: >>> > > > > > > > Hi, all >>> > > > > > > > Kylin currently manages the cluster itself, so far >>>it >>> > works >>> > > > fine >>> > > > > > > since >>> > > > > > > > there are only 2 roles for the cluster, job engine and >>>query >>> > > > engine. >>> > > > > > > > In order to reduce the complexity of deployment and >>> > support >>> > > > the >>> > > > > > > coming >>> > > > > > > > streaming feature, I suggest kylin have a better cluster >>> > > > management. >>> > > > > > > > There are several candidates I think can be >>>leveraged. >>> > > Apache >>> > > > > > Helix >>> > > > > > > > seem suitable for Kylin in my opinion. >>> > > > > > > > I would love to start a discussion about cluster >>>manager >>> > of >>> > > > > Kylin, >>> > > > > > > how >>> > > > > > > > we can do it the better way, and which component we >>>should >>> > > choose. >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >
