One important goal is the cluster management should be as simple as possible and introduces minimal dependency. This will allow Kylin to be self-contained, easy to deploy, and is able to run on different computing platform. Especially we don't want to be locked-in by any platform.
Helix is good as it's just a lightweight 700 KB lib and the only dependency is zookeeper (which everyone depends on nowadays). YARN is great but implies Hadoop, meaning Kylin may have trouble running on Spark alone. Marathon and Mesos is super great but seems a overkill. All Kylin needs at the moment is leader selection and task dispatching. On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <[email protected]> wrote: > True, but it is one of the few options for running outside of a cluster. > > > > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <[email protected]> wrote: > > > Marathon need Mesos as extra dependency. For those who use hdp or > cloudera > > as sandbox for POC or some production env, Mesos does not seem to be the > > standard component. I think it might be a concern for the users. > > > > Luke Han <[email protected]>于2015年11月3日周二 下午1:31写道： > > > > > Very interesting one, Marathon can run on Mesos which also could manage > > > containers... > > > Worth to have a try... > > > > > > https://github.com/mesosphere/marathon > > > > > > > > > > > > > > > Best Regards! > > > --------------------- > > > > > > Luke Han > > > > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning <[email protected]> > > wrote: > > > > > > > Try Marathon then. > > > > > > > > > > > > > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <[email protected]> wrote: > > > > > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster. > > > > > > > > > > Based on today's practices, Kylin's services always sit beside > > clusters > > > > not > > > > > inside to have capability to interactive with more than 2 clusters > > with > > > > > just one Kylin deployment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards! > > > > > --------------------- > > > > > > > > > > Luke Han > > > > > > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Why dont we just leverage Hadoop YARN? > > > > > > > > > > > > We could use Apache Twill [1] to wrap it as YARN application. > > > > > > > > > > > > Helix is great but one of the limitations it could not provision > > new > > > > > > node when not already available. > > > > > > > > > > > > [1] http://twill.incubator.apache.org/ > > > > > > > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <[email protected]> wrote: > > > > > > > Hi, all > > > > > > > Kylin currently manages the cluster itself, so far it > works > > > fine > > > > > > since > > > > > > > there are only 2 roles for the cluster, job engine and query > > > engine. > > > > > > > In order to reduce the complexity of deployment and > support > > > the > > > > > > coming > > > > > > > streaming feature, I suggest kylin have a better cluster > > > management. > > > > > > > There are several candidates I think can be leveraged. > > Apache > > > > > Helix > > > > > > > seem suitable for Kylin in my opinion. > > > > > > > I would love to start a discussion about cluster manager > of > > > > Kylin, > > > > > > how > > > > > > > we can do it the better way, and which component we should > > choose. > > > > > > > > > > > > > > > > > > > > >
