There are other options as well such as Hazelcast and Infinispan. I pinged somebody on Drill to see what they say.
On Tue, Nov 3, 2015 at 2:00 AM, Li Yang <[email protected]> wrote: > One important goal is the cluster management should be as simple as > possible and introduces minimal dependency. This will allow Kylin to be > self-contained, easy to deploy, and is able to run on different computing > platform. Especially we don't want to be locked-in by any platform. > > Helix is good as it's just a lightweight 700 KB lib and the only dependency > is zookeeper (which everyone depends on nowadays). > > YARN is great but implies Hadoop, meaning Kylin may have trouble running on > Spark alone. > > Marathon and Mesos is super great but seems a overkill. All Kylin needs at > the moment is leader selection and task dispatching. > > > On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <[email protected]> wrote: > > > True, but it is one of the few options for running outside of a cluster. > > > > > > > > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <[email protected]> wrote: > > > > > Marathon need Mesos as extra dependency. For those who use hdp or > > cloudera > > > as sandbox for POC or some production env, Mesos does not seem to be > the > > > standard component. I think it might be a concern for the users. > > > > > > Luke Han <[email protected]>于2015年11月3日周二 下午1:31写道： > > > > > > > Very interesting one, Marathon can run on Mesos which also could > manage > > > > containers... > > > > Worth to have a try... > > > > > > > > https://github.com/mesosphere/marathon > > > > > > > > > > > > > > > > > > > > Best Regards! > > > > --------------------- > > > > > > > > Luke Han > > > > > > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning <[email protected]> > > > wrote: > > > > > > > > > Try Marathon then. > > > > > > > > > > > > > > > > > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <[email protected]> > wrote: > > > > > > > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster. > > > > > > > > > > > > Based on today's practices, Kylin's services always sit beside > > > clusters > > > > > not > > > > > > inside to have capability to interactive with more than 2 > clusters > > > with > > > > > > just one Kylin deployment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards! > > > > > > --------------------- > > > > > > > > > > > > Luke Han > > > > > > > > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Why dont we just leverage Hadoop YARN? > > > > > > > > > > > > > > We could use Apache Twill [1] to wrap it as YARN application. > > > > > > > > > > > > > > Helix is great but one of the limitations it could not > provision > > > new > > > > > > > node when not already available. > > > > > > > > > > > > > > [1] http://twill.incubator.apache.org/ > > > > > > > > > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <[email protected]> wrote: > > > > > > > > Hi, all > > > > > > > > Kylin currently manages the cluster itself, so far it > > works > > > > fine > > > > > > > since > > > > > > > > there are only 2 roles for the cluster, job engine and query > > > > engine. > > > > > > > > In order to reduce the complexity of deployment and > > support > > > > the > > > > > > > coming > > > > > > > > streaming feature, I suggest kylin have a better cluster > > > > management. > > > > > > > > There are several candidates I think can be leveraged. > > > Apache > > > > > > Helix > > > > > > > > seem suitable for Kylin in my opinion. > > > > > > > > I would love to start a discussion about cluster manager > > of > > > > > Kylin, > > > > > > > how > > > > > > > > we can do it the better way, and which component we should > > > choose. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
