Thats a good suggestion. We also need to think how the controller pipeline works.
That means the targetprovider will now be across multiple services. Currently we have ability to plugin rebalancer and provisioners per resource. But sometimes this is too fine and might need logic that spans across resources and services. I started another thread on composite rebalancers, I think we might need a similar concept in targetprovider, provisioners. On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <[email protected]>wrote: > Maybe it should be at both scopes? Sort of like putting bounds cross-app > and cross-service within an app. This allows full control over an > organization's physical resources. > > > Date: Sat, 18 Jan 2014 09:28:32 -0800 > > Subject: Re: helix provisioning > > From: [email protected] > > To: [email protected]; [email protected] > > > > > Thanks Swaroop for adding interfaces to specify the application spec. > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning > > > > From what I understand > > AppName maps to cluster name > > Service maps to instances, there can be multiple service components in an > > application. For example, in distributed data system, there may be a > > storage layer and router layer. > > > > The min/max containers seems to be at application level but it probably > > should be at a Service scope ? > > > > what do you think ? > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <[email protected]> wrote: > > > > > Hi, > > > > > > Here is a prototype of supporting the concept of provisioning in Helix. > > > This was the work done by one of the interns at LinkedIn. You can find > more > > > info here > > > > > > > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn > > > > > > The implementation done by intern was outside of Helix. Kanak and I > took a > > > stab at it to treat provisioning as a first class citizen of Helix. > > > > > > The work is in helix-provisioning branch > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning > > > > > > Most of the code is here, > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning > > > > > > Here is a YARN specific implementation > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning > > > > > > Its still a WIP and main goal was to see if we can abstract Resource > > > Managers like YARN, Mesos, EC2 from the application. > > > > > > I am working on providing the steps to try out launching a Helix > cluster > > > on YARN. > > > > > > Feedback/Thoughts? Any interest in contributing to make this production > > > ready ? > > > > > > thanks, > > > Kishore G > > > > > > > > > > > > > > > > > > > > > > > > >
