Looks good to me. It might also be nice to have support for key-value service-level configs. Also, is it easy to externally change the upper and lower bounds for min/max containers?
Date: Sun, 26 Jan 2014 14:17:51 -0800 Subject: Re: helix provisioning From: [email protected] To: [email protected] CC: [email protected] Here is a sample app spec. does it make sense ? appName: test configs: k1: v1 services: - name: myservice participantClass: org.apache.helix.myApp.SimpleWebserver minContainers:3 maxContainers:3 configs: - participantId: myservice_0 port: 9500 - participantId: myservice_1 port: 9501 - participantId: myservice_2 port: 9502 resources: - name: distributedLock numPartitions: 6 numReplicas: 2 rebalanceMode: FULL_AUTO stateModel: OnlineOffline tag: myservice configs: k1: v1 On Sat, Jan 18, 2014 at 11:22 AM, kishore g <[email protected]> wrote: Thats a good suggestion. We also need to think how the controller pipeline works. That means the targetprovider will now be across multiple services. Currently we have ability to plugin rebalancer and provisioners per resource. But sometimes this is too fine and might need logic that spans across resources and services. I started another thread on composite rebalancers, I think we might need a similar concept in targetprovider, provisioners. On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <[email protected]> wrote: Maybe it should be at both scopes? Sort of like putting bounds cross-app and cross-service within an app. This allows full control over an organization's physical resources. > Date: Sat, 18 Jan 2014 09:28:32 -0800 > Subject: Re: helix provisioning > From: [email protected] > To: [email protected]; [email protected] > > Thanks Swaroop for adding interfaces to specify the application spec. > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning > > From what I understand > AppName maps to cluster name > Service maps to instances, there can be multiple service components in an > application. For example, in distributed data system, there may be a > storage layer and router layer. > > The min/max containers seems to be at application level but it probably > should be at a Service scope ? > > what do you think ? > > > > > > > > > > > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <[email protected]> wrote: > > > Hi, > > > > Here is a prototype of supporting the concept of provisioning in Helix. > > This was the work done by one of the interns at LinkedIn. You can find more > > info here > > > > http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn > > > > The implementation done by intern was outside of Helix. Kanak and I took a > > stab at it to treat provisioning as a first class citizen of Helix. > > > > The work is in helix-provisioning branch > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning > > > > Most of the code is here, > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning > > > > Here is a YARN specific implementation > > > > > > https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning > > > > Its still a WIP and main goal was to see if we can abstract Resource > > Managers like YARN, Mesos, EC2 from the application. > > > > I am working on providing the steps to try out launching a Helix cluster > > on YARN. > > > > Feedback/Thoughts? Any interest in contributing to make this production > > ready ? > > > > thanks, > > Kishore G > > > > > > > > > > > > > > > >
