Here is a sample app spec. does it make sense ?
appName: test
configs:
k1: v1
services:
- name: myservice
participantClass: org.apache.helix.myApp.SimpleWebserver
minContainers:3
maxContainers:3
configs:
- participantId: myservice_0
port: 9500
- participantId: myservice_1
port: 9501
- participantId: myservice_2
port: 9502
resources:
- name: distributedLock
numPartitions: 6
numReplicas: 2
rebalanceMode: FULL_AUTO
stateModel: OnlineOffline
tag: myservice
configs:
k1: v1
On Sat, Jan 18, 2014 at 11:22 AM, kishore g <[email protected]> wrote:
> Thats a good suggestion.
>
> We also need to think how the controller pipeline works.
>
> That means the targetprovider will now be across multiple services.
> Currently we have ability to plugin rebalancer and provisioners per
> resource. But sometimes this is too fine and might need logic that spans
> across resources and services.
>
> I started another thread on composite rebalancers, I think we might need a
> similar concept in targetprovider, provisioners.
>
>
>
>
> On Sat, Jan 18, 2014 at 9:36 AM, Kanak Biscuitwala <[email protected]>wrote:
>
>> Maybe it should be at both scopes? Sort of like putting bounds cross-app
>> and cross-service within an app. This allows full control over an
>> organization's physical resources.
>>
>> > Date: Sat, 18 Jan 2014 09:28:32 -0800
>> > Subject: Re: helix provisioning
>> > From: [email protected]
>> > To: [email protected]; [email protected]
>>
>> >
>> > Thanks Swaroop for adding interfaces to specify the application spec.
>> >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=blob;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn/ApplicationSpec.java;h=16b23fa02ffbe40c18b9cff286eb5eb321947d84;hb=helix-provisioning
>> >
>> > From what I understand
>> > AppName maps to cluster name
>> > Service maps to instances, there can be multiple service components in
>> an
>> > application. For example, in distributed data system, there may be a
>> > storage layer and router layer.
>> >
>> > The min/max containers seems to be at application level but it probably
>> > should be at a Service scope ?
>> >
>> > what do you think ?
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jan 14, 2014 at 11:27 AM, kishore g <[email protected]>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > Here is a prototype of supporting the concept of provisioning in
>> Helix.
>> > > This was the work done by one of the interns at LinkedIn. You can
>> find more
>> > > info here
>> > >
>> > >
>> http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
>> > >
>> > > The implementation done by intern was outside of Helix. Kanak and I
>> took a
>> > > stab at it to treat provisioning as a first class citizen of Helix.
>> > >
>> > > The work is in helix-provisioning branch
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;h=refs/heads/helix-provisioning;hb=helix-provisioning
>> > >
>> > > Most of the code is here,
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/controller/provisioner;h=2a2650b053206ac4c521bf71899dc286cd0ef961;hb=helix-provisioning
>> > >
>> > > Here is a YARN specific implementation
>> > >
>> > >
>> > >
>> https://git-wip-us.apache.org/repos/asf?p=helix.git;a=tree;f=helix-provisioning/src/main/java/org/apache/helix/provisioning/yarn;h=93d08917feb88e06140b506eff8f498fcb588fca;hb=helix-provisioning
>> > >
>> > > Its still a WIP and main goal was to see if we can abstract Resource
>> > > Managers like YARN, Mesos, EC2 from the application.
>> > >
>> > > I am working on providing the steps to try out launching a Helix
>> cluster
>> > > on YARN.
>> > >
>> > > Feedback/Thoughts? Any interest in contributing to make this
>> production
>> > > ready ?
>> > >
>> > > thanks,
>> > > Kishore G
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>>
>
>
