Yes, the intention of bringing in the separation between SPI and API
is exactly to allow for separation of two things
(a) how a model is presented to the user --- API concern
(b) how the model is stored -- SPI concern
Sure, lets start with the APIs and see it would shape. The only
request I have is lets define the logical model, then go to how that
logical model is created through the Admin API. I think getting the
logical model right will take more time and thought, than the Admin
API.
Here is an example to illustrate what I am trying to convey. This is
just to illustrate my thought process not to suggest this is the exact
API so please bear with me. I am deliberately leaving out exceptions
because the API is to give an idea of what I am talking about.
public interface Cluster {
List<Instance> getParticipants();
String getName();
Instance getLocalParticipant();
...
}
public interface Instance{
boolean isLocal();
String getId();
int getPort();
String getHostName();
List<Resource> getResources();
}
Sandeep
On Sun, Feb 23, 2014 at 5:31 PM, kishore g <[email protected]> wrote:
> Basically what we have is now is that the logical model exposed to user
> more or less matches the physical model (how data is stored in ZK). We can
> definitely separate these two (I think this is what Sandeep is suggesting).
> I am not sure how invasive and hard it would be to get that right. We
> definitely need to make sure that we are backward compatible with physical
> model.
>
>
> Given this, does it make sense to define two models, physical and logical.
> I think what we have currently in our model package can be called as
> physical model and we can work on logical model. I would say we are
> actually pretty close to what we want in 0.7.0.
>
> In terms of entities. Yes the high level concepts are cluster, instance,
> resource, partition. Don't see a need for User.
>
> Entities have what we can probably refer to as attributes
> - configurations
> --- static/not yet realized
> --- runtime configuration (this is more or less the state + additional
> stuff)
> --- user configuration (Helix has nothing to do with it). This again has
> static and dynamic
> - Roles, will only be associated with instances, i dont see any other
> entity having roles
> - State, for now only partition/instance has a state. But we can define a
> state at resource and cluster scope
>
> The problem with configuration is that it becomes too generic, especially
> with Resource. We need to make this as specific as possible.
>
> I feel the best way to come up with these api's is to use an example
> similar to what Kanak wrote. That will give us an idea of how user facing
> api will look like. These are the operations a user performs
>
> -- Create, read, update, delete a cluster, resource, partition, instance
> and its configuration
> -- Register listeners to changes in each of above
> -- Handle state transitions.
>
> Lets define the set of operations the user performs and write an example on
> how it looks and should look. We need to see how all these tie together in
> the flow.
>
> Sandeep at a high level what you say makes sense. I think rest of us know a
> lot of the internals which means our views are biased, which is both good
> and bad :-).
>
> If it helps, we can start defining the classes and see how it looks.
>
> What do you think.
>
> thanks,
> Kishore G
>
>
>
>
>
>
>
>
>
>
> On Sun, Feb 23, 2014 at 11:42 AM, Sandeep Nayak <[email protected]> wrote:
>
>> I am not entirely conversant with the internals so dont have enough to
>> sway one way or the other.
>>
>> But it seems like we are mixing two separate concerns here. Modeling
>> an entity to be surfaced in an API should have no bearing on how it is
>> passivated. You can still store the data in zookeeper the way you are
>> doing so today. Having an SPI which talks to how you passivate data
>> v/s an API to how the user interacts with data are two separate
>> concerns in my view.
>>
>> Sandeep
>>
>> On Sun, Feb 23, 2014 at 11:33 AM, Kanak Biscuitwala <[email protected]>
>> wrote:
>> >
>> > Changing/removing entities in ZK is highly undesirable. We still want
>> old-version controllers to manage new-version participants, and vice versa.
>> Making a change like that would require system-wide upgrades and downtime.
>> >
>> > What we can do is change how the entities are exposed to the users
>> through an API. That is really the goal here: avoid changing the backing
>> store, but rethink how the user interacts with it.
>> >
>> > Kanak
>> >
>> > ----------------------------------------
>> >> Date: Sun, 23 Feb 2014 11:22:55 -0800
>> >> Subject: Re: [DISCUSS] New Helix Api
>> >> From: [email protected]
>> >> To: [email protected]
>> >>
>> >> Hey guys,
>> >>
>> >> I understand we want to quickly move to defining APIs but I am hoping
>> >> the entities defined in the first mail are not a closed decision and
>> >> is open for discussion
>> >>
>> >> So here is the list of entities defined in the first mail
>> >>
>> >> Entities
>> >> -- Idealstate
>> >> -- ExternalView
>> >> -- CurrentState
>> >> -- LiveInstance
>> >> -- HelixConfig
>> >> -- UserConfig
>> >> -- StateModelDefinition
>> >> -- Alerts
>> >>
>> >> I feel like there is a overlap of a lot of different constructs here
>> >> which are not really entities.
>> >>
>> >> The basic entities in Helix IMO should be something like
>> >>
>> >> * Entities
>> >> -- Cluster
>> >> -- Instance
>> >> -- Resource //I imagine replicas are a specialization of a resource so
>> >> dont seem to be first class entities
>> >> -- Partition
>> >> -- User? //not clear how this is modelled but I put a place holder,
>> >> probably not needed
>> >>
>> >> I imagine given that we could have multiple clusters, we may need
>> >> something which contains clusters i.e. a HelixSystem entity.
>> >>
>> >> * Entities have configurations (there are again two perspectives to
>> >> configurations)
>> >> -- Configuration passed in during creation i.e. which is not yet
>> realized
>> >> -- Runtime configuration of the entity i.e. the current values in the
>> >> system which are realized
>> >>
>> >> * Entities may or may not have a role
>> >> -- This is where all the instance roles come into play
>> >>
>> >> * Entities have a state
>> >>
>> >> Given that we are redefining things anyway what do you guys think
>> >> about this perspective?
>> >>
>> >> Thanks,
>> >>
>> >> Sandeep
>> >>
>> >> On Sun, Feb 23, 2014 at 10:20 AM, Kanak Biscuitwala <
>> [email protected]> wrote:
>> >>> Entities are the basic units of information necessary in order for
>> Helix to function. It's the "language" Helix speaks. If you look at
>> ZooKeeper, this is what you would see.
>> >>>
>> >>> Scopes are the levels at which things can be configured. That is to
>> say, you can configure the entire cluster, a single resource, a
>> participant, etc.
>> >>>
>> >>> Roles are the way nodes are labeled in the system. Currently Helix
>> nodes can have the following roles: Participant, Spectator, Controller, and
>> Administrator. A single machine may take on multiple roles.
>> >>>
>> >>> The purpose of the API is to sufficiently abstract the entities at
>> appropriate scopes for each role.
>> >>>
>> >>> When we started brainstorming APIs, we identified the following use
>> cases: configuration, registration, state transitions, and rebalancing.
>> Here is a take on what part of configuration and registration could look
>> like:
>> >>>
>> >>> Administrator admin = new ZKHelixAdministrator();
>> >>> admin.connect();
>> >>> ClusterConfig = new ClusterConfig().setName().setuserproperties();
>> >>> admin.createCluster(clusterconfig);
>> >>> StateModelDef def = new MasterSlaveModel();
>> >>> ResourceConfig resourceConfig = new
>> ResourceConfig(resourceId).withFSM(def).partitioned(24).replicated(3).withRebalancer(RebalancerConfig);
>> >>> admin.addResource(resourceConfig);
>> >>>
>> >>> //add Instances, we shud probably call this ParticipantConfig
>> >>> ParticipantConfig instanceConfig = new ParticipantConfig();
>> >>>
>> >>> // We can probably have controllerService that wraps controller --
>> this can help people who want to control which events trigger the pipeline
>> >>>
>> >>> // not StandaloneController? no -- standalone means its only one
>> controller, people think its not fault tolerant
>> >>> SingleClusterController controller = new SingleClusterController();
>> >>> controller.start();
>> >>>
>> >>> // controls multiple clusters
>> >>> MultiClusterController controller = new MultiClusterController();
>> >>> controller.start();
>> >>>
>> >>> Participant participant = new Participant();
>> >>> ParticipantService service = new PartiticipantService(participant);
>> >>> service.start(); // service has some callbacks init/start getListeners
>> callback that will be called appropriately
>> >>>
>> >>> Specator spectator = new Spectator();
>> >>> SpecatorService service = new SpectatorService().watch(resource(s));
>> >>> service.start(); //similar to participant gives a callback to get the
>> listeners to be added
>> >>>
>> >>> ----------------------------------------
>> >>>> Date: Sun, 23 Feb 2014 09:51:01 -0800
>> >>>> Subject: Re: [DISCUSS] New Helix Api
>> >>>> From: [email protected]
>> >>>> To: [email protected]
>> >>>>
>> >>>> Hey guys,
>> >>>>
>> >>>> For the sake of the newcomer could you please define/describe what you
>> >>>> mean by the following
>> >>>>
>> >>>> (a) Entity
>> >>>> (b) Scope
>> >>>>
>> >>>> I have some thoughts on the entities etc but wanted to understand what
>> >>>> their definitions are before I suggest something.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Sandeep
>> >>>>
>> >>>> On Sun, Feb 23, 2014 at 7:54 AM, kishore g <[email protected]>
>> wrote:
>> >>>>> Hi,
>> >>>>>
>> >>>>> Here are the entities, scopes and roles defined in Helix.
>> >>>>> Entities
>> >>>>> -- Idealstate
>> >>>>> -- ExternalView
>> >>>>> -- CurrentState
>> >>>>> -- LiveInstance
>> >>>>> -- HelixConfig
>> >>>>> -- UserConfig
>> >>>>> -- StateModelDefinition
>> >>>>> -- Alerts
>> >>>>>
>> >>>>>
>> >>>>> Scopes
>> >>>>> -- Cluster
>> >>>>> -- Participant/Group
>> >>>>> -- Resource/Group
>> >>>>> -- Partition
>> >>>>> -- Transition
>> >>>>>
>> >>>>> Misc
>> >>>>> -- One time read v/s cache v/s cache and update automatically on
>> >>>>> notification
>> >>>>>
>> >>>>> ====================
>> >>>>> HelixConfig
>> >>>>> -- Cluster
>> >>>>> -- Participant
>> >>>>> -- Resource
>> >>>>> -- Partition
>> >>>>>
>> >>>>> UserConfig
>> >>>>> -- Cluster
>> >>>>> -- Participant
>> >>>>> -- Resource
>> >>>>> -- Partition
>> >>>>>
>> >>>>> StateModelDefinition
>> >>>>> -- States
>> >>>>> -- Transitions
>> >>>>> -- Constraints (Partition only)
>> >>>>>
>> >>>>>
>> >>>>> Constraints
>> >>>>> -- Cluster
>> >>>>> -- Node
>> >>>>> -- Resource
>> >>>>> -- Partition
>> >>>>> -- Resource/ResourceGroup
>> >>>>>
>> >>>>> Kanak and I did one pass on this and made some changes. Kanak, do
>> you have
>> >>>>> send across the changed version.
>> >>>>>
>> >>>>> We need to come up with the user facing api's that support simple
>> CRUD on
>> >>>>> these entities.
>> >>>>>
>> >>>>> thanks,
>> >>>>> Kishore G
>> >
>>
