Basically what we have is now is that the logical model exposed to user more or less matches the physical model (how data is stored in ZK). We can definitely separate these two (I think this is what Sandeep is suggesting). I am not sure how invasive and hard it would be to get that right. We definitely need to make sure that we are backward compatible with physical model.
Given this, does it make sense to define two models, physical and logical. I think what we have currently in our model package can be called as physical model and we can work on logical model. I would say we are actually pretty close to what we want in 0.7.0. In terms of entities. Yes the high level concepts are cluster, instance, resource, partition. Don't see a need for User. Entities have what we can probably refer to as attributes - configurations --- static/not yet realized --- runtime configuration (this is more or less the state + additional stuff) --- user configuration (Helix has nothing to do with it). This again has static and dynamic - Roles, will only be associated with instances, i dont see any other entity having roles - State, for now only partition/instance has a state. But we can define a state at resource and cluster scope The problem with configuration is that it becomes too generic, especially with Resource. We need to make this as specific as possible. I feel the best way to come up with these api's is to use an example similar to what Kanak wrote. That will give us an idea of how user facing api will look like. These are the operations a user performs -- Create, read, update, delete a cluster, resource, partition, instance and its configuration -- Register listeners to changes in each of above -- Handle state transitions. Lets define the set of operations the user performs and write an example on how it looks and should look. We need to see how all these tie together in the flow. Sandeep at a high level what you say makes sense. I think rest of us know a lot of the internals which means our views are biased, which is both good and bad :-). If it helps, we can start defining the classes and see how it looks. What do you think. thanks, Kishore G On Sun, Feb 23, 2014 at 11:42 AM, Sandeep Nayak <[email protected]> wrote: > I am not entirely conversant with the internals so dont have enough to > sway one way or the other. > > But it seems like we are mixing two separate concerns here. Modeling > an entity to be surfaced in an API should have no bearing on how it is > passivated. You can still store the data in zookeeper the way you are > doing so today. Having an SPI which talks to how you passivate data > v/s an API to how the user interacts with data are two separate > concerns in my view. > > Sandeep > > On Sun, Feb 23, 2014 at 11:33 AM, Kanak Biscuitwala <[email protected]> > wrote: > > > > Changing/removing entities in ZK is highly undesirable. We still want > old-version controllers to manage new-version participants, and vice versa. > Making a change like that would require system-wide upgrades and downtime. > > > > What we can do is change how the entities are exposed to the users > through an API. That is really the goal here: avoid changing the backing > store, but rethink how the user interacts with it. > > > > Kanak > > > > ---------------------------------------- > >> Date: Sun, 23 Feb 2014 11:22:55 -0800 > >> Subject: Re: [DISCUSS] New Helix Api > >> From: [email protected] > >> To: [email protected] > >> > >> Hey guys, > >> > >> I understand we want to quickly move to defining APIs but I am hoping > >> the entities defined in the first mail are not a closed decision and > >> is open for discussion > >> > >> So here is the list of entities defined in the first mail > >> > >> Entities > >> -- Idealstate > >> -- ExternalView > >> -- CurrentState > >> -- LiveInstance > >> -- HelixConfig > >> -- UserConfig > >> -- StateModelDefinition > >> -- Alerts > >> > >> I feel like there is a overlap of a lot of different constructs here > >> which are not really entities. > >> > >> The basic entities in Helix IMO should be something like > >> > >> * Entities > >> -- Cluster > >> -- Instance > >> -- Resource //I imagine replicas are a specialization of a resource so > >> dont seem to be first class entities > >> -- Partition > >> -- User? //not clear how this is modelled but I put a place holder, > >> probably not needed > >> > >> I imagine given that we could have multiple clusters, we may need > >> something which contains clusters i.e. a HelixSystem entity. > >> > >> * Entities have configurations (there are again two perspectives to > >> configurations) > >> -- Configuration passed in during creation i.e. which is not yet > realized > >> -- Runtime configuration of the entity i.e. the current values in the > >> system which are realized > >> > >> * Entities may or may not have a role > >> -- This is where all the instance roles come into play > >> > >> * Entities have a state > >> > >> Given that we are redefining things anyway what do you guys think > >> about this perspective? > >> > >> Thanks, > >> > >> Sandeep > >> > >> On Sun, Feb 23, 2014 at 10:20 AM, Kanak Biscuitwala < > [email protected]> wrote: > >>> Entities are the basic units of information necessary in order for > Helix to function. It's the "language" Helix speaks. If you look at > ZooKeeper, this is what you would see. > >>> > >>> Scopes are the levels at which things can be configured. That is to > say, you can configure the entire cluster, a single resource, a > participant, etc. > >>> > >>> Roles are the way nodes are labeled in the system. Currently Helix > nodes can have the following roles: Participant, Spectator, Controller, and > Administrator. A single machine may take on multiple roles. > >>> > >>> The purpose of the API is to sufficiently abstract the entities at > appropriate scopes for each role. > >>> > >>> When we started brainstorming APIs, we identified the following use > cases: configuration, registration, state transitions, and rebalancing. > Here is a take on what part of configuration and registration could look > like: > >>> > >>> Administrator admin = new ZKHelixAdministrator(); > >>> admin.connect(); > >>> ClusterConfig = new ClusterConfig().setName().setuserproperties(); > >>> admin.createCluster(clusterconfig); > >>> StateModelDef def = new MasterSlaveModel(); > >>> ResourceConfig resourceConfig = new > ResourceConfig(resourceId).withFSM(def).partitioned(24).replicated(3).withRebalancer(RebalancerConfig); > >>> admin.addResource(resourceConfig); > >>> > >>> //add Instances, we shud probably call this ParticipantConfig > >>> ParticipantConfig instanceConfig = new ParticipantConfig(); > >>> > >>> // We can probably have controllerService that wraps controller -- > this can help people who want to control which events trigger the pipeline > >>> > >>> // not StandaloneController? no -- standalone means its only one > controller, people think its not fault tolerant > >>> SingleClusterController controller = new SingleClusterController(); > >>> controller.start(); > >>> > >>> // controls multiple clusters > >>> MultiClusterController controller = new MultiClusterController(); > >>> controller.start(); > >>> > >>> Participant participant = new Participant(); > >>> ParticipantService service = new PartiticipantService(participant); > >>> service.start(); // service has some callbacks init/start getListeners > callback that will be called appropriately > >>> > >>> Specator spectator = new Spectator(); > >>> SpecatorService service = new SpectatorService().watch(resource(s)); > >>> service.start(); //similar to participant gives a callback to get the > listeners to be added > >>> > >>> ---------------------------------------- > >>>> Date: Sun, 23 Feb 2014 09:51:01 -0800 > >>>> Subject: Re: [DISCUSS] New Helix Api > >>>> From: [email protected] > >>>> To: [email protected] > >>>> > >>>> Hey guys, > >>>> > >>>> For the sake of the newcomer could you please define/describe what you > >>>> mean by the following > >>>> > >>>> (a) Entity > >>>> (b) Scope > >>>> > >>>> I have some thoughts on the entities etc but wanted to understand what > >>>> their definitions are before I suggest something. > >>>> > >>>> Thanks, > >>>> > >>>> Sandeep > >>>> > >>>> On Sun, Feb 23, 2014 at 7:54 AM, kishore g <[email protected]> > wrote: > >>>>> Hi, > >>>>> > >>>>> Here are the entities, scopes and roles defined in Helix. > >>>>> Entities > >>>>> -- Idealstate > >>>>> -- ExternalView > >>>>> -- CurrentState > >>>>> -- LiveInstance > >>>>> -- HelixConfig > >>>>> -- UserConfig > >>>>> -- StateModelDefinition > >>>>> -- Alerts > >>>>> > >>>>> > >>>>> Scopes > >>>>> -- Cluster > >>>>> -- Participant/Group > >>>>> -- Resource/Group > >>>>> -- Partition > >>>>> -- Transition > >>>>> > >>>>> Misc > >>>>> -- One time read v/s cache v/s cache and update automatically on > >>>>> notification > >>>>> > >>>>> ==================== > >>>>> HelixConfig > >>>>> -- Cluster > >>>>> -- Participant > >>>>> -- Resource > >>>>> -- Partition > >>>>> > >>>>> UserConfig > >>>>> -- Cluster > >>>>> -- Participant > >>>>> -- Resource > >>>>> -- Partition > >>>>> > >>>>> StateModelDefinition > >>>>> -- States > >>>>> -- Transitions > >>>>> -- Constraints (Partition only) > >>>>> > >>>>> > >>>>> Constraints > >>>>> -- Cluster > >>>>> -- Node > >>>>> -- Resource > >>>>> -- Partition > >>>>> -- Resource/ResourceGroup > >>>>> > >>>>> Kanak and I did one pass on this and made some changes. Kanak, do > you have > >>>>> send across the changed version. > >>>>> > >>>>> We need to come up with the user facing api's that support simple > CRUD on > >>>>> these entities. > >>>>> > >>>>> thanks, > >>>>> Kishore G > > >
