Overall it's could be nice conversion, as a new comer I have learnt a lot Sent from my iPhone
> On Feb 25, 2014, at 12:24 AM, "Kanak Biscuitwala" <[email protected]> wrote: > > > Hi Sandeep, > > We definitely would like you to participate. You've shown from your messages > on this list that you're good with Java development patterns, plus you're > just starting out with Helix so ideally we want to cater to people like you. > > Honestly, we've never really done changes of this scale in a fully public way > up until now. In the past, one or two of us would take charge on something, > gather some feedback on the mailing list, and then come back with a code > review. Clearly we need to be a little more diligent in this case because > this is entirely user-facing. > > What I imagine is that with what I have posted, people will comment to say > things like: "___ was done very awkwardly, what if we changed it to _____?" > Or "great, ____ works well for ____ case, but what if the user does ____ > instead?" Then, when we're comfortable with how we answer usage patterns, we > can see how to consolidate it into interfaces, package layouts, modules, etc. > Finally, we can discuss how each function call should behave if it fails > (going back to the exceptions discussion you started last week). > > So to answer your question, please provide feedback in whatever channel is > easiest for you, and then we can evaluate what works best; we're learning too > :) > > Kanak > > ---------------------------------------- >> Date: Mon, 24 Feb 2014 21:12:51 -0800 >> Subject: Re: [DISCUSS] New Helix Api >> From: [email protected] >> To: [email protected] >> >> Hi Kanak, >> >> Thanks for putting this together, it certainly gives an idea of how to >> go about creating the different components of Helix. I wanted to ask >> how do you guys discuss questions or exchange comments on what is >> posted? Do you ask questions on the wiki? How does this get distilled >> into an API, what is the process? >> >> I ask because I would like to participate if you guys are ok with my >> feedback/participation. >> >> Thanks, >> >> Sandeep >> >>> On Mon, Feb 24, 2014 at 6:12 PM, Kanak Biscuitwala <[email protected]> >>> wrote: >>> >>> Formatting got messed up. Resending: >>> >>> >>> Hi, >>> >>> This discussion was starting to lose focus somewhat, so I wrote a bunch of >>> example code of what code users would need to write in order to interact >>> with Helix. The hope is that we can use this as a starting point, >>> understand where it isn't sufficiently convenient, and find ways to >>> simplify things in such a way that we arrive at an API that we can agree on. >>> >>> In my opinion, it's really important that we start with use cases and >>> derive interfaces from them, rather than the other way around. That is how >>> I tried to structure the wiki pages. >>> >>> The reasons for doing this: (1) it gives us a readable starting point, (2) >>> it starts with what users expect to be able to do with the cluster through >>> Helix, and (3) it's broken down by role and interaction type so that no one >>> has to digest everything in one sitting. >>> >>> Here are the wiki pages: >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620875 >>> (Participant) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620887 >>> (Spectator) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620893 >>> (Controller) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620931 >>> (Administrator) >>> >>> Here is where I put all the source code from the examples: >>> https://code.stypi.com/kanak/logicalmodel.java >>> >>> Kanak >>> >>> >>> From: [email protected] >>> To: [email protected] >>> Subject: RE: [DISCUSS] New Helix Api >>> Date: Mon, 24 Feb 2014 18:08:39 -0800 >>> >>> >>> Hi, >>> >>> This discussion was starting to lose focus somewhat, so I wrote a bunch of >>> example code of what code users would need to write in order to interact >>> with Helix. The hope is that we can use this as a starting point, >>> understand where it isn't sufficiently convenient, and find ways to >>> simplify things in such a way that we arrive at an API that we can agree on. >>> >>> In my opinion, it's really important that we start with use cases and >>> derive interfaces from them, rather than the other way around. That is how >>> I tried to structure the wiki pages. >>> >>> The reasons for doing this: (1) it gives us a readable starting point, (2) >>> it starts with what users expect to be able to do with the cluster through >>> Helix, and (3) it's broken down by role and interaction type so that no one >>> has to digest everything in one sitting. >>> >>> Here are the wiki pages: >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620875 >>> (Participant) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620887 >>> (Spectator) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620893 >>> (Controller) >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=39620931 >>> (Administrator) >>> >>> Here is where I put all the source code from the examples: >>> https://code.stypi.com/kanak/logicalmodel.java >>> >>> Kanak >>> >>> ---------------------------------------- >>>> Date: Sun, 23 Feb 2014 18:00:15 -0800 >>>> Subject: Re: [DISCUSS] New Helix Api >>>> From: [email protected] >>>> To: [email protected] >>>> >>>> Yes, the intention of bringing in the separation between SPI and API >>>> is exactly to allow for separation of two things >>>> >>>> (a) how a model is presented to the user --- API concern >>>> (b) how the model is stored -- SPI concern >>>> >>>> Sure, lets start with the APIs and see it would shape. The only >>>> request I have is lets define the logical model, then go to how that >>>> logical model is created through the Admin API. I think getting the >>>> logical model right will take more time and thought, than the Admin >>>> API. >>>> >>>> Here is an example to illustrate what I am trying to convey. This is >>>> just to illustrate my thought process not to suggest this is the exact >>>> API so please bear with me. I am deliberately leaving out exceptions >>>> because the API is to give an idea of what I am talking about. >>>> >>>> public interface Cluster { >>>> >>>> List<Instance> getParticipants(); >>>> >>>> String getName(); >>>> >>>> Instance getLocalParticipant(); >>>> >>>> ... >>>> } >>>> >>>> public interface Instance{ >>>> >>>> boolean isLocal(); >>>> >>>> String getId(); >>>> >>>> int getPort(); >>>> >>>> String getHostName(); >>>> >>>> List<Resource> getResources(); >>>> >>>> } >>>> >>>> Sandeep >>>> >>>>> On Sun, Feb 23, 2014 at 5:31 PM, kishore g <[email protected]> wrote: >>>>> Basically what we have is now is that the logical model exposed to user >>>>> more or less matches the physical model (how data is stored in ZK). We can >>>>> definitely separate these two (I think this is what Sandeep is >>>>> suggesting). >>>>> I am not sure how invasive and hard it would be to get that right. We >>>>> definitely need to make sure that we are backward compatible with physical >>>>> model. >>>>> >>>>> >>>>> Given this, does it make sense to define two models, physical and logical. >>>>> I think what we have currently in our model package can be called as >>>>> physical model and we can work on logical model. I would say we are >>>>> actually pretty close to what we want in 0.7.0. >>>>> >>>>> In terms of entities. Yes the high level concepts are cluster, instance, >>>>> resource, partition. Don't see a need for User. >>>>> >>>>> Entities have what we can probably refer to as attributes >>>>> - configurations >>>>> --- static/not yet realized >>>>> --- runtime configuration (this is more or less the state + additional >>>>> stuff) >>>>> --- user configuration (Helix has nothing to do with it). This again has >>>>> static and dynamic >>>>> - Roles, will only be associated with instances, i dont see any other >>>>> entity having roles >>>>> - State, for now only partition/instance has a state. But we can define a >>>>> state at resource and cluster scope >>>>> >>>>> The problem with configuration is that it becomes too generic, especially >>>>> with Resource. We need to make this as specific as possible. >>>>> >>>>> I feel the best way to come up with these api's is to use an example >>>>> similar to what Kanak wrote. That will give us an idea of how user facing >>>>> api will look like. These are the operations a user performs >>>>> >>>>> -- Create, read, update, delete a cluster, resource, partition, instance >>>>> and its configuration >>>>> -- Register listeners to changes in each of above >>>>> -- Handle state transitions. >>>>> >>>>> Lets define the set of operations the user performs and write an example >>>>> on >>>>> how it looks and should look. We need to see how all these tie together in >>>>> the flow. >>>>> >>>>> Sandeep at a high level what you say makes sense. I think rest of us know >>>>> a >>>>> lot of the internals which means our views are biased, which is both good >>>>> and bad :-). >>>>> >>>>> If it helps, we can start defining the classes and see how it looks. >>>>> >>>>> What do you think. >>>>> >>>>> thanks, >>>>> Kishore G >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On Sun, Feb 23, 2014 at 11:42 AM, Sandeep Nayak <[email protected]> >>>>>> wrote: >>>>>> >>>>>> I am not entirely conversant with the internals so dont have enough to >>>>>> sway one way or the other. >>>>>> >>>>>> But it seems like we are mixing two separate concerns here. Modeling >>>>>> an entity to be surfaced in an API should have no bearing on how it is >>>>>> passivated. You can still store the data in zookeeper the way you are >>>>>> doing so today. Having an SPI which talks to how you passivate data >>>>>> v/s an API to how the user interacts with data are two separate >>>>>> concerns in my view. >>>>>> >>>>>> Sandeep >>>>>> >>>>>> On Sun, Feb 23, 2014 at 11:33 AM, Kanak Biscuitwala <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Changing/removing entities in ZK is highly undesirable. We still want >>>>>> old-version controllers to manage new-version participants, and vice >>>>>> versa. >>>>>> Making a change like that would require system-wide upgrades and >>>>>> downtime. >>>>>>> >>>>>>> What we can do is change how the entities are exposed to the users >>>>>> through an API. That is really the goal here: avoid changing the backing >>>>>> store, but rethink how the user interacts with it. >>>>>>> >>>>>>> Kanak >>>>>>> >>>>>>> ---------------------------------------- >>>>>>>> Date: Sun, 23 Feb 2014 11:22:55 -0800 >>>>>>>> Subject: Re: [DISCUSS] New Helix Api >>>>>>>> From: [email protected] >>>>>>>> To: [email protected] >>>>>>>> >>>>>>>> Hey guys, >>>>>>>> >>>>>>>> I understand we want to quickly move to defining APIs but I am hoping >>>>>>>> the entities defined in the first mail are not a closed decision and >>>>>>>> is open for discussion >>>>>>>> >>>>>>>> So here is the list of entities defined in the first mail >>>>>>>> >>>>>>>> Entities >>>>>>>> -- Idealstate >>>>>>>> -- ExternalView >>>>>>>> -- CurrentState >>>>>>>> -- LiveInstance >>>>>>>> -- HelixConfig >>>>>>>> -- UserConfig >>>>>>>> -- StateModelDefinition >>>>>>>> -- Alerts >>>>>>>> >>>>>>>> I feel like there is a overlap of a lot of different constructs here >>>>>>>> which are not really entities. >>>>>>>> >>>>>>>> The basic entities in Helix IMO should be something like >>>>>>>> >>>>>>>> * Entities >>>>>>>> -- Cluster >>>>>>>> -- Instance >>>>>>>> -- Resource //I imagine replicas are a specialization of a resource so >>>>>>>> dont seem to be first class entities >>>>>>>> -- Partition >>>>>>>> -- User? //not clear how this is modelled but I put a place holder, >>>>>>>> probably not needed >>>>>>>> >>>>>>>> I imagine given that we could have multiple clusters, we may need >>>>>>>> something which contains clusters i.e. a HelixSystem entity. >>>>>>>> >>>>>>>> * Entities have configurations (there are again two perspectives to >>>>>>>> configurations) >>>>>>>> -- Configuration passed in during creation i.e. which is not yet >>>>>> realized >>>>>>>> -- Runtime configuration of the entity i.e. the current values in the >>>>>>>> system which are realized >>>>>>>> >>>>>>>> * Entities may or may not have a role >>>>>>>> -- This is where all the instance roles come into play >>>>>>>> >>>>>>>> * Entities have a state >>>>>>>> >>>>>>>> Given that we are redefining things anyway what do you guys think >>>>>>>> about this perspective? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Sandeep >>>>>>>> >>>>>>>> On Sun, Feb 23, 2014 at 10:20 AM, Kanak Biscuitwala < >>>>>> [email protected]> wrote: >>>>>>>>> Entities are the basic units of information necessary in order for >>>>>> Helix to function. It's the "language" Helix speaks. If you look at >>>>>> ZooKeeper, this is what you would see. >>>>>>>>> >>>>>>>>> Scopes are the levels at which things can be configured. That is to >>>>>> say, you can configure the entire cluster, a single resource, a >>>>>> participant, etc. >>>>>>>>> >>>>>>>>> Roles are the way nodes are labeled in the system. Currently Helix >>>>>> nodes can have the following roles: Participant, Spectator, Controller, >>>>>> and >>>>>> Administrator. A single machine may take on multiple roles. >>>>>>>>> >>>>>>>>> The purpose of the API is to sufficiently abstract the entities at >>>>>> appropriate scopes for each role. >>>>>>>>> >>>>>>>>> When we started brainstorming APIs, we identified the following use >>>>>> cases: configuration, registration, state transitions, and rebalancing. >>>>>> Here is a take on what part of configuration and registration could look >>>>>> like: >>>>>>>>> >>>>>>>>> Administrator admin = new ZKHelixAdministrator(); >>>>>>>>> admin.connect(); >>>>>>>>> ClusterConfig = new ClusterConfig().setName().setuserproperties(); >>>>>>>>> admin.createCluster(clusterconfig); >>>>>>>>> StateModelDef def = new MasterSlaveModel(); >>>>>>>>> ResourceConfig resourceConfig = new >>>>>> ResourceConfig(resourceId).withFSM(def).partitioned(24).replicated(3).withRebalancer(RebalancerConfig); >>>>>>>>> admin.addResource(resourceConfig); >>>>>>>>> >>>>>>>>> //add Instances, we shud probably call this ParticipantConfig >>>>>>>>> ParticipantConfig instanceConfig = new ParticipantConfig(); >>>>>>>>> >>>>>>>>> // We can probably have controllerService that wraps controller -- >>>>>> this can help people who want to control which events trigger the >>>>>> pipeline >>>>>>>>> >>>>>>>>> // not StandaloneController? no -- standalone means its only one >>>>>> controller, people think its not fault tolerant >>>>>>>>> SingleClusterController controller = new SingleClusterController(); >>>>>>>>> controller.start(); >>>>>>>>> >>>>>>>>> // controls multiple clusters >>>>>>>>> MultiClusterController controller = new MultiClusterController(); >>>>>>>>> controller.start(); >>>>>>>>> >>>>>>>>> Participant participant = new Participant(); >>>>>>>>> ParticipantService service = new PartiticipantService(participant); >>>>>>>>> service.start(); // service has some callbacks init/start getListeners >>>>>> callback that will be called appropriately >>>>>>>>> >>>>>>>>> Specator spectator = new Spectator(); >>>>>>>>> SpecatorService service = new SpectatorService().watch(resource(s)); >>>>>>>>> service.start(); //similar to participant gives a callback to get the >>>>>> listeners to be added >>>>>>>>> >>>>>>>>> ---------------------------------------- >>>>>>>>>> Date: Sun, 23 Feb 2014 09:51:01 -0800 >>>>>>>>>> Subject: Re: [DISCUSS] New Helix Api >>>>>>>>>> From: [email protected] >>>>>>>>>> To: [email protected] >>>>>>>>>> >>>>>>>>>> Hey guys, >>>>>>>>>> >>>>>>>>>> For the sake of the newcomer could you please define/describe what >>>>>>>>>> you >>>>>>>>>> mean by the following >>>>>>>>>> >>>>>>>>>> (a) Entity >>>>>>>>>> (b) Scope >>>>>>>>>> >>>>>>>>>> I have some thoughts on the entities etc but wanted to understand >>>>>>>>>> what >>>>>>>>>> their definitions are before I suggest something. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Sandeep >>>>>>>>>> >>>>>>>>>> On Sun, Feb 23, 2014 at 7:54 AM, kishore g <[email protected]> >>>>>> wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Here are the entities, scopes and roles defined in Helix. >>>>>>>>>>> Entities >>>>>>>>>>> -- Idealstate >>>>>>>>>>> -- ExternalView >>>>>>>>>>> -- CurrentState >>>>>>>>>>> -- LiveInstance >>>>>>>>>>> -- HelixConfig >>>>>>>>>>> -- UserConfig >>>>>>>>>>> -- StateModelDefinition >>>>>>>>>>> -- Alerts >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Scopes >>>>>>>>>>> -- Cluster >>>>>>>>>>> -- Participant/Group >>>>>>>>>>> -- Resource/Group >>>>>>>>>>> -- Partition >>>>>>>>>>> -- Transition >>>>>>>>>>> >>>>>>>>>>> Misc >>>>>>>>>>> -- One time read v/s cache v/s cache and update automatically on >>>>>>>>>>> notification >>>>>>>>>>> >>>>>>>>>>> ==================== >>>>>>>>>>> HelixConfig >>>>>>>>>>> -- Cluster >>>>>>>>>>> -- Participant >>>>>>>>>>> -- Resource >>>>>>>>>>> -- Partition >>>>>>>>>>> >>>>>>>>>>> UserConfig >>>>>>>>>>> -- Cluster >>>>>>>>>>> -- Participant >>>>>>>>>>> -- Resource >>>>>>>>>>> -- Partition >>>>>>>>>>> >>>>>>>>>>> StateModelDefinition >>>>>>>>>>> -- States >>>>>>>>>>> -- Transitions >>>>>>>>>>> -- Constraints (Partition only) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Constraints >>>>>>>>>>> -- Cluster >>>>>>>>>>> -- Node >>>>>>>>>>> -- Resource >>>>>>>>>>> -- Partition >>>>>>>>>>> -- Resource/ResourceGroup >>>>>>>>>>> >>>>>>>>>>> Kanak and I did one pass on this and made some changes. Kanak, do >>>>>> you have >>>>>>>>>>> send across the changed version. >>>>>>>>>>> >>>>>>>>>>> We need to come up with the user facing api's that support simple >>>>>> CRUD on >>>>>>>>>>> these entities. >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> Kishore G >
