Summary of IRC Meeting in #apachehelix at Wed Feb 26 05:50:29 2014: Attendees: zzhang1, gmcdonald, osgigeek1, kanakb, kishoreg1
- Preface IRC log follows: ## Preface ## [Wed Feb 26 05:50:35 2014] <gmcdonald>: cool [Wed Feb 26 05:50:38 2014] <kanakb>: sweet [Wed Feb 26 05:50:44 2014] <gmcdonald>: Im outta here, enjoy [Wed Feb 26 05:51:23 2014] <kanakb>: okay anyway [Wed Feb 26 05:51:43 2014] <kanakb>: i think we can use the pattern that osgigeek1 suggested [Wed Feb 26 05:52:07 2014] <kanakb>: i.e. [Wed Feb 26 05:52:21 2014] <kanakb>: new HelixAdministratorBuilder.usingProvider(HelixStoreProvider.ZOOKEEPER).toAddress(zkAddress).build(); [Wed Feb 26 05:52:37 2014] <kanakb>: and [Wed Feb 26 05:53:00 2014] <kanakb>: new HelixAdministratorBuilder.usingProvider(HelixStoreProvider.MYDB).username(username).host(hostname).build() [Wed Feb 26 05:54:19 2014] <kishoreg1>: looks good, [Wed Feb 26 05:54:50 2014] <kishoreg1>: Sandeep, is Provider the right term ? [Wed Feb 26 05:55:00 2014] <osgigeek1>: yes provider is the right term here [Wed Feb 26 05:55:30 2014] <kishoreg1>: ok, sounds good [Wed Feb 26 05:55:52 2014] <osgigeek1>: ok so we now have an instance of HelixAdmin [Wed Feb 26 05:55:59 2014] <kanakb>: ok now how does a participant get an admin? [Wed Feb 26 05:56:16 2014] <kanakb>: participant.getAdmin()? [Wed Feb 26 05:56:22 2014] <kanakb>: spectator.getAdmin()? [Wed Feb 26 05:57:05 2014] <kanakb>: or it could even be a getter within the service class if we go that direction [Wed Feb 26 05:57:58 2014] <kanakb>: but i guess we can come back to that [Wed Feb 26 05:58:01 2014] <osgigeek1>: so before we go to participant should we capture what APIs hang off the HelixAdmin or HelixAdministrator? [Wed Feb 26 05:58:07 2014] <kanakb>: yeah sure [Wed Feb 26 05:58:43 2014] <kishoreg1>: we have two options [Wed Feb 26 05:58:59 2014] <kishoreg1>: #1. flat methods similar to what we have [Wed Feb 26 05:59:49 2014] <kishoreg1>: create/delete/update/read cluster|resource|instance| etc [Wed Feb 26 06:00:33 2014] <kishoreg1>: there will be too many methods in this [Wed Feb 26 06:00:54 2014] <kishoreg1>: #2. heirarchical based on scope [Wed Feb 26 06:01:04 2014] <kishoreg1>: based on scope/entity [Wed Feb 26 06:01:26 2014] <kishoreg1>: clusteradmin resourceadmin instanceadmin [Wed Feb 26 06:01:44 2014] <osgigeek1>: I prefer #1 [Wed Feb 26 06:02:37 2014] <kishoreg1>: how many methods do we think we will have in this interface [Wed Feb 26 06:02:40 2014] <kishoreg1>: 100? [Wed Feb 26 06:02:59 2014] <kanakb>: probably 50 [Wed Feb 26 06:03:07 2014] <kanakb>: there are some things we can consolidate [Wed Feb 26 06:03:10 2014] <kishoreg1>: ok [Wed Feb 26 06:03:20 2014] <kanakb>: e.g. the things that involve userconfig [Wed Feb 26 06:03:25 2014] <kanakb>: those can take scope as a parameter [Wed Feb 26 06:03:55 2014] <kishoreg1>: sandeep, you should look at the api's in the current helixadmin [Wed Feb 26 06:03:57 2014] <osgigeek1>: so why do we think it will be 50? [Wed Feb 26 06:04:20 2014] <kanakb>: the current number of helixadmin method calls is like 20-30 [Wed Feb 26 06:04:20 2014] <osgigeek1>: looking at it now [Wed Feb 26 06:04:31 2014] <kanakb>: so say it doubles because we cover more use cases [Wed Feb 26 06:04:34 2014] <kanakb>: then it's 50 [Wed Feb 26 06:04:43 2014] <kishoreg1>: there are too many combinations i think, we need to figure out how to reduce this [Wed Feb 26 06:05:33 2014] <osgigeek1>: Can we consider like a command pattern? [Wed Feb 26 06:05:48 2014] <osgigeek1>: like I see several addResource(paramâ¦.) [Wed Feb 26 06:05:58 2014] <osgigeek1>: instead how about addResource(ResourceCommand) [Wed Feb 26 06:06:15 2014] <kishoreg1>: i like that [Wed Feb 26 06:06:16 2014] <osgigeek1>: the command encapsulates the different params [Wed Feb 26 06:06:24 2014] <osgigeek1>: that way we can decipher from the command [Wed Feb 26 06:06:40 2014] <kanakb>: well, if we're adding using ResourceConfig like in the wiki, then is this still necessary? [Wed Feb 26 06:08:14 2014] <kishoreg1>: probably yes, [Wed Feb 26 06:08:28 2014] <kishoreg1>: lets say we want to only update provisionerconfig [Wed Feb 26 06:08:36 2014] <kishoreg1>: we have two options delta [Wed Feb 26 06:12:01 2014] <osgigeek1>: So kanakb I think if command == config and that was the notion then we can go with config [Wed Feb 26 06:12:31 2014] <osgigeek1>: but it appears kishoreg1 is pointing to the something missing [Wed Feb 26 06:13:11 2014] <kishoreg1>: its along the same line but its too generic [Wed Feb 26 06:13:30 2014] <kishoreg1>: There is only one ResourceCommand [Wed Feb 26 06:14:03 2014] <kanakb>: are you suggesting something like [Wed Feb 26 06:14:14 2014] <kanakb>: administrateResource(command, args) [Wed Feb 26 06:14:17 2014] <kanakb>: or something to that effect? [Wed Feb 26 06:14:33 2014] <kishoreg1>: nope [Wed Feb 26 06:15:15 2014] <kishoreg1>: currently we have updateResource(ResourceCommand) [Wed Feb 26 06:16:07 2014] <kishoreg1>: but the usecases are updating partitions, replicas, rebalancerconfig, provisionerconfig etc [Wed Feb 26 06:19:37 2014] <kishoreg1>: https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/HelixAdmin.java [Wed Feb 26 06:22:11 2014] <kishoreg1>: what do u guys think [Wed Feb 26 06:23:15 2014] <osgigeek1>: I did not quite follow the part where kishoreg1 you point to updating partitions, replicas, rebalancerconfig etc can you elaborate? [Wed Feb 26 06:23:59 2014] <kishoreg1>: after the resource is created [Wed Feb 26 06:24:42 2014] <kishoreg1>: we might have to update some parts of the resources, for example change the number of partitions from 10 to 20 [Wed Feb 26 06:24:46 2014] <osgigeek1>: ah gotcha [Wed Feb 26 06:24:50 2014] <kishoreg1>: we have two options [Wed Feb 26 06:25:17 2014] <kishoreg1>: have a method called updateParitionNumber(resourceId, numberof partitions) [Wed Feb 26 06:25:30 2014] <kishoreg1>: or updateResource(ResourceConfig) [Wed Feb 26 06:25:45 2014] <kishoreg1>: updateResource(ResourceConfig.Delta) [Wed Feb 26 06:25:51 2014] <osgigeek1>: now I follow [Wed Feb 26 06:25:54 2014] <kishoreg1>: ok [Wed Feb 26 06:26:18 2014] <kishoreg1>: so if we just have one method for all operations, its too overloaded [Wed Feb 26 06:28:34 2014] <osgigeek1>: ok so lets look at the primary entities here [Wed Feb 26 06:29:01 2014] <osgigeek1>: (1) cluster (2) resource (3) instance [Wed Feb 26 06:29:04 2014] <osgigeek1>: am I correct? [Wed Feb 26 06:29:25 2014] <kanakb>: arguably partition [Wed Feb 26 06:29:33 2014] <osgigeek1>: ok so 4 [Wed Feb 26 06:30:07 2014] <osgigeek1>: so what if we think of having CRUD operations for those through commands on the HelixAdmin [Wed Feb 26 06:30:42 2014] <osgigeek1>: everything else which is not a primary entity lets say we model on the Command or as a derived object off the correct Command? [Wed Feb 26 06:31:26 2014] <kishoreg1>: others I can think of are adding statemodel, constraints etc [Wed Feb 26 06:31:29 2014] <osgigeek1>: e.g. ResourceCommand, ResourceReplicaCommand extends ResourceCommand? or should the ResourceCommand have Replica as a composite ? [Wed Feb 26 06:31:59 2014] <kishoreg1>: hmm thinking [Wed Feb 26 06:32:11 2014] <osgigeek1>: I am thinking of statemodel, constraints all as parts that can be attached to the primary entities [Wed Feb 26 06:32:25 2014] <osgigeek1>: arguably not attacheable to all but most I imagine [Wed Feb 26 06:32:30 2014] <kishoreg1>: yes [Wed Feb 26 06:33:10 2014] <kishoreg1>: why should it extend resourcecommand [Wed Feb 26 06:34:02 2014] <osgigeek1>: on a second thought no lets not extend it, the more I think the more little sense it makes [Wed Feb 26 06:34:19 2014] <osgigeek1>: I think this is what kanakb modelled in his wiki [Wed Feb 26 06:34:33 2014] <osgigeek1>: a builder which allows attaching the composites into one command [Wed Feb 26 06:34:39 2014] <osgigeek1>: or in his case config [Wed Feb 26 06:35:26 2014] <osgigeek1>: So we have a command builder [Wed Feb 26 06:35:48 2014] <osgigeek1>: ResourceCommandBuilder().usingStateModel().withConstraints().build() [Wed Feb 26 06:36:03 2014] <osgigeek1>: InstanceCommandBuilder().withConstraints().build() [Wed Feb 26 06:36:30 2014] <kishoreg1>: hmm [Wed Feb 26 06:36:56 2014] <osgigeek1>: so the key thing I am after is separating the primary entities [Wed Feb 26 06:37:01 2014] <osgigeek1>: and calling them out at the top level [Wed Feb 26 06:37:10 2014] <kishoreg1>: do we need builder pattern for commands [Wed Feb 26 06:38:57 2014] <osgigeek1>: Lets ask this question a bit differently [Wed Feb 26 06:39:17 2014] <osgigeek1>: how well defined are the commands, do we think they are pretty well formalized? Do we foresee them changing ? [Wed Feb 26 06:40:01 2014] <zzhang1>: we may add new fields to the configs [Wed Feb 26 06:40:16 2014] <kishoreg1>: they might change but most likely we will be adding new stuff [Wed Feb 26 06:40:33 2014] <kishoreg1>: yeah, agree with jason [Wed Feb 26 06:40:42 2014] <osgigeek1>: the new stuff added would that be in existing buckets of statemodel, constraints? I imagine yes [Wed Feb 26 06:41:01 2014] <kishoreg1>: yes, [Wed Feb 26 06:41:41 2014] <osgigeek1>: my instinct tells me its better to hide the command objects and give builders [Wed Feb 26 06:41:52 2014] <osgigeek1>: but you guys would be better judges [Wed Feb 26 06:42:29 2014] <osgigeek1>: when I say hide I mean not give out the setters [Wed Feb 26 06:42:30 2014] <kishoreg1>: we can come to builders v/s concrete after we decide how many command classes we will have [Wed Feb 26 06:42:40 2014] <osgigeek1>: yeah lets do that [Wed Feb 26 06:43:16 2014] <kishoreg1>: lets take simple cases like change partition, replica [Wed Feb 26 06:43:32 2014] <osgigeek1>: ok [Wed Feb 26 06:43:45 2014] <kishoreg1>: or some flags in idealstate like enable/disable [Wed Feb 26 06:44:01 2014] <kanakb>: enable/disable resource? [Wed Feb 26 06:44:21 2014] <kishoreg1>: resource, bucketization, group message mode [Wed Feb 26 06:44:24 2014] <kishoreg1>: etc [Wed Feb 26 06:44:32 2014] <kanakb>: right [Wed Feb 26 06:45:16 2014] <kishoreg1>: so we have these options [Wed Feb 26 06:46:07 2014] <kishoreg1>: #1 direct method in helixadmin admin.updateNumPartitions(resourceId, 20) [Wed Feb 26 06:47:06 2014] <kishoreg1>: #2. config delta method, resource.delta= â¦, delta.setpartitions(20), admin.updateresource(resourcedelta) [Wed Feb 26 06:47:33 2014] <zzhang1>: i prefer #2, it saves lots of admin#methods [Wed Feb 26 06:47:45 2014] <osgigeek1>: may I ask this and you will see where I am going with this⦠what do we return on helixAdmin.addResource(ResourceCommand) [Wed Feb 26 06:48:04 2014] <osgigeek1>: void? HelixResource? [Wed Feb 26 06:48:20 2014] <kishoreg1>: #3. command pattern UpdatePartitionCommand.updatePartitions(20) admin.updateResource(command); [Wed Feb 26 06:48:34 2014] <kishoreg1>: dunno [Wed Feb 26 06:48:58 2014] <kishoreg1>: i will be back in 10-15 minutes [Wed Feb 26 06:48:59 2014] <zzhang1>: currently we are returning void [Wed Feb 26 06:49:37 2014] <kanakb>: i think it should either: return boolean or throw a meaningful checked exception [Wed Feb 26 06:49:59 2014] <osgigeek1>: so should we return a HelixResource? [Wed Feb 26 06:50:17 2014] <osgigeek1>: rather I meant to ask why not return a HelixResource? [Wed Feb 26 06:51:12 2014] <kanakb>: a resource is a resource config plus runtime state, like its external view [Wed Feb 26 06:51:18 2014] <kanakb>: initially this external view will always be empty [Wed Feb 26 06:51:38 2014] <kanakb>: so it's not super useful to just return that [Wed Feb 26 06:52:24 2014] <kishoreg1>: thats another item, shud we mix runtime and static info in one class [Wed Feb 26 06:52:48 2014] <kishoreg1>: prefer to defer that to a later stage [Wed Feb 26 06:52:51 2014] <osgigeek1>: ok [Wed Feb 26 06:53:05 2014] <osgigeek1>: the reason for the question was [Wed Feb 26 06:53:24 2014] <osgigeek1>: if we return the object we could have ResourceCommand.using(object) [Wed Feb 26 06:53:29 2014] <osgigeek1>: that creates an updateCommand [Wed Feb 26 06:53:37 2014] <osgigeek1>: which then can carry existing state [Wed Feb 26 06:53:58 2014] <kanakb>: the hope is that you don't need existing state in order to make an incremental update [Wed Feb 26 06:54:34 2014] <osgigeek1>: ok so lets forget that route for now, lets go back to the 3 options from kishoreg1 [Wed Feb 26 06:54:48 2014] <osgigeek1>: I am leaning towards #3 [Wed Feb 26 06:55:03 2014] <kanakb>: yeah i like #3 as well [Wed Feb 26 06:55:48 2014] <kanakb>: zzhang1: is #3 generic enough to support things like monitoring config? [Wed Feb 26 06:55:52 2014] <kanakb>: my intuition is yes [Wed Feb 26 06:55:59 2014] <zzhang1>: yea [Wed Feb 26 06:56:49 2014] <zzhang1>: for #3, shall we have a command pattern for updating each field? [Wed Feb 26 06:57:34 2014] <kishoreg1>: what ever makes sense [Wed Feb 26 06:57:51 2014] <osgigeek1>: zzhang1: I think yes that way we remove APIs like setResourceIdealState from the HelixAdmin [Wed Feb 26 06:57:52 2014] <zzhang1>: ok [Wed Feb 26 06:58:03 2014] <kanakb>: can we do something like updateResource(ResourceCommand... commands)? [Wed Feb 26 06:58:25 2014] <kishoreg1>: compositecommands? [Wed Feb 26 06:58:28 2014] <osgigeek1>: ooh, multiple commands you mean? [Wed Feb 26 06:58:47 2014] <kanakb>: yeah or a composite command [Wed Feb 26 06:58:57 2014] <osgigeek1>: I think we dont do that for the first iteration [Wed Feb 26 06:59:08 2014] <kanakb>: i would imagine that there are certain configs that we'd like to set together as an atomic unit [Wed Feb 26 06:59:19 2014] <osgigeek1>: what happens if some commands fail and some succeed [Wed Feb 26 06:59:20 2014] <kishoreg1>: yeah i was thinking the same [Wed Feb 26 06:59:31 2014] <kishoreg1>: same as kanakb [Wed Feb 26 06:59:41 2014] <kishoreg1>: for example if i want to change both partition and replica [Wed Feb 26 06:59:44 2014] <kanakb>: osgigeek1: then they should all succeed or all fail [Wed Feb 26 07:00:03 2014] <kanakb>: and it's our responsibility to guarantee that [Wed Feb 26 07:00:25 2014] <kanakb>: fortunately this is generally easy to guarantee in the ZK case [Wed Feb 26 07:00:38 2014] <kanakb>: as long as things don't span znodes [Wed Feb 26 07:00:58 2014] <osgigeek1>: kanakb: if we can guarantee that then sure, I was concerned that it might not be possible given the distributed nature of the system [Wed Feb 26 07:01:46 2014] <kanakb>: there is one interesting case [Wed Feb 26 07:01:48 2014] <kishoreg1>: kanakb: how can we gaurantee things that span multiple znodes [Wed Feb 26 07:01:58 2014] <kanakb>: we'd need to do a rollback or something [Wed Feb 26 07:02:10 2014] <kanakb>: so maybe it's not so easy [Wed Feb 26 07:02:23 2014] <kanakb>: hmm [Wed Feb 26 07:02:50 2014] <kanakb>: right now all our admin apis work on a single znode [Wed Feb 26 07:02:56 2014] <kanakb>: but now that won't be true [Wed Feb 26 07:03:21 2014] <kishoreg1>: yeah, we have rely on the spi to make it atomic [Wed Feb 26 07:03:44 2014] <kishoreg1>: but we might need composite commands [Wed Feb 26 07:04:01 2014] <kishoreg1>: for example what if we want to update both partitions and replicas [Wed Feb 26 07:04:10 2014] <kanakb>: right [Wed Feb 26 07:04:23 2014] <kanakb>: so we either need to allow multiple commands, or a single command that does multiple things [Wed Feb 26 07:04:29 2014] <osgigeek1>: kishoreg1: we should model that as one command which carries both partition change and replica change [Wed Feb 26 07:04:49 2014] <kanakb>: i.e. ResourceCommand.partitions(10).replicas(3)? [Wed Feb 26 07:04:53 2014] <osgigeek1>: yes [Wed Feb 26 07:05:19 2014] <zzhang1>: btw, i think commands like addCluster() will cross multiple znodes [Wed Feb 26 07:05:41 2014] <kanakb>: which brings in the atomic api issues [Wed Feb 26 07:06:12 2014] <kishoreg1>: its upto the implementation, we shud assume all api's are atomic [Wed Feb 26 07:06:36 2014] <osgigeek1>: I think we should make that a principle of the command APIs [Wed Feb 26 07:06:36 2014] <kishoreg1>: we can use multitransaction updates in ZK [Wed Feb 26 07:06:52 2014] <kanakb>: are those available in 3.3.x? [Wed Feb 26 07:07:09 2014] <kishoreg1>: nope 3.4.5 i think [Wed Feb 26 07:07:14 2014] <kishoreg1>: but u already have locks [Wed Feb 26 07:07:18 2014] <kanakb>: yeah [Wed Feb 26 07:07:21 2014] <kanakb>: i was going to say [Wed Feb 26 07:07:27 2014] <kishoreg1>: so its ok to rely on that [Wed Feb 26 07:07:44 2014] <kishoreg1>: what u have is better than multi transactions i think [Wed Feb 26 07:08:09 2014] <kanakb>: well yes and no [Wed Feb 26 07:08:30 2014] <kishoreg1>: anyhow ResourceCommand.partitions(1) will return a type of ResourceCommand? [Wed Feb 26 07:09:39 2014] <osgigeek1>: yes, the builder should allow changing any of the facets i.e. partition or replica of the entity i.e. Resource in this case [Wed Feb 26 07:09:47 2014] <kanakb>: chaining probably isn't possible unless we change it to new ResourceCommand().partitions(1) [Wed Feb 26 07:10:26 2014] <kishoreg1>: ok, i am not good at builder patterns, i hope we dont have too many classes [Wed Feb 26 07:10:38 2014] <kanakb>: that would just be 1 class [Wed Feb 26 07:10:45 2014] <osgigeek1>: yes [Wed Feb 26 07:10:56 2014] <kishoreg1>: ok [Wed Feb 26 07:11:07 2014] <osgigeek1>: i.e. agree with kanakb [Wed Feb 26 07:11:36 2014] <kishoreg1>: ok [Wed Feb 26 07:13:26 2014] <osgigeek1>: ok well I might have to drop off in a few mins ⦠do you guys plan to carry on for long? [Wed Feb 26 07:13:36 2014] <kanakb>: probably not too long [Wed Feb 26 07:13:43 2014] <kishoreg1>: i think we shud take break as well [Wed Feb 26 07:13:44 2014] <kanakb>: at least we have some direction with admin now [Wed Feb 26 07:14:03 2014] <kishoreg1>: lets write the apis for administrator and see how it looks [Wed Feb 26 07:14:13 2014] <kishoreg1>: i need to get going as well [Wed Feb 26 07:15:05 2014] <kanakb>: ok let's end here? [Wed Feb 26 07:15:10 2014] <osgigeek1>: well this was good, we made good progress, yeah lets end [Wed Feb 26 07:15:37 2014] <kanakb>: ASFBot: meeting end Meeting ended at Wed Feb 26 07:15:37 2014
