Hi Guys, Do you guys do this (IIRC chat) every Wednesdays?
- Henry On Tue, Feb 25, 2014 at 11:16 PM, ASF IRC Bot <[email protected]> wrote: > Summary of IRC Meeting in #apachehelix at Wed Feb 26 05:50:29 2014: > > Attendees: zzhang1, gmcdonald, osgigeek1, kanakb, kishoreg1 > > - Preface > > > IRC log follows: > > ## Preface ## > [Wed Feb 26 05:50:35 2014] <gmcdonald>: cool > [Wed Feb 26 05:50:38 2014] <kanakb>: sweet > [Wed Feb 26 05:50:44 2014] <gmcdonald>: Im outta here, enjoy > [Wed Feb 26 05:51:23 2014] <kanakb>: okay anyway > [Wed Feb 26 05:51:43 2014] <kanakb>: i think we can use the pattern that > osgigeek1 suggested > [Wed Feb 26 05:52:07 2014] <kanakb>: i.e. > [Wed Feb 26 05:52:21 2014] <kanakb>: new > HelixAdministratorBuilder.usingProvider(HelixStoreProvider.ZOOKEEPER).toAddress(zkAddress).build(); > [Wed Feb 26 05:52:37 2014] <kanakb>: and > [Wed Feb 26 05:53:00 2014] <kanakb>: new > HelixAdministratorBuilder.usingProvider(HelixStoreProvider.MYDB).username(username).host(hostname).build() > [Wed Feb 26 05:54:19 2014] <kishoreg1>: looks good, > [Wed Feb 26 05:54:50 2014] <kishoreg1>: Sandeep, is Provider the right term ? > [Wed Feb 26 05:55:00 2014] <osgigeek1>: yes provider is the right term here > [Wed Feb 26 05:55:30 2014] <kishoreg1>: ok, sounds good > [Wed Feb 26 05:55:52 2014] <osgigeek1>: ok so we now have an instance of > HelixAdmin > [Wed Feb 26 05:55:59 2014] <kanakb>: ok now how does a participant get an > admin? > [Wed Feb 26 05:56:16 2014] <kanakb>: participant.getAdmin()? > [Wed Feb 26 05:56:22 2014] <kanakb>: spectator.getAdmin()? > [Wed Feb 26 05:57:05 2014] <kanakb>: or it could even be a getter within the > service class if we go that direction > [Wed Feb 26 05:57:58 2014] <kanakb>: but i guess we can come back to that > [Wed Feb 26 05:58:01 2014] <osgigeek1>: so before we go to participant should > we capture what APIs hang off the HelixAdmin or HelixAdministrator? > [Wed Feb 26 05:58:07 2014] <kanakb>: yeah sure > [Wed Feb 26 05:58:43 2014] <kishoreg1>: we have two options > [Wed Feb 26 05:58:59 2014] <kishoreg1>: #1. flat methods similar to what we > have > [Wed Feb 26 05:59:49 2014] <kishoreg1>: create/delete/update/read > cluster|resource|instance| etc > [Wed Feb 26 06:00:33 2014] <kishoreg1>: there will be too many methods in this > [Wed Feb 26 06:00:54 2014] <kishoreg1>: #2. heirarchical based on scope > [Wed Feb 26 06:01:04 2014] <kishoreg1>: based on scope/entity > [Wed Feb 26 06:01:26 2014] <kishoreg1>: clusteradmin resourceadmin > instanceadmin > [Wed Feb 26 06:01:44 2014] <osgigeek1>: I prefer #1 > [Wed Feb 26 06:02:37 2014] <kishoreg1>: how many methods do we think we will > have in this interface > [Wed Feb 26 06:02:40 2014] <kishoreg1>: 100? > [Wed Feb 26 06:02:59 2014] <kanakb>: probably 50 > [Wed Feb 26 06:03:07 2014] <kanakb>: there are some things we can consolidate > [Wed Feb 26 06:03:10 2014] <kishoreg1>: ok > [Wed Feb 26 06:03:20 2014] <kanakb>: e.g. the things that involve userconfig > [Wed Feb 26 06:03:25 2014] <kanakb>: those can take scope as a parameter > [Wed Feb 26 06:03:55 2014] <kishoreg1>: sandeep, you should look at the api's > in the current helixadmin > [Wed Feb 26 06:03:57 2014] <osgigeek1>: so why do we think it will be 50? > [Wed Feb 26 06:04:20 2014] <kanakb>: the current number of helixadmin method > calls is like 20-30 > [Wed Feb 26 06:04:20 2014] <osgigeek1>: looking at it now > [Wed Feb 26 06:04:31 2014] <kanakb>: so say it doubles because we cover more > use cases > [Wed Feb 26 06:04:34 2014] <kanakb>: then it's 50 > [Wed Feb 26 06:04:43 2014] <kishoreg1>: there are too many combinations i > think, we need to figure out how to reduce this > [Wed Feb 26 06:05:33 2014] <osgigeek1>: Can we consider like a command > pattern? > [Wed Feb 26 06:05:48 2014] <osgigeek1>: like I see several > addResource(param….) > [Wed Feb 26 06:05:58 2014] <osgigeek1>: instead how about > addResource(ResourceCommand) > [Wed Feb 26 06:06:15 2014] <kishoreg1>: i like that > [Wed Feb 26 06:06:16 2014] <osgigeek1>: the command encapsulates the > different params > [Wed Feb 26 06:06:24 2014] <osgigeek1>: that way we can decipher from the > command > [Wed Feb 26 06:06:40 2014] <kanakb>: well, if we're adding using > ResourceConfig like in the wiki, then is this still necessary? > [Wed Feb 26 06:08:14 2014] <kishoreg1>: probably yes, > [Wed Feb 26 06:08:28 2014] <kishoreg1>: lets say we want to only update > provisionerconfig > [Wed Feb 26 06:08:36 2014] <kishoreg1>: we have two options delta > [Wed Feb 26 06:12:01 2014] <osgigeek1>: So kanakb I think if command == > config and that was the notion then we can go with config > [Wed Feb 26 06:12:31 2014] <osgigeek1>: but it appears kishoreg1 is pointing > to the something missing > [Wed Feb 26 06:13:11 2014] <kishoreg1>: its along the same line but its too > generic > [Wed Feb 26 06:13:30 2014] <kishoreg1>: There is only one ResourceCommand > [Wed Feb 26 06:14:03 2014] <kanakb>: are you suggesting something like > [Wed Feb 26 06:14:14 2014] <kanakb>: administrateResource(command, args) > [Wed Feb 26 06:14:17 2014] <kanakb>: or something to that effect? > [Wed Feb 26 06:14:33 2014] <kishoreg1>: nope > [Wed Feb 26 06:15:15 2014] <kishoreg1>: currently we have > updateResource(ResourceCommand) > [Wed Feb 26 06:16:07 2014] <kishoreg1>: but the usecases are updating > partitions, replicas, rebalancerconfig, provisionerconfig etc > [Wed Feb 26 06:19:37 2014] <kishoreg1>: > https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/HelixAdmin.java > [Wed Feb 26 06:22:11 2014] <kishoreg1>: what do u guys think > [Wed Feb 26 06:23:15 2014] <osgigeek1>: I did not quite follow the part where > kishoreg1 you point to updating partitions, replicas, rebalancerconfig etc > can you elaborate? > [Wed Feb 26 06:23:59 2014] <kishoreg1>: after the resource is created > [Wed Feb 26 06:24:42 2014] <kishoreg1>: we might have to update some parts of > the resources, for example change the number of partitions from 10 to 20 > [Wed Feb 26 06:24:46 2014] <osgigeek1>: ah gotcha > [Wed Feb 26 06:24:50 2014] <kishoreg1>: we have two options > [Wed Feb 26 06:25:17 2014] <kishoreg1>: have a method called > updateParitionNumber(resourceId, numberof partitions) > [Wed Feb 26 06:25:30 2014] <kishoreg1>: or updateResource(ResourceConfig) > [Wed Feb 26 06:25:45 2014] <kishoreg1>: updateResource(ResourceConfig.Delta) > [Wed Feb 26 06:25:51 2014] <osgigeek1>: now I follow > [Wed Feb 26 06:25:54 2014] <kishoreg1>: ok > [Wed Feb 26 06:26:18 2014] <kishoreg1>: so if we just have one method for all > operations, its too overloaded > [Wed Feb 26 06:28:34 2014] <osgigeek1>: ok so lets look at the primary > entities here > [Wed Feb 26 06:29:01 2014] <osgigeek1>: (1) cluster (2) resource (3) instance > [Wed Feb 26 06:29:04 2014] <osgigeek1>: am I correct? > [Wed Feb 26 06:29:25 2014] <kanakb>: arguably partition > [Wed Feb 26 06:29:33 2014] <osgigeek1>: ok so 4 > [Wed Feb 26 06:30:07 2014] <osgigeek1>: so what if we think of having CRUD > operations for those through commands on the HelixAdmin > [Wed Feb 26 06:30:42 2014] <osgigeek1>: everything else which is not a > primary entity lets say we model on the Command or as a derived object off > the correct Command? > [Wed Feb 26 06:31:26 2014] <kishoreg1>: others I can think of are adding > statemodel, constraints etc > [Wed Feb 26 06:31:29 2014] <osgigeek1>: e.g. ResourceCommand, > ResourceReplicaCommand extends ResourceCommand? or should the ResourceCommand > have Replica as a composite ? > [Wed Feb 26 06:31:59 2014] <kishoreg1>: hmm thinking > [Wed Feb 26 06:32:11 2014] <osgigeek1>: I am thinking of statemodel, > constraints all as parts that can be attached to the primary entities > [Wed Feb 26 06:32:25 2014] <osgigeek1>: arguably not attacheable to all but > most I imagine > [Wed Feb 26 06:32:30 2014] <kishoreg1>: yes > [Wed Feb 26 06:33:10 2014] <kishoreg1>: why should it extend resourcecommand > [Wed Feb 26 06:34:02 2014] <osgigeek1>: on a second thought no lets not > extend it, the more I think the more little sense it makes > [Wed Feb 26 06:34:19 2014] <osgigeek1>: I think this is what kanakb modelled > in his wiki > [Wed Feb 26 06:34:33 2014] <osgigeek1>: a builder which allows attaching the > composites into one command > [Wed Feb 26 06:34:39 2014] <osgigeek1>: or in his case config > [Wed Feb 26 06:35:26 2014] <osgigeek1>: So we have a command builder > [Wed Feb 26 06:35:48 2014] <osgigeek1>: > ResourceCommandBuilder().usingStateModel().withConstraints().build() > [Wed Feb 26 06:36:03 2014] <osgigeek1>: > InstanceCommandBuilder().withConstraints().build() > [Wed Feb 26 06:36:30 2014] <kishoreg1>: hmm > [Wed Feb 26 06:36:56 2014] <osgigeek1>: so the key thing I am after is > separating the primary entities > [Wed Feb 26 06:37:01 2014] <osgigeek1>: and calling them out at the top level > [Wed Feb 26 06:37:10 2014] <kishoreg1>: do we need builder pattern for > commands > [Wed Feb 26 06:38:57 2014] <osgigeek1>: Lets ask this question a bit > differently > [Wed Feb 26 06:39:17 2014] <osgigeek1>: how well defined are the commands, do > we think they are pretty well formalized? Do we foresee them changing ? > [Wed Feb 26 06:40:01 2014] <zzhang1>: we may add new fields to the configs > [Wed Feb 26 06:40:16 2014] <kishoreg1>: they might change but most likely we > will be adding new stuff > [Wed Feb 26 06:40:33 2014] <kishoreg1>: yeah, agree with jason > [Wed Feb 26 06:40:42 2014] <osgigeek1>: the new stuff added would that be in > existing buckets of statemodel, constraints? I imagine yes > [Wed Feb 26 06:41:01 2014] <kishoreg1>: yes, > [Wed Feb 26 06:41:41 2014] <osgigeek1>: my instinct tells me its better to > hide the command objects and give builders > [Wed Feb 26 06:41:52 2014] <osgigeek1>: but you guys would be better judges > [Wed Feb 26 06:42:29 2014] <osgigeek1>: when I say hide I mean not give out > the setters > [Wed Feb 26 06:42:30 2014] <kishoreg1>: we can come to builders v/s concrete > after we decide how many command classes we will have > [Wed Feb 26 06:42:40 2014] <osgigeek1>: yeah lets do that > [Wed Feb 26 06:43:16 2014] <kishoreg1>: lets take simple cases like change > partition, replica > [Wed Feb 26 06:43:32 2014] <osgigeek1>: ok > [Wed Feb 26 06:43:45 2014] <kishoreg1>: or some flags in idealstate like > enable/disable > [Wed Feb 26 06:44:01 2014] <kanakb>: enable/disable resource? > [Wed Feb 26 06:44:21 2014] <kishoreg1>: resource, bucketization, group > message mode > [Wed Feb 26 06:44:24 2014] <kishoreg1>: etc > [Wed Feb 26 06:44:32 2014] <kanakb>: right > [Wed Feb 26 06:45:16 2014] <kishoreg1>: so we have these options > [Wed Feb 26 06:46:07 2014] <kishoreg1>: #1 direct method in helixadmin > admin.updateNumPartitions(resourceId, 20) > [Wed Feb 26 06:47:06 2014] <kishoreg1>: #2. config delta method, > resource.delta= …, delta.setpartitions(20), > admin.updateresource(resourcedelta) > [Wed Feb 26 06:47:33 2014] <zzhang1>: i prefer #2, it saves lots of > admin#methods > [Wed Feb 26 06:47:45 2014] <osgigeek1>: may I ask this and you will see where > I am going with this… what do we return on > helixAdmin.addResource(ResourceCommand) > [Wed Feb 26 06:48:04 2014] <osgigeek1>: void? HelixResource? > [Wed Feb 26 06:48:20 2014] <kishoreg1>: #3. command pattern > UpdatePartitionCommand.updatePartitions(20) admin.updateResource(command); > [Wed Feb 26 06:48:34 2014] <kishoreg1>: dunno > [Wed Feb 26 06:48:58 2014] <kishoreg1>: i will be back in 10-15 minutes > [Wed Feb 26 06:48:59 2014] <zzhang1>: currently we are returning void > [Wed Feb 26 06:49:37 2014] <kanakb>: i think it should either: return boolean > or throw a meaningful checked exception > [Wed Feb 26 06:49:59 2014] <osgigeek1>: so should we return a HelixResource? > [Wed Feb 26 06:50:17 2014] <osgigeek1>: rather I meant to ask why not return > a HelixResource? > [Wed Feb 26 06:51:12 2014] <kanakb>: a resource is a resource config plus > runtime state, like its external view > [Wed Feb 26 06:51:18 2014] <kanakb>: initially this external view will always > be empty > [Wed Feb 26 06:51:38 2014] <kanakb>: so it's not super useful to just return > that > [Wed Feb 26 06:52:24 2014] <kishoreg1>: thats another item, shud we mix > runtime and static info in one class > [Wed Feb 26 06:52:48 2014] <kishoreg1>: prefer to defer that to a later stage > [Wed Feb 26 06:52:51 2014] <osgigeek1>: ok > [Wed Feb 26 06:53:05 2014] <osgigeek1>: the reason for the question was > [Wed Feb 26 06:53:24 2014] <osgigeek1>: if we return the object we could have > ResourceCommand.using(object) > [Wed Feb 26 06:53:29 2014] <osgigeek1>: that creates an updateCommand > [Wed Feb 26 06:53:37 2014] <osgigeek1>: which then can carry existing state > [Wed Feb 26 06:53:58 2014] <kanakb>: the hope is that you don't need existing > state in order to make an incremental update > [Wed Feb 26 06:54:34 2014] <osgigeek1>: ok so lets forget that route for now, > lets go back to the 3 options from kishoreg1 > [Wed Feb 26 06:54:48 2014] <osgigeek1>: I am leaning towards #3 > [Wed Feb 26 06:55:03 2014] <kanakb>: yeah i like #3 as well > [Wed Feb 26 06:55:48 2014] <kanakb>: zzhang1: is #3 generic enough to support > things like monitoring config? > [Wed Feb 26 06:55:52 2014] <kanakb>: my intuition is yes > [Wed Feb 26 06:55:59 2014] <zzhang1>: yea > [Wed Feb 26 06:56:49 2014] <zzhang1>: for #3, shall we have a command pattern > for updating each field? > [Wed Feb 26 06:57:34 2014] <kishoreg1>: what ever makes sense > [Wed Feb 26 06:57:51 2014] <osgigeek1>: zzhang1: I think yes that way we > remove APIs like setResourceIdealState from the HelixAdmin > [Wed Feb 26 06:57:52 2014] <zzhang1>: ok > [Wed Feb 26 06:58:03 2014] <kanakb>: can we do something like > updateResource(ResourceCommand... commands)? > [Wed Feb 26 06:58:25 2014] <kishoreg1>: compositecommands? > [Wed Feb 26 06:58:28 2014] <osgigeek1>: ooh, multiple commands you mean? > [Wed Feb 26 06:58:47 2014] <kanakb>: yeah or a composite command > [Wed Feb 26 06:58:57 2014] <osgigeek1>: I think we dont do that for the first > iteration > [Wed Feb 26 06:59:08 2014] <kanakb>: i would imagine that there are certain > configs that we'd like to set together as an atomic unit > [Wed Feb 26 06:59:19 2014] <osgigeek1>: what happens if some commands fail > and some succeed > [Wed Feb 26 06:59:20 2014] <kishoreg1>: yeah i was thinking the same > [Wed Feb 26 06:59:31 2014] <kishoreg1>: same as kanakb > [Wed Feb 26 06:59:41 2014] <kishoreg1>: for example if i want to change both > partition and replica > [Wed Feb 26 06:59:44 2014] <kanakb>: osgigeek1: then they should all succeed > or all fail > [Wed Feb 26 07:00:03 2014] <kanakb>: and it's our responsibility to guarantee > that > [Wed Feb 26 07:00:25 2014] <kanakb>: fortunately this is generally easy to > guarantee in the ZK case > [Wed Feb 26 07:00:38 2014] <kanakb>: as long as things don't span znodes > [Wed Feb 26 07:00:58 2014] <osgigeek1>: kanakb: if we can guarantee that then > sure, I was concerned that it might not be possible given the distributed > nature of the system > [Wed Feb 26 07:01:46 2014] <kanakb>: there is one interesting case > [Wed Feb 26 07:01:48 2014] <kishoreg1>: kanakb: how can we gaurantee things > that span multiple znodes > [Wed Feb 26 07:01:58 2014] <kanakb>: we'd need to do a rollback or something > [Wed Feb 26 07:02:10 2014] <kanakb>: so maybe it's not so easy > [Wed Feb 26 07:02:23 2014] <kanakb>: hmm > [Wed Feb 26 07:02:50 2014] <kanakb>: right now all our admin apis work on a > single znode > [Wed Feb 26 07:02:56 2014] <kanakb>: but now that won't be true > [Wed Feb 26 07:03:21 2014] <kishoreg1>: yeah, we have rely on the spi to make > it atomic > [Wed Feb 26 07:03:44 2014] <kishoreg1>: but we might need composite commands > [Wed Feb 26 07:04:01 2014] <kishoreg1>: for example what if we want to update > both partitions and replicas > [Wed Feb 26 07:04:10 2014] <kanakb>: right > [Wed Feb 26 07:04:23 2014] <kanakb>: so we either need to allow multiple > commands, or a single command that does multiple things > [Wed Feb 26 07:04:29 2014] <osgigeek1>: kishoreg1: we should model that as > one command which carries both partition change and replica change > [Wed Feb 26 07:04:49 2014] <kanakb>: i.e. > ResourceCommand.partitions(10).replicas(3)? > [Wed Feb 26 07:04:53 2014] <osgigeek1>: yes > [Wed Feb 26 07:05:19 2014] <zzhang1>: btw, i think commands like addCluster() > will cross multiple znodes > [Wed Feb 26 07:05:41 2014] <kanakb>: which brings in the atomic api issues > [Wed Feb 26 07:06:12 2014] <kishoreg1>: its upto the implementation, we shud > assume all api's are atomic > [Wed Feb 26 07:06:36 2014] <osgigeek1>: I think we should make that a > principle of the command APIs > [Wed Feb 26 07:06:36 2014] <kishoreg1>: we can use multitransaction updates > in ZK > [Wed Feb 26 07:06:52 2014] <kanakb>: are those available in 3.3.x? > [Wed Feb 26 07:07:09 2014] <kishoreg1>: nope 3.4.5 i think > [Wed Feb 26 07:07:14 2014] <kishoreg1>: but u already have locks > [Wed Feb 26 07:07:18 2014] <kanakb>: yeah > [Wed Feb 26 07:07:21 2014] <kanakb>: i was going to say > [Wed Feb 26 07:07:27 2014] <kishoreg1>: so its ok to rely on that > [Wed Feb 26 07:07:44 2014] <kishoreg1>: what u have is better than multi > transactions i think > [Wed Feb 26 07:08:09 2014] <kanakb>: well yes and no > [Wed Feb 26 07:08:30 2014] <kishoreg1>: anyhow ResourceCommand.partitions(1) > will return a type of ResourceCommand? > [Wed Feb 26 07:09:39 2014] <osgigeek1>: yes, the builder should allow > changing any of the facets i.e. partition or replica of the entity i.e. > Resource in this case > [Wed Feb 26 07:09:47 2014] <kanakb>: chaining probably isn't possible unless > we change it to new ResourceCommand().partitions(1) > [Wed Feb 26 07:10:26 2014] <kishoreg1>: ok, i am not good at builder > patterns, i hope we dont have too many classes > [Wed Feb 26 07:10:38 2014] <kanakb>: that would just be 1 class > [Wed Feb 26 07:10:45 2014] <osgigeek1>: yes > [Wed Feb 26 07:10:56 2014] <kishoreg1>: ok > [Wed Feb 26 07:11:07 2014] <osgigeek1>: i.e. agree with kanakb > [Wed Feb 26 07:11:36 2014] <kishoreg1>: ok > [Wed Feb 26 07:13:26 2014] <osgigeek1>: ok well I might have to drop off in a > few mins … do you guys plan to carry on for long? > [Wed Feb 26 07:13:36 2014] <kanakb>: probably not too long > [Wed Feb 26 07:13:43 2014] <kishoreg1>: i think we shud take break as well > [Wed Feb 26 07:13:44 2014] <kanakb>: at least we have some direction with > admin now > [Wed Feb 26 07:14:03 2014] <kishoreg1>: lets write the apis for administrator > and see how it looks > [Wed Feb 26 07:14:13 2014] <kishoreg1>: i need to get going as well > [Wed Feb 26 07:15:05 2014] <kanakb>: ok let's end here? > [Wed Feb 26 07:15:10 2014] <osgigeek1>: well this was good, we made good > progress, yeah lets end > [Wed Feb 26 07:15:37 2014] <kanakb>: ASFBot: meeting end > > > Meeting ended at Wed Feb 26 07:15:37 2014
