Hi Henry, We don't currently have scheduled meetings -- this one just happened ad-hoc. Do you think this is useful/valuable? If so, what frequency do you think makes sense?
Other devs: same question. Kanak ---------------------------------------- > Date: Thu, 27 Feb 2014 13:16:35 -0800 > Subject: Re: Summary of IRC Meeting in #apachehelix > From: [email protected] > To: [email protected] > > Hi Guys, > > Do you guys do this (IIRC chat) every Wednesdays? > > - Henry > > On Tue, Feb 25, 2014 at 11:16 PM, ASF IRC Bot > <[email protected]> wrote: >> Summary of IRC Meeting in #apachehelix at Wed Feb 26 05:50:29 2014: >> >> Attendees: zzhang1, gmcdonald, osgigeek1, kanakb, kishoreg1 >> >> - Preface >> >> >> IRC log follows: >> >> ## Preface ## >> [Wed Feb 26 05:50:35 2014] <gmcdonald>: cool >> [Wed Feb 26 05:50:38 2014] <kanakb>: sweet >> [Wed Feb 26 05:50:44 2014] <gmcdonald>: Im outta here, enjoy >> [Wed Feb 26 05:51:23 2014] <kanakb>: okay anyway >> [Wed Feb 26 05:51:43 2014] <kanakb>: i think we can use the pattern that >> osgigeek1 suggested >> [Wed Feb 26 05:52:07 2014] <kanakb>: i.e. >> [Wed Feb 26 05:52:21 2014] <kanakb>: new >> HelixAdministratorBuilder.usingProvider(HelixStoreProvider.ZOOKEEPER).toAddress(zkAddress).build(); >> [Wed Feb 26 05:52:37 2014] <kanakb>: and >> [Wed Feb 26 05:53:00 2014] <kanakb>: new >> HelixAdministratorBuilder.usingProvider(HelixStoreProvider.MYDB).username(username).host(hostname).build() >> [Wed Feb 26 05:54:19 2014] <kishoreg1>: looks good, >> [Wed Feb 26 05:54:50 2014] <kishoreg1>: Sandeep, is Provider the right term ? >> [Wed Feb 26 05:55:00 2014] <osgigeek1>: yes provider is the right term here >> [Wed Feb 26 05:55:30 2014] <kishoreg1>: ok, sounds good >> [Wed Feb 26 05:55:52 2014] <osgigeek1>: ok so we now have an instance of >> HelixAdmin >> [Wed Feb 26 05:55:59 2014] <kanakb>: ok now how does a participant get an >> admin? >> [Wed Feb 26 05:56:16 2014] <kanakb>: participant.getAdmin()? >> [Wed Feb 26 05:56:22 2014] <kanakb>: spectator.getAdmin()? >> [Wed Feb 26 05:57:05 2014] <kanakb>: or it could even be a getter within the >> service class if we go that direction >> [Wed Feb 26 05:57:58 2014] <kanakb>: but i guess we can come back to that >> [Wed Feb 26 05:58:01 2014] <osgigeek1>: so before we go to participant >> should we capture what APIs hang off the HelixAdmin or HelixAdministrator? >> [Wed Feb 26 05:58:07 2014] <kanakb>: yeah sure >> [Wed Feb 26 05:58:43 2014] <kishoreg1>: we have two options >> [Wed Feb 26 05:58:59 2014] <kishoreg1>: #1. flat methods similar to what we >> have >> [Wed Feb 26 05:59:49 2014] <kishoreg1>: create/delete/update/read >> cluster|resource|instance| etc >> [Wed Feb 26 06:00:33 2014] <kishoreg1>: there will be too many methods in >> this >> [Wed Feb 26 06:00:54 2014] <kishoreg1>: #2. heirarchical based on scope >> [Wed Feb 26 06:01:04 2014] <kishoreg1>: based on scope/entity >> [Wed Feb 26 06:01:26 2014] <kishoreg1>: clusteradmin resourceadmin >> instanceadmin >> [Wed Feb 26 06:01:44 2014] <osgigeek1>: I prefer #1 >> [Wed Feb 26 06:02:37 2014] <kishoreg1>: how many methods do we think we will >> have in this interface >> [Wed Feb 26 06:02:40 2014] <kishoreg1>: 100? >> [Wed Feb 26 06:02:59 2014] <kanakb>: probably 50 >> [Wed Feb 26 06:03:07 2014] <kanakb>: there are some things we can consolidate >> [Wed Feb 26 06:03:10 2014] <kishoreg1>: ok >> [Wed Feb 26 06:03:20 2014] <kanakb>: e.g. the things that involve userconfig >> [Wed Feb 26 06:03:25 2014] <kanakb>: those can take scope as a parameter >> [Wed Feb 26 06:03:55 2014] <kishoreg1>: sandeep, you should look at the >> api's in the current helixadmin >> [Wed Feb 26 06:03:57 2014] <osgigeek1>: so why do we think it will be 50? >> [Wed Feb 26 06:04:20 2014] <kanakb>: the current number of helixadmin method >> calls is like 20-30 >> [Wed Feb 26 06:04:20 2014] <osgigeek1>: looking at it now >> [Wed Feb 26 06:04:31 2014] <kanakb>: so say it doubles because we cover more >> use cases >> [Wed Feb 26 06:04:34 2014] <kanakb>: then it's 50 >> [Wed Feb 26 06:04:43 2014] <kishoreg1>: there are too many combinations i >> think, we need to figure out how to reduce this >> [Wed Feb 26 06:05:33 2014] <osgigeek1>: Can we consider like a command >> pattern? >> [Wed Feb 26 06:05:48 2014] <osgigeek1>: like I see several >> addResource(param….) >> [Wed Feb 26 06:05:58 2014] <osgigeek1>: instead how about >> addResource(ResourceCommand) >> [Wed Feb 26 06:06:15 2014] <kishoreg1>: i like that >> [Wed Feb 26 06:06:16 2014] <osgigeek1>: the command encapsulates the >> different params >> [Wed Feb 26 06:06:24 2014] <osgigeek1>: that way we can decipher from the >> command >> [Wed Feb 26 06:06:40 2014] <kanakb>: well, if we're adding using >> ResourceConfig like in the wiki, then is this still necessary? >> [Wed Feb 26 06:08:14 2014] <kishoreg1>: probably yes, >> [Wed Feb 26 06:08:28 2014] <kishoreg1>: lets say we want to only update >> provisionerconfig >> [Wed Feb 26 06:08:36 2014] <kishoreg1>: we have two options delta >> [Wed Feb 26 06:12:01 2014] <osgigeek1>: So kanakb I think if command == >> config and that was the notion then we can go with config >> [Wed Feb 26 06:12:31 2014] <osgigeek1>: but it appears kishoreg1 is pointing >> to the something missing >> [Wed Feb 26 06:13:11 2014] <kishoreg1>: its along the same line but its too >> generic >> [Wed Feb 26 06:13:30 2014] <kishoreg1>: There is only one ResourceCommand >> [Wed Feb 26 06:14:03 2014] <kanakb>: are you suggesting something like >> [Wed Feb 26 06:14:14 2014] <kanakb>: administrateResource(command, args) >> [Wed Feb 26 06:14:17 2014] <kanakb>: or something to that effect? >> [Wed Feb 26 06:14:33 2014] <kishoreg1>: nope >> [Wed Feb 26 06:15:15 2014] <kishoreg1>: currently we have >> updateResource(ResourceCommand) >> [Wed Feb 26 06:16:07 2014] <kishoreg1>: but the usecases are updating >> partitions, replicas, rebalancerconfig, provisionerconfig etc >> [Wed Feb 26 06:19:37 2014] <kishoreg1>: >> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/HelixAdmin.java >> [Wed Feb 26 06:22:11 2014] <kishoreg1>: what do u guys think >> [Wed Feb 26 06:23:15 2014] <osgigeek1>: I did not quite follow the part >> where kishoreg1 you point to updating partitions, replicas, rebalancerconfig >> etc can you elaborate? >> [Wed Feb 26 06:23:59 2014] <kishoreg1>: after the resource is created >> [Wed Feb 26 06:24:42 2014] <kishoreg1>: we might have to update some parts >> of the resources, for example change the number of partitions from 10 to 20 >> [Wed Feb 26 06:24:46 2014] <osgigeek1>: ah gotcha >> [Wed Feb 26 06:24:50 2014] <kishoreg1>: we have two options >> [Wed Feb 26 06:25:17 2014] <kishoreg1>: have a method called >> updateParitionNumber(resourceId, numberof partitions) >> [Wed Feb 26 06:25:30 2014] <kishoreg1>: or updateResource(ResourceConfig) >> [Wed Feb 26 06:25:45 2014] <kishoreg1>: updateResource(ResourceConfig.Delta) >> [Wed Feb 26 06:25:51 2014] <osgigeek1>: now I follow >> [Wed Feb 26 06:25:54 2014] <kishoreg1>: ok >> [Wed Feb 26 06:26:18 2014] <kishoreg1>: so if we just have one method for >> all operations, its too overloaded >> [Wed Feb 26 06:28:34 2014] <osgigeek1>: ok so lets look at the primary >> entities here >> [Wed Feb 26 06:29:01 2014] <osgigeek1>: (1) cluster (2) resource (3) instance >> [Wed Feb 26 06:29:04 2014] <osgigeek1>: am I correct? >> [Wed Feb 26 06:29:25 2014] <kanakb>: arguably partition >> [Wed Feb 26 06:29:33 2014] <osgigeek1>: ok so 4 >> [Wed Feb 26 06:30:07 2014] <osgigeek1>: so what if we think of having CRUD >> operations for those through commands on the HelixAdmin >> [Wed Feb 26 06:30:42 2014] <osgigeek1>: everything else which is not a >> primary entity lets say we model on the Command or as a derived object off >> the correct Command? >> [Wed Feb 26 06:31:26 2014] <kishoreg1>: others I can think of are adding >> statemodel, constraints etc >> [Wed Feb 26 06:31:29 2014] <osgigeek1>: e.g. ResourceCommand, >> ResourceReplicaCommand extends ResourceCommand? or should the >> ResourceCommand have Replica as a composite ? >> [Wed Feb 26 06:31:59 2014] <kishoreg1>: hmm thinking >> [Wed Feb 26 06:32:11 2014] <osgigeek1>: I am thinking of statemodel, >> constraints all as parts that can be attached to the primary entities >> [Wed Feb 26 06:32:25 2014] <osgigeek1>: arguably not attacheable to all but >> most I imagine >> [Wed Feb 26 06:32:30 2014] <kishoreg1>: yes >> [Wed Feb 26 06:33:10 2014] <kishoreg1>: why should it extend resourcecommand >> [Wed Feb 26 06:34:02 2014] <osgigeek1>: on a second thought no lets not >> extend it, the more I think the more little sense it makes >> [Wed Feb 26 06:34:19 2014] <osgigeek1>: I think this is what kanakb modelled >> in his wiki >> [Wed Feb 26 06:34:33 2014] <osgigeek1>: a builder which allows attaching the >> composites into one command >> [Wed Feb 26 06:34:39 2014] <osgigeek1>: or in his case config >> [Wed Feb 26 06:35:26 2014] <osgigeek1>: So we have a command builder >> [Wed Feb 26 06:35:48 2014] <osgigeek1>: >> ResourceCommandBuilder().usingStateModel().withConstraints().build() >> [Wed Feb 26 06:36:03 2014] <osgigeek1>: >> InstanceCommandBuilder().withConstraints().build() >> [Wed Feb 26 06:36:30 2014] <kishoreg1>: hmm >> [Wed Feb 26 06:36:56 2014] <osgigeek1>: so the key thing I am after is >> separating the primary entities >> [Wed Feb 26 06:37:01 2014] <osgigeek1>: and calling them out at the top level >> [Wed Feb 26 06:37:10 2014] <kishoreg1>: do we need builder pattern for >> commands >> [Wed Feb 26 06:38:57 2014] <osgigeek1>: Lets ask this question a bit >> differently >> [Wed Feb 26 06:39:17 2014] <osgigeek1>: how well defined are the commands, >> do we think they are pretty well formalized? Do we foresee them changing ? >> [Wed Feb 26 06:40:01 2014] <zzhang1>: we may add new fields to the configs >> [Wed Feb 26 06:40:16 2014] <kishoreg1>: they might change but most likely we >> will be adding new stuff >> [Wed Feb 26 06:40:33 2014] <kishoreg1>: yeah, agree with jason >> [Wed Feb 26 06:40:42 2014] <osgigeek1>: the new stuff added would that be in >> existing buckets of statemodel, constraints? I imagine yes >> [Wed Feb 26 06:41:01 2014] <kishoreg1>: yes, >> [Wed Feb 26 06:41:41 2014] <osgigeek1>: my instinct tells me its better to >> hide the command objects and give builders >> [Wed Feb 26 06:41:52 2014] <osgigeek1>: but you guys would be better judges >> [Wed Feb 26 06:42:29 2014] <osgigeek1>: when I say hide I mean not give out >> the setters >> [Wed Feb 26 06:42:30 2014] <kishoreg1>: we can come to builders v/s concrete >> after we decide how many command classes we will have >> [Wed Feb 26 06:42:40 2014] <osgigeek1>: yeah lets do that >> [Wed Feb 26 06:43:16 2014] <kishoreg1>: lets take simple cases like change >> partition, replica >> [Wed Feb 26 06:43:32 2014] <osgigeek1>: ok >> [Wed Feb 26 06:43:45 2014] <kishoreg1>: or some flags in idealstate like >> enable/disable >> [Wed Feb 26 06:44:01 2014] <kanakb>: enable/disable resource? >> [Wed Feb 26 06:44:21 2014] <kishoreg1>: resource, bucketization, group >> message mode >> [Wed Feb 26 06:44:24 2014] <kishoreg1>: etc >> [Wed Feb 26 06:44:32 2014] <kanakb>: right >> [Wed Feb 26 06:45:16 2014] <kishoreg1>: so we have these options >> [Wed Feb 26 06:46:07 2014] <kishoreg1>: #1 direct method in helixadmin >> admin.updateNumPartitions(resourceId, 20) >> [Wed Feb 26 06:47:06 2014] <kishoreg1>: #2. config delta method, >> resource.delta= …, delta.setpartitions(20), >> admin.updateresource(resourcedelta) >> [Wed Feb 26 06:47:33 2014] <zzhang1>: i prefer #2, it saves lots of >> admin#methods >> [Wed Feb 26 06:47:45 2014] <osgigeek1>: may I ask this and you will see >> where I am going with this… what do we return on >> helixAdmin.addResource(ResourceCommand) >> [Wed Feb 26 06:48:04 2014] <osgigeek1>: void? HelixResource? >> [Wed Feb 26 06:48:20 2014] <kishoreg1>: #3. command pattern >> UpdatePartitionCommand.updatePartitions(20) admin.updateResource(command); >> [Wed Feb 26 06:48:34 2014] <kishoreg1>: dunno >> [Wed Feb 26 06:48:58 2014] <kishoreg1>: i will be back in 10-15 minutes >> [Wed Feb 26 06:48:59 2014] <zzhang1>: currently we are returning void >> [Wed Feb 26 06:49:37 2014] <kanakb>: i think it should either: return >> boolean or throw a meaningful checked exception >> [Wed Feb 26 06:49:59 2014] <osgigeek1>: so should we return a HelixResource? >> [Wed Feb 26 06:50:17 2014] <osgigeek1>: rather I meant to ask why not return >> a HelixResource? >> [Wed Feb 26 06:51:12 2014] <kanakb>: a resource is a resource config plus >> runtime state, like its external view >> [Wed Feb 26 06:51:18 2014] <kanakb>: initially this external view will >> always be empty >> [Wed Feb 26 06:51:38 2014] <kanakb>: so it's not super useful to just return >> that >> [Wed Feb 26 06:52:24 2014] <kishoreg1>: thats another item, shud we mix >> runtime and static info in one class >> [Wed Feb 26 06:52:48 2014] <kishoreg1>: prefer to defer that to a later stage >> [Wed Feb 26 06:52:51 2014] <osgigeek1>: ok >> [Wed Feb 26 06:53:05 2014] <osgigeek1>: the reason for the question was >> [Wed Feb 26 06:53:24 2014] <osgigeek1>: if we return the object we could >> have ResourceCommand.using(object) >> [Wed Feb 26 06:53:29 2014] <osgigeek1>: that creates an updateCommand >> [Wed Feb 26 06:53:37 2014] <osgigeek1>: which then can carry existing state >> [Wed Feb 26 06:53:58 2014] <kanakb>: the hope is that you don't need >> existing state in order to make an incremental update >> [Wed Feb 26 06:54:34 2014] <osgigeek1>: ok so lets forget that route for >> now, lets go back to the 3 options from kishoreg1 >> [Wed Feb 26 06:54:48 2014] <osgigeek1>: I am leaning towards #3 >> [Wed Feb 26 06:55:03 2014] <kanakb>: yeah i like #3 as well >> [Wed Feb 26 06:55:48 2014] <kanakb>: zzhang1: is #3 generic enough to >> support things like monitoring config? >> [Wed Feb 26 06:55:52 2014] <kanakb>: my intuition is yes >> [Wed Feb 26 06:55:59 2014] <zzhang1>: yea >> [Wed Feb 26 06:56:49 2014] <zzhang1>: for #3, shall we have a command >> pattern for updating each field? >> [Wed Feb 26 06:57:34 2014] <kishoreg1>: what ever makes sense >> [Wed Feb 26 06:57:51 2014] <osgigeek1>: zzhang1: I think yes that way we >> remove APIs like setResourceIdealState from the HelixAdmin >> [Wed Feb 26 06:57:52 2014] <zzhang1>: ok >> [Wed Feb 26 06:58:03 2014] <kanakb>: can we do something like >> updateResource(ResourceCommand... commands)? >> [Wed Feb 26 06:58:25 2014] <kishoreg1>: compositecommands? >> [Wed Feb 26 06:58:28 2014] <osgigeek1>: ooh, multiple commands you mean? >> [Wed Feb 26 06:58:47 2014] <kanakb>: yeah or a composite command >> [Wed Feb 26 06:58:57 2014] <osgigeek1>: I think we dont do that for the >> first iteration >> [Wed Feb 26 06:59:08 2014] <kanakb>: i would imagine that there are certain >> configs that we'd like to set together as an atomic unit >> [Wed Feb 26 06:59:19 2014] <osgigeek1>: what happens if some commands fail >> and some succeed >> [Wed Feb 26 06:59:20 2014] <kishoreg1>: yeah i was thinking the same >> [Wed Feb 26 06:59:31 2014] <kishoreg1>: same as kanakb >> [Wed Feb 26 06:59:41 2014] <kishoreg1>: for example if i want to change both >> partition and replica >> [Wed Feb 26 06:59:44 2014] <kanakb>: osgigeek1: then they should all succeed >> or all fail >> [Wed Feb 26 07:00:03 2014] <kanakb>: and it's our responsibility to >> guarantee that >> [Wed Feb 26 07:00:25 2014] <kanakb>: fortunately this is generally easy to >> guarantee in the ZK case >> [Wed Feb 26 07:00:38 2014] <kanakb>: as long as things don't span znodes >> [Wed Feb 26 07:00:58 2014] <osgigeek1>: kanakb: if we can guarantee that >> then sure, I was concerned that it might not be possible given the >> distributed nature of the system >> [Wed Feb 26 07:01:46 2014] <kanakb>: there is one interesting case >> [Wed Feb 26 07:01:48 2014] <kishoreg1>: kanakb: how can we gaurantee things >> that span multiple znodes >> [Wed Feb 26 07:01:58 2014] <kanakb>: we'd need to do a rollback or something >> [Wed Feb 26 07:02:10 2014] <kanakb>: so maybe it's not so easy >> [Wed Feb 26 07:02:23 2014] <kanakb>: hmm >> [Wed Feb 26 07:02:50 2014] <kanakb>: right now all our admin apis work on a >> single znode >> [Wed Feb 26 07:02:56 2014] <kanakb>: but now that won't be true >> [Wed Feb 26 07:03:21 2014] <kishoreg1>: yeah, we have rely on the spi to >> make it atomic >> [Wed Feb 26 07:03:44 2014] <kishoreg1>: but we might need composite commands >> [Wed Feb 26 07:04:01 2014] <kishoreg1>: for example what if we want to >> update both partitions and replicas >> [Wed Feb 26 07:04:10 2014] <kanakb>: right >> [Wed Feb 26 07:04:23 2014] <kanakb>: so we either need to allow multiple >> commands, or a single command that does multiple things >> [Wed Feb 26 07:04:29 2014] <osgigeek1>: kishoreg1: we should model that as >> one command which carries both partition change and replica change >> [Wed Feb 26 07:04:49 2014] <kanakb>: i.e. >> ResourceCommand.partitions(10).replicas(3)? >> [Wed Feb 26 07:04:53 2014] <osgigeek1>: yes >> [Wed Feb 26 07:05:19 2014] <zzhang1>: btw, i think commands like >> addCluster() will cross multiple znodes >> [Wed Feb 26 07:05:41 2014] <kanakb>: which brings in the atomic api issues >> [Wed Feb 26 07:06:12 2014] <kishoreg1>: its upto the implementation, we shud >> assume all api's are atomic >> [Wed Feb 26 07:06:36 2014] <osgigeek1>: I think we should make that a >> principle of the command APIs >> [Wed Feb 26 07:06:36 2014] <kishoreg1>: we can use multitransaction updates >> in ZK >> [Wed Feb 26 07:06:52 2014] <kanakb>: are those available in 3.3.x? >> [Wed Feb 26 07:07:09 2014] <kishoreg1>: nope 3.4.5 i think >> [Wed Feb 26 07:07:14 2014] <kishoreg1>: but u already have locks >> [Wed Feb 26 07:07:18 2014] <kanakb>: yeah >> [Wed Feb 26 07:07:21 2014] <kanakb>: i was going to say >> [Wed Feb 26 07:07:27 2014] <kishoreg1>: so its ok to rely on that >> [Wed Feb 26 07:07:44 2014] <kishoreg1>: what u have is better than multi >> transactions i think >> [Wed Feb 26 07:08:09 2014] <kanakb>: well yes and no >> [Wed Feb 26 07:08:30 2014] <kishoreg1>: anyhow ResourceCommand.partitions(1) >> will return a type of ResourceCommand? >> [Wed Feb 26 07:09:39 2014] <osgigeek1>: yes, the builder should allow >> changing any of the facets i.e. partition or replica of the entity i.e. >> Resource in this case >> [Wed Feb 26 07:09:47 2014] <kanakb>: chaining probably isn't possible unless >> we change it to new ResourceCommand().partitions(1) >> [Wed Feb 26 07:10:26 2014] <kishoreg1>: ok, i am not good at builder >> patterns, i hope we dont have too many classes >> [Wed Feb 26 07:10:38 2014] <kanakb>: that would just be 1 class >> [Wed Feb 26 07:10:45 2014] <osgigeek1>: yes >> [Wed Feb 26 07:10:56 2014] <kishoreg1>: ok >> [Wed Feb 26 07:11:07 2014] <osgigeek1>: i.e. agree with kanakb >> [Wed Feb 26 07:11:36 2014] <kishoreg1>: ok >> [Wed Feb 26 07:13:26 2014] <osgigeek1>: ok well I might have to drop off in >> a few mins … do you guys plan to carry on for long? >> [Wed Feb 26 07:13:36 2014] <kanakb>: probably not too long >> [Wed Feb 26 07:13:43 2014] <kishoreg1>: i think we shud take break as well >> [Wed Feb 26 07:13:44 2014] <kanakb>: at least we have some direction with >> admin now >> [Wed Feb 26 07:14:03 2014] <kishoreg1>: lets write the apis for >> administrator and see how it looks >> [Wed Feb 26 07:14:13 2014] <kishoreg1>: i need to get going as well >> [Wed Feb 26 07:15:05 2014] <kanakb>: ok let's end here? >> [Wed Feb 26 07:15:10 2014] <osgigeek1>: well this was good, we made good >> progress, yeah lets end >> [Wed Feb 26 07:15:37 2014] <kanakb>: ASFBot: meeting end >> >> >> Meeting ended at Wed Feb 26 07:15:37 2014
