Code sketch: https://code.stypi.com/kanak/helixapimeeting.java
----------------------------------------
> Subject: Summary of IRC Meeting in #apachehelix
> From: [email protected]
> To: [email protected]
> Date: Wed, 26 Feb 2014 07:16:06 +0000
>
> Summary of IRC Meeting in #apachehelix at Wed Feb 26 05:50:29 2014:
>
> Attendees: zzhang1, gmcdonald, osgigeek1, kanakb, kishoreg1
>
> - Preface
>
>
> IRC log follows:
>
> ## Preface ##
> [Wed Feb 26 05:50:35 2014] <gmcdonald>: cool
> [Wed Feb 26 05:50:38 2014] <kanakb>: sweet
> [Wed Feb 26 05:50:44 2014] <gmcdonald>: Im outta here, enjoy
> [Wed Feb 26 05:51:23 2014] <kanakb>: okay anyway
> [Wed Feb 26 05:51:43 2014] <kanakb>: i think we can use the pattern that
> osgigeek1 suggested
> [Wed Feb 26 05:52:07 2014] <kanakb>: i.e.
> [Wed Feb 26 05:52:21 2014] <kanakb>: new
> HelixAdministratorBuilder.usingProvider(HelixStoreProvider.ZOOKEEPER).toAddress(zkAddress).build();
> [Wed Feb 26 05:52:37 2014] <kanakb>: and
> [Wed Feb 26 05:53:00 2014] <kanakb>: new
> HelixAdministratorBuilder.usingProvider(HelixStoreProvider.MYDB).username(username).host(hostname).build()
> [Wed Feb 26 05:54:19 2014] <kishoreg1>: looks good,
> [Wed Feb 26 05:54:50 2014] <kishoreg1>: Sandeep, is Provider the right term ?
> [Wed Feb 26 05:55:00 2014] <osgigeek1>: yes provider is the right term here
> [Wed Feb 26 05:55:30 2014] <kishoreg1>: ok, sounds good
> [Wed Feb 26 05:55:52 2014] <osgigeek1>: ok so we now have an instance of
> HelixAdmin
> [Wed Feb 26 05:55:59 2014] <kanakb>: ok now how does a participant get an
> admin?
> [Wed Feb 26 05:56:16 2014] <kanakb>: participant.getAdmin()?
> [Wed Feb 26 05:56:22 2014] <kanakb>: spectator.getAdmin()?
> [Wed Feb 26 05:57:05 2014] <kanakb>: or it could even be a getter within the
> service class if we go that direction
> [Wed Feb 26 05:57:58 2014] <kanakb>: but i guess we can come back to that
> [Wed Feb 26 05:58:01 2014] <osgigeek1>: so before we go to participant should
> we capture what APIs hang off the HelixAdmin or HelixAdministrator?
> [Wed Feb 26 05:58:07 2014] <kanakb>: yeah sure
> [Wed Feb 26 05:58:43 2014] <kishoreg1>: we have two options
> [Wed Feb 26 05:58:59 2014] <kishoreg1>: #1. flat methods similar to what we
> have
> [Wed Feb 26 05:59:49 2014] <kishoreg1>: create/delete/update/read
> cluster|resource|instance| etc
> [Wed Feb 26 06:00:33 2014] <kishoreg1>: there will be too many methods in this
> [Wed Feb 26 06:00:54 2014] <kishoreg1>: #2. heirarchical based on scope
> [Wed Feb 26 06:01:04 2014] <kishoreg1>: based on scope/entity
> [Wed Feb 26 06:01:26 2014] <kishoreg1>: clusteradmin resourceadmin
> instanceadmin
> [Wed Feb 26 06:01:44 2014] <osgigeek1>: I prefer #1
> [Wed Feb 26 06:02:37 2014] <kishoreg1>: how many methods do we think we will
> have in this interface
> [Wed Feb 26 06:02:40 2014] <kishoreg1>: 100?
> [Wed Feb 26 06:02:59 2014] <kanakb>: probably 50
> [Wed Feb 26 06:03:07 2014] <kanakb>: there are some things we can consolidate
> [Wed Feb 26 06:03:10 2014] <kishoreg1>: ok
> [Wed Feb 26 06:03:20 2014] <kanakb>: e.g. the things that involve userconfig
> [Wed Feb 26 06:03:25 2014] <kanakb>: those can take scope as a parameter
> [Wed Feb 26 06:03:55 2014] <kishoreg1>: sandeep, you should look at the api's
> in the current helixadmin
> [Wed Feb 26 06:03:57 2014] <osgigeek1>: so why do we think it will be 50?
> [Wed Feb 26 06:04:20 2014] <kanakb>: the current number of helixadmin method
> calls is like 20-30
> [Wed Feb 26 06:04:20 2014] <osgigeek1>: looking at it now
> [Wed Feb 26 06:04:31 2014] <kanakb>: so say it doubles because we cover more
> use cases
> [Wed Feb 26 06:04:34 2014] <kanakb>: then it's 50
> [Wed Feb 26 06:04:43 2014] <kishoreg1>: there are too many combinations i
> think, we need to figure out how to reduce this
> [Wed Feb 26 06:05:33 2014] <osgigeek1>: Can we consider like a command
> pattern?
> [Wed Feb 26 06:05:48 2014] <osgigeek1>: like I see several
> addResource(param….)
> [Wed Feb 26 06:05:58 2014] <osgigeek1>: instead how about
> addResource(ResourceCommand)
> [Wed Feb 26 06:06:15 2014] <kishoreg1>: i like that
> [Wed Feb 26 06:06:16 2014] <osgigeek1>: the command encapsulates the
> different params
> [Wed Feb 26 06:06:24 2014] <osgigeek1>: that way we can decipher from the
> command
> [Wed Feb 26 06:06:40 2014] <kanakb>: well, if we're adding using
> ResourceConfig like in the wiki, then is this still necessary?
> [Wed Feb 26 06:08:14 2014] <kishoreg1>: probably yes,
> [Wed Feb 26 06:08:28 2014] <kishoreg1>: lets say we want to only update
> provisionerconfig
> [Wed Feb 26 06:08:36 2014] <kishoreg1>: we have two options delta
> [Wed Feb 26 06:12:01 2014] <osgigeek1>: So kanakb I think if command ==
> config and that was the notion then we can go with config
> [Wed Feb 26 06:12:31 2014] <osgigeek1>: but it appears kishoreg1 is pointing
> to the something missing
> [Wed Feb 26 06:13:11 2014] <kishoreg1>: its along the same line but its too
> generic
> [Wed Feb 26 06:13:30 2014] <kishoreg1>: There is only one ResourceCommand
> [Wed Feb 26 06:14:03 2014] <kanakb>: are you suggesting something like
> [Wed Feb 26 06:14:14 2014] <kanakb>: administrateResource(command, args)
> [Wed Feb 26 06:14:17 2014] <kanakb>: or something to that effect?
> [Wed Feb 26 06:14:33 2014] <kishoreg1>: nope
> [Wed Feb 26 06:15:15 2014] <kishoreg1>: currently we have
> updateResource(ResourceCommand)
> [Wed Feb 26 06:16:07 2014] <kishoreg1>: but the usecases are updating
> partitions, replicas, rebalancerconfig, provisionerconfig etc
> [Wed Feb 26 06:19:37 2014] <kishoreg1>:
> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/HelixAdmin.java
> [Wed Feb 26 06:22:11 2014] <kishoreg1>: what do u guys think
> [Wed Feb 26 06:23:15 2014] <osgigeek1>: I did not quite follow the part where
> kishoreg1 you point to updating partitions, replicas, rebalancerconfig etc
> can you elaborate?
> [Wed Feb 26 06:23:59 2014] <kishoreg1>: after the resource is created
> [Wed Feb 26 06:24:42 2014] <kishoreg1>: we might have to update some parts of
> the resources, for example change the number of partitions from 10 to 20
> [Wed Feb 26 06:24:46 2014] <osgigeek1>: ah gotcha
> [Wed Feb 26 06:24:50 2014] <kishoreg1>: we have two options
> [Wed Feb 26 06:25:17 2014] <kishoreg1>: have a method called
> updateParitionNumber(resourceId, numberof partitions)
> [Wed Feb 26 06:25:30 2014] <kishoreg1>: or updateResource(ResourceConfig)
> [Wed Feb 26 06:25:45 2014] <kishoreg1>: updateResource(ResourceConfig.Delta)
> [Wed Feb 26 06:25:51 2014] <osgigeek1>: now I follow
> [Wed Feb 26 06:25:54 2014] <kishoreg1>: ok
> [Wed Feb 26 06:26:18 2014] <kishoreg1>: so if we just have one method for all
> operations, its too overloaded
> [Wed Feb 26 06:28:34 2014] <osgigeek1>: ok so lets look at the primary
> entities here
> [Wed Feb 26 06:29:01 2014] <osgigeek1>: (1) cluster (2) resource (3) instance
> [Wed Feb 26 06:29:04 2014] <osgigeek1>: am I correct?
> [Wed Feb 26 06:29:25 2014] <kanakb>: arguably partition
> [Wed Feb 26 06:29:33 2014] <osgigeek1>: ok so 4
> [Wed Feb 26 06:30:07 2014] <osgigeek1>: so what if we think of having CRUD
> operations for those through commands on the HelixAdmin
> [Wed Feb 26 06:30:42 2014] <osgigeek1>: everything else which is not a
> primary entity lets say we model on the Command or as a derived object off
> the correct Command?
> [Wed Feb 26 06:31:26 2014] <kishoreg1>: others I can think of are adding
> statemodel, constraints etc
> [Wed Feb 26 06:31:29 2014] <osgigeek1>: e.g. ResourceCommand,
> ResourceReplicaCommand extends ResourceCommand? or should the ResourceCommand
> have Replica as a composite ?
> [Wed Feb 26 06:31:59 2014] <kishoreg1>: hmm thinking
> [Wed Feb 26 06:32:11 2014] <osgigeek1>: I am thinking of statemodel,
> constraints all as parts that can be attached to the primary entities
> [Wed Feb 26 06:32:25 2014] <osgigeek1>: arguably not attacheable to all but
> most I imagine
> [Wed Feb 26 06:32:30 2014] <kishoreg1>: yes
> [Wed Feb 26 06:33:10 2014] <kishoreg1>: why should it extend resourcecommand
> [Wed Feb 26 06:34:02 2014] <osgigeek1>: on a second thought no lets not
> extend it, the more I think the more little sense it makes
> [Wed Feb 26 06:34:19 2014] <osgigeek1>: I think this is what kanakb modelled
> in his wiki
> [Wed Feb 26 06:34:33 2014] <osgigeek1>: a builder which allows attaching the
> composites into one command
> [Wed Feb 26 06:34:39 2014] <osgigeek1>: or in his case config
> [Wed Feb 26 06:35:26 2014] <osgigeek1>: So we have a command builder
> [Wed Feb 26 06:35:48 2014] <osgigeek1>:
> ResourceCommandBuilder().usingStateModel().withConstraints().build()
> [Wed Feb 26 06:36:03 2014] <osgigeek1>:
> InstanceCommandBuilder().withConstraints().build()
> [Wed Feb 26 06:36:30 2014] <kishoreg1>: hmm
> [Wed Feb 26 06:36:56 2014] <osgigeek1>: so the key thing I am after is
> separating the primary entities
> [Wed Feb 26 06:37:01 2014] <osgigeek1>: and calling them out at the top level
> [Wed Feb 26 06:37:10 2014] <kishoreg1>: do we need builder pattern for
> commands
> [Wed Feb 26 06:38:57 2014] <osgigeek1>: Lets ask this question a bit
> differently
> [Wed Feb 26 06:39:17 2014] <osgigeek1>: how well defined are the commands, do
> we think they are pretty well formalized? Do we foresee them changing ?
> [Wed Feb 26 06:40:01 2014] <zzhang1>: we may add new fields to the configs
> [Wed Feb 26 06:40:16 2014] <kishoreg1>: they might change but most likely we
> will be adding new stuff
> [Wed Feb 26 06:40:33 2014] <kishoreg1>: yeah, agree with jason
> [Wed Feb 26 06:40:42 2014] <osgigeek1>: the new stuff added would that be in
> existing buckets of statemodel, constraints? I imagine yes
> [Wed Feb 26 06:41:01 2014] <kishoreg1>: yes,
> [Wed Feb 26 06:41:41 2014] <osgigeek1>: my instinct tells me its better to
> hide the command objects and give builders
> [Wed Feb 26 06:41:52 2014] <osgigeek1>: but you guys would be better judges
> [Wed Feb 26 06:42:29 2014] <osgigeek1>: when I say hide I mean not give out
> the setters
> [Wed Feb 26 06:42:30 2014] <kishoreg1>: we can come to builders v/s concrete
> after we decide how many command classes we will have
> [Wed Feb 26 06:42:40 2014] <osgigeek1>: yeah lets do that
> [Wed Feb 26 06:43:16 2014] <kishoreg1>: lets take simple cases like change
> partition, replica
> [Wed Feb 26 06:43:32 2014] <osgigeek1>: ok
> [Wed Feb 26 06:43:45 2014] <kishoreg1>: or some flags in idealstate like
> enable/disable
> [Wed Feb 26 06:44:01 2014] <kanakb>: enable/disable resource?
> [Wed Feb 26 06:44:21 2014] <kishoreg1>: resource, bucketization, group
> message mode
> [Wed Feb 26 06:44:24 2014] <kishoreg1>: etc
> [Wed Feb 26 06:44:32 2014] <kanakb>: right
> [Wed Feb 26 06:45:16 2014] <kishoreg1>: so we have these options
> [Wed Feb 26 06:46:07 2014] <kishoreg1>: #1 direct method in helixadmin
> admin.updateNumPartitions(resourceId, 20)
> [Wed Feb 26 06:47:06 2014] <kishoreg1>: #2. config delta method,
> resource.delta= …, delta.setpartitions(20),
> admin.updateresource(resourcedelta)
> [Wed Feb 26 06:47:33 2014] <zzhang1>: i prefer #2, it saves lots of
> admin#methods
> [Wed Feb 26 06:47:45 2014] <osgigeek1>: may I ask this and you will see where
> I am going with this… what do we return on
> helixAdmin.addResource(ResourceCommand)
> [Wed Feb 26 06:48:04 2014] <osgigeek1>: void? HelixResource?
> [Wed Feb 26 06:48:20 2014] <kishoreg1>: #3. command pattern
> UpdatePartitionCommand.updatePartitions(20) admin.updateResource(command);
> [Wed Feb 26 06:48:34 2014] <kishoreg1>: dunno
> [Wed Feb 26 06:48:58 2014] <kishoreg1>: i will be back in 10-15 minutes
> [Wed Feb 26 06:48:59 2014] <zzhang1>: currently we are returning void
> [Wed Feb 26 06:49:37 2014] <kanakb>: i think it should either: return boolean
> or throw a meaningful checked exception
> [Wed Feb 26 06:49:59 2014] <osgigeek1>: so should we return a HelixResource?
> [Wed Feb 26 06:50:17 2014] <osgigeek1>: rather I meant to ask why not return
> a HelixResource?
> [Wed Feb 26 06:51:12 2014] <kanakb>: a resource is a resource config plus
> runtime state, like its external view
> [Wed Feb 26 06:51:18 2014] <kanakb>: initially this external view will always
> be empty
> [Wed Feb 26 06:51:38 2014] <kanakb>: so it's not super useful to just return
> that
> [Wed Feb 26 06:52:24 2014] <kishoreg1>: thats another item, shud we mix
> runtime and static info in one class
> [Wed Feb 26 06:52:48 2014] <kishoreg1>: prefer to defer that to a later stage
> [Wed Feb 26 06:52:51 2014] <osgigeek1>: ok
> [Wed Feb 26 06:53:05 2014] <osgigeek1>: the reason for the question was
> [Wed Feb 26 06:53:24 2014] <osgigeek1>: if we return the object we could have
> ResourceCommand.using(object)
> [Wed Feb 26 06:53:29 2014] <osgigeek1>: that creates an updateCommand
> [Wed Feb 26 06:53:37 2014] <osgigeek1>: which then can carry existing state
> [Wed Feb 26 06:53:58 2014] <kanakb>: the hope is that you don't need existing
> state in order to make an incremental update
> [Wed Feb 26 06:54:34 2014] <osgigeek1>: ok so lets forget that route for now,
> lets go back to the 3 options from kishoreg1
> [Wed Feb 26 06:54:48 2014] <osgigeek1>: I am leaning towards #3
> [Wed Feb 26 06:55:03 2014] <kanakb>: yeah i like #3 as well
> [Wed Feb 26 06:55:48 2014] <kanakb>: zzhang1: is #3 generic enough to support
> things like monitoring config?
> [Wed Feb 26 06:55:52 2014] <kanakb>: my intuition is yes
> [Wed Feb 26 06:55:59 2014] <zzhang1>: yea
> [Wed Feb 26 06:56:49 2014] <zzhang1>: for #3, shall we have a command pattern
> for updating each field?
> [Wed Feb 26 06:57:34 2014] <kishoreg1>: what ever makes sense
> [Wed Feb 26 06:57:51 2014] <osgigeek1>: zzhang1: I think yes that way we
> remove APIs like setResourceIdealState from the HelixAdmin
> [Wed Feb 26 06:57:52 2014] <zzhang1>: ok
> [Wed Feb 26 06:58:03 2014] <kanakb>: can we do something like
> updateResource(ResourceCommand... commands)?
> [Wed Feb 26 06:58:25 2014] <kishoreg1>: compositecommands?
> [Wed Feb 26 06:58:28 2014] <osgigeek1>: ooh, multiple commands you mean?
> [Wed Feb 26 06:58:47 2014] <kanakb>: yeah or a composite command
> [Wed Feb 26 06:58:57 2014] <osgigeek1>: I think we dont do that for the first
> iteration
> [Wed Feb 26 06:59:08 2014] <kanakb>: i would imagine that there are certain
> configs that we'd like to set together as an atomic unit
> [Wed Feb 26 06:59:19 2014] <osgigeek1>: what happens if some commands fail
> and some succeed
> [Wed Feb 26 06:59:20 2014] <kishoreg1>: yeah i was thinking the same
> [Wed Feb 26 06:59:31 2014] <kishoreg1>: same as kanakb
> [Wed Feb 26 06:59:41 2014] <kishoreg1>: for example if i want to change both
> partition and replica
> [Wed Feb 26 06:59:44 2014] <kanakb>: osgigeek1: then they should all succeed
> or all fail
> [Wed Feb 26 07:00:03 2014] <kanakb>: and it's our responsibility to guarantee
> that
> [Wed Feb 26 07:00:25 2014] <kanakb>: fortunately this is generally easy to
> guarantee in the ZK case
> [Wed Feb 26 07:00:38 2014] <kanakb>: as long as things don't span znodes
> [Wed Feb 26 07:00:58 2014] <osgigeek1>: kanakb: if we can guarantee that then
> sure, I was concerned that it might not be possible given the distributed
> nature of the system
> [Wed Feb 26 07:01:46 2014] <kanakb>: there is one interesting case
> [Wed Feb 26 07:01:48 2014] <kishoreg1>: kanakb: how can we gaurantee things
> that span multiple znodes
> [Wed Feb 26 07:01:58 2014] <kanakb>: we'd need to do a rollback or something
> [Wed Feb 26 07:02:10 2014] <kanakb>: so maybe it's not so easy
> [Wed Feb 26 07:02:23 2014] <kanakb>: hmm
> [Wed Feb 26 07:02:50 2014] <kanakb>: right now all our admin apis work on a
> single znode
> [Wed Feb 26 07:02:56 2014] <kanakb>: but now that won't be true
> [Wed Feb 26 07:03:21 2014] <kishoreg1>: yeah, we have rely on the spi to make
> it atomic
> [Wed Feb 26 07:03:44 2014] <kishoreg1>: but we might need composite commands
> [Wed Feb 26 07:04:01 2014] <kishoreg1>: for example what if we want to update
> both partitions and replicas
> [Wed Feb 26 07:04:10 2014] <kanakb>: right
> [Wed Feb 26 07:04:23 2014] <kanakb>: so we either need to allow multiple
> commands, or a single command that does multiple things
> [Wed Feb 26 07:04:29 2014] <osgigeek1>: kishoreg1: we should model that as
> one command which carries both partition change and replica change
> [Wed Feb 26 07:04:49 2014] <kanakb>: i.e.
> ResourceCommand.partitions(10).replicas(3)?
> [Wed Feb 26 07:04:53 2014] <osgigeek1>: yes
> [Wed Feb 26 07:05:19 2014] <zzhang1>: btw, i think commands like addCluster()
> will cross multiple znodes
> [Wed Feb 26 07:05:41 2014] <kanakb>: which brings in the atomic api issues
> [Wed Feb 26 07:06:12 2014] <kishoreg1>: its upto the implementation, we shud
> assume all api's are atomic
> [Wed Feb 26 07:06:36 2014] <osgigeek1>: I think we should make that a
> principle of the command APIs
> [Wed Feb 26 07:06:36 2014] <kishoreg1>: we can use multitransaction updates
> in ZK
> [Wed Feb 26 07:06:52 2014] <kanakb>: are those available in 3.3.x?
> [Wed Feb 26 07:07:09 2014] <kishoreg1>: nope 3.4.5 i think
> [Wed Feb 26 07:07:14 2014] <kishoreg1>: but u already have locks
> [Wed Feb 26 07:07:18 2014] <kanakb>: yeah
> [Wed Feb 26 07:07:21 2014] <kanakb>: i was going to say
> [Wed Feb 26 07:07:27 2014] <kishoreg1>: so its ok to rely on that
> [Wed Feb 26 07:07:44 2014] <kishoreg1>: what u have is better than multi
> transactions i think
> [Wed Feb 26 07:08:09 2014] <kanakb>: well yes and no
> [Wed Feb 26 07:08:30 2014] <kishoreg1>: anyhow ResourceCommand.partitions(1)
> will return a type of ResourceCommand?
> [Wed Feb 26 07:09:39 2014] <osgigeek1>: yes, the builder should allow
> changing any of the facets i.e. partition or replica of the entity i.e.
> Resource in this case
> [Wed Feb 26 07:09:47 2014] <kanakb>: chaining probably isn't possible unless
> we change it to new ResourceCommand().partitions(1)
> [Wed Feb 26 07:10:26 2014] <kishoreg1>: ok, i am not good at builder
> patterns, i hope we dont have too many classes
> [Wed Feb 26 07:10:38 2014] <kanakb>: that would just be 1 class
> [Wed Feb 26 07:10:45 2014] <osgigeek1>: yes
> [Wed Feb 26 07:10:56 2014] <kishoreg1>: ok
> [Wed Feb 26 07:11:07 2014] <osgigeek1>: i.e. agree with kanakb
> [Wed Feb 26 07:11:36 2014] <kishoreg1>: ok
> [Wed Feb 26 07:13:26 2014] <osgigeek1>: ok well I might have to drop off in a
> few mins … do you guys plan to carry on for long?
> [Wed Feb 26 07:13:36 2014] <kanakb>: probably not too long
> [Wed Feb 26 07:13:43 2014] <kishoreg1>: i think we shud take break as well
> [Wed Feb 26 07:13:44 2014] <kanakb>: at least we have some direction with
> admin now
> [Wed Feb 26 07:14:03 2014] <kishoreg1>: lets write the apis for administrator
> and see how it looks
> [Wed Feb 26 07:14:13 2014] <kishoreg1>: i need to get going as well
> [Wed Feb 26 07:15:05 2014] <kanakb>: ok let's end here?
> [Wed Feb 26 07:15:10 2014] <osgigeek1>: well this was good, we made good
> progress, yeah lets end
> [Wed Feb 26 07:15:37 2014] <kanakb>: ASFBot: meeting end
>
>
> Meeting ended at Wed Feb 26 07:15:37 2014
