Summary of IRC Meeting in #apachehelix at Wed Mar 19 06:03:54 2014: Attendees: kishoreg, osgigeek, kanakb
- Preface - quick listener discussion IRC log follows: ## Preface ## [Wed Mar 19 06:04:02 2014] <kishoreg>: currently we have StateChangeListeners ## quick listener discussion ## [Wed Mar 19 06:04:15 2014] <kishoreg>: ConfigChangeListeners [Wed Mar 19 06:04:32 2014] <kishoreg>: MessageListeners etc [Wed Mar 19 06:05:03 2014] <kishoreg>: we have 4 model classes [Wed Mar 19 06:05:26 2014] <kishoreg>: Cluster, Resource, Member, Participant [Wed Mar 19 06:05:41 2014] <kishoreg>: Member can be controller, participant, admin, spectator [Wed Mar 19 06:06:04 2014] <kanakb>: what about partition? [Wed Mar 19 06:06:08 2014] <kishoreg>: Each model class has some attributed [Wed Mar 19 06:06:19 2014] <kishoreg>: sorry participant [Wed Mar 19 06:06:44 2014] <osgigeek>: he meant Cluster, Resource, Member and Partition [Wed Mar 19 06:06:53 2014] <kanakb>: yeah got it [Wed Mar 19 06:07:09 2014] <kanakb>: admin is configurable? [Wed Mar 19 06:08:24 2014] <kishoreg>: probably not [Wed Mar 19 06:08:49 2014] <kishoreg>: admin is a role [Wed Mar 19 06:08:55 2014] <kanakb>: right [Wed Mar 19 06:08:56 2014] <kishoreg>: type of member [Wed Mar 19 06:09:11 2014] <kishoreg>: we can potentially have some configuration for types of admin [Wed Mar 19 06:09:21 2014] <kishoreg>: but not mandatory [Wed Mar 19 06:09:42 2014] <kanakb>: ok i'm just trying to think if having a member hierarchy is necessary [Wed Mar 19 06:09:46 2014] <kishoreg>: and each of these model classes will have attributes [Wed Mar 19 06:09:49 2014] <kanakb>: but i guess it buys future flexibility [Wed Mar 19 06:09:54 2014] <kishoreg>: yes [Wed Mar 19 06:10:04 2014] <kishoreg>: i definitely see more types of members [Wed Mar 19 06:10:11 2014] <kishoreg>: like standalone agents [Wed Mar 19 06:10:22 2014] <kishoreg>: but lets not get there for now [Wed Mar 19 06:10:29 2014] <kanakb>: ok back to listeners? [Wed Mar 19 06:10:34 2014] <kishoreg>: yes [Wed Mar 19 06:10:42 2014] <kishoreg>: so we have models and models have attributes [Wed Mar 19 06:10:51 2014] <kishoreg>: right now only attribute thats finalized is State [Wed Mar 19 06:11:47 2014] <kanakb>: other things that can change are helix-defined configs and user-defined configs [Wed Mar 19 06:12:09 2014] <kanakb>: so a question is whether or not those should be separate callbacks [Wed Mar 19 06:12:11 2014] <kanakb>: or the same callback [Wed Mar 19 06:12:49 2014] <kanakb>: and then there's cluster-wide runtime changes like live participant change and controller change [Wed Mar 19 06:15:08 2014] <kanakb>: so then are those runtime things also states of the cluster [Wed Mar 19 06:15:10 2014] <kanakb>: or something else [Wed Mar 19 06:17:47 2014] <osgigeek>: one question, do we need separate listeners like StateChangeListeners, ConfigChangeListeners? [Wed Mar 19 06:18:12 2014] <kanakb>: one could react to a config change to trigger a state change [Wed Mar 19 06:19:02 2014] <osgigeek>: which is ok but can we think of modeling this as separate Events on the same listener ? [Wed Mar 19 06:19:13 2014] <kishoreg>: yes [Wed Mar 19 06:19:17 2014] <osgigeek>: So StateChangeEvent and ConfigChangeEvent [Wed Mar 19 06:19:23 2014] <kishoreg>: we should probably start defining events then [Wed Mar 19 06:19:25 2014] <osgigeek>: one listener which receives HelixEvent [Wed Mar 19 06:19:40 2014] <osgigeek>: correct [Wed Mar 19 06:19:54 2014] <osgigeek>: What events and which model can generate what events [Wed Mar 19 06:20:10 2014] <osgigeek>: *generate which events [Wed Mar 19 06:22:27 2014] <osgigeek>: so far we have StateChange and ConfigChange what does it mean to have runtime changes like live participant change and controller change I imagine they are state change events [Wed Mar 19 06:22:29 2014] <kishoreg>: every model can generate configchangeevent that can be user or helix [Wed Mar 19 06:22:47 2014] <kishoreg>: it can also generate statechange [Wed Mar 19 06:23:03 2014] <kishoreg>: we can also have addition/removal events [Wed Mar 19 06:23:12 2014] <osgigeek>: yes [Wed Mar 19 06:23:17 2014] <kishoreg>: but we can probably model them as NEW [Wed Mar 19 06:24:31 2014] <osgigeek>: Do we need a notion of lifecycle for models? if not all some of them? [Wed Mar 19 06:25:27 2014] <kanakb>: that's what state is for, i think [Wed Mar 19 06:26:22 2014] <osgigeek>: I see [Wed Mar 19 06:26:55 2014] <osgigeek>: yeah that works [Wed Mar 19 06:27:09 2014] <kanakb>: so i have a question about scoping [Wed Mar 19 06:27:20 2014] <kanakb>: listening for addition/removal of ideal states is cluster scoped [Wed Mar 19 06:27:28 2014] <kanakb>: listening for an ideal state change is resource scoped [Wed Mar 19 06:27:35 2014] <kanakb>: how do we reconcile that? [Wed Mar 19 06:29:21 2014] <osgigeek>: So here is an approach [Wed Mar 19 06:29:30 2014] <osgigeek>: The event gives out the model entity [Wed Mar 19 06:29:36 2014] <kishoreg>: yeah i had the same question [Wed Mar 19 06:29:45 2014] <osgigeek>: for the Cluster scoped [Wed Mar 19 06:29:50 2014] <osgigeek>: the model returned is Cluster [Wed Mar 19 06:30:01 2014] <osgigeek>: for the State change we return Resource [Wed Mar 19 06:30:44 2014] <osgigeek>: What I am trying to do is attach the model to the event to indicate at what level/scope it was generated [Wed Mar 19 06:31:05 2014] <kanakb>: but unless cluster contains resources [Wed Mar 19 06:31:12 2014] <kanakb>: returning me the cluster is useless [Wed Mar 19 06:31:16 2014] <kanakb>: at least today [Wed Mar 19 06:31:48 2014] <kishoreg>: it wud be nice to attach listeners to the model [Wed Mar 19 06:32:51 2014] <kanakb>: you mean something like register(model, attribute) [Wed Mar 19 06:33:05 2014] <kanakb>: or model.registerchange(attribute) [Wed Mar 19 06:33:14 2014] <kishoreg>: yeah and pass the listener [Wed Mar 19 06:33:41 2014] <kanakb>: ok but if we do cluster.registerchange(resource_ideal_states) [Wed Mar 19 06:33:48 2014] <kishoreg>: dont want to bloat the models [Wed Mar 19 06:33:48 2014] <kanakb>: then what would we get back in the listener? [Wed Mar 19 06:34:01 2014] <kishoreg>: listeners shud be registered on connection [Wed Mar 19 06:34:19 2014] <kanakb>: ok [Wed Mar 19 06:34:30 2014] <kanakb>: so connection.register(model, attribute, listener)? [Wed Mar 19 06:34:34 2014] <osgigeek>: so that would mean having the user needs to know about what events can be generated at what levels and the onus is on him to register them at the appropriate level [Wed Mar 19 06:34:56 2014] <kishoreg>: yes [Wed Mar 19 06:35:35 2014] <kishoreg>: we can make the method name explicit [Wed Mar 19 06:35:37 2014] <osgigeek>: rather than having him know the model and traverse through the model why not give him the event and let him filter out the level [Wed Mar 19 06:36:00 2014] <kishoreg>: listeners can be expensive [Wed Mar 19 06:36:34 2014] <kishoreg>: for example as a spectator i wants only resourcestate change listener [Wed Mar 19 06:36:35 2014] <osgigeek>: expensive because of the volume of events? [Wed Mar 19 06:37:11 2014] <kishoreg>: i dont want resourceconfig change [Wed Mar 19 06:37:25 2014] <kishoreg>: not only volume, its not neccessary [Wed Mar 19 06:37:32 2014] <osgigeek>: which is fine you can say registerListener(Set<HelixEvent> events, listener) [Wed Mar 19 06:38:00 2014] <osgigeek>: I think the notion should be you are subscribing for 'a' event [Wed Mar 19 06:38:08 2014] <osgigeek>: or if we can 'a set of' events [Wed Mar 19 06:38:35 2014] <osgigeek>: rather it will be Set<T extends HelixEvent> [Wed Mar 19 06:39:08 2014] <kanakb>: so each event is a class? [Wed Mar 19 06:39:13 2014] <kishoreg>: yes [Wed Mar 19 06:39:42 2014] <osgigeek>: yes, we can have a static set of EventTypes which is already available as a catalogue [Wed Mar 19 06:39:43 2014] <kanakb>: if it's Set<T extends HelixEvent> then we're assuming the set only contains events of the same type? [Wed Mar 19 06:40:04 2014] <kishoreg>: all events extends helixevent [Wed Mar 19 06:40:14 2014] <osgigeek>: so you point out correctly kanakb: [Wed Mar 19 06:40:24 2014] <osgigeek>: the registration should be Set<T extends EventType> [Wed Mar 19 06:40:42 2014] <osgigeek>: you receive Events of the Types you subscribed for [Wed Mar 19 06:40:59 2014] <kishoreg>: so there will be only one listener? [Wed Mar 19 06:41:40 2014] <kishoreg>: onEvent(HelixEvent event) [Wed Mar 19 06:41:42 2014] <osgigeek>: I think it would be easier [Wed Mar 19 06:41:46 2014] <osgigeek>: to track the listeners [Wed Mar 19 06:41:51 2014] <osgigeek>: one destination to deliver [Wed Mar 19 06:41:53 2014] <kishoreg>: yeah [Wed Mar 19 06:42:06 2014] <kishoreg>: kanak what do you think [Wed Mar 19 06:42:14 2014] <osgigeek>: from the users point of view its also hard to track if he registers multiple listeners [Wed Mar 19 06:42:16 2014] <kishoreg>: easier to add eventtypes [Wed Mar 19 06:42:20 2014] <kanakb>: swaroop and others have asked for *more* listeners, not fewer [Wed Mar 19 06:42:31 2014] <kanakb>: because it's too easy to miss corner cases checking for event state [Wed Mar 19 06:42:57 2014] <kishoreg>: i think what we are doing is adding more eventtypes [Wed Mar 19 06:43:13 2014] <kishoreg>: listeners will have init callback finalize [Wed Mar 19 06:43:49 2014] <kanakb>: it's still somewhat of a tradeoff between number of listeners and number of branches within the listener [Wed Mar 19 06:43:50 2014] <osgigeek>: yes that is what I think will be simpler, dont know about the ask for more listenersâ¦maybe I need to understand their requirement [Wed Mar 19 06:44:24 2014] <kishoreg>: swaroop requirement was to know the type of notication init, callback, finalize [Wed Mar 19 06:44:25 2014] <kanakb>: personally i don't have a strong opinion either way [Wed Mar 19 06:44:39 2014] <kishoreg>: have separate methods for each call [Wed Mar 19 06:46:12 2014] <osgigeek>: if you think about the API registration of listeners they could potentially register one per event type unless we figure a way to restrict the registration by client [Wed Mar 19 06:46:34 2014] <osgigeek>: but it gives them the ability to write one which can handle all events [Wed Mar 19 06:46:48 2014] <osgigeek>: rather than force them to write multiple listeners [Wed Mar 19 06:47:06 2014] <kanakb>: i think it's easier to scale the code if we have a single listener [Wed Mar 19 06:47:12 2014] <osgigeek>: I agree [Wed Mar 19 06:47:18 2014] <kanakb>: i just need to see what that would look like when actually using it [Wed Mar 19 06:47:19 2014] <kishoreg>: yes, the biggest benefit is we can add events [Wed Mar 19 06:47:25 2014] <kishoreg>: without breaking anything [Wed Mar 19 06:47:31 2014] <osgigeek>: yes [Wed Mar 19 06:47:32 2014] <kishoreg>: ok 3 +1;s [Wed Mar 19 06:47:58 2014] <kishoreg>: we will define the eventtypes as needed [Wed Mar 19 06:48:28 2014] <kishoreg>: for now we will have model * attributes [Wed Mar 19 06:48:38 2014] <kishoreg>: events [Wed Mar 19 06:48:52 2014] <osgigeek>: If we build a hierarchy properly users can even just get new events added under the root event [Wed Mar 19 06:49:07 2014] <osgigeek>: no code change needed [Wed Mar 19 06:49:29 2014] <osgigeek>: they can chose to act on it by adding logic [Wed Mar 19 06:50:53 2014] <kanakb>: right [Wed Mar 19 06:52:36 2014] <kishoreg>: example? [Wed Mar 19 06:52:43 2014] <osgigeek>: So for example [Wed Mar 19 06:52:50 2014] <osgigeek>: if you have base event as StateChangeEvent [Wed Mar 19 06:53:07 2014] <osgigeek>: and hang off all StateChangeEvents from under it like ParticipantStateChange, ControllerStateChange [Wed Mar 19 06:53:12 2014] <osgigeek>: then if you add another role tomorrow [Wed Mar 19 06:53:19 2014] <osgigeek>: and want to push state change event [Wed Mar 19 06:53:32 2014] <osgigeek>: the subscriber need only susbcribe to StateChangeEvent [Wed Mar 19 06:53:39 2014] <osgigeek>: and he receives the new event [Wed Mar 19 06:53:48 2014] <osgigeek>: he is not handling it but he receives it [Wed Mar 19 06:53:53 2014] <kishoreg>: i prefer flat [Wed Mar 19 06:53:54 2014] <osgigeek>: we dont need to push a new API call [Wed Mar 19 06:54:25 2014] <kanakb>: generally speaking everyone but the controller only cares about a small number of events [Wed Mar 19 06:54:30 2014] <kishoreg>: hard to come up with heirarchy [Wed Mar 19 06:54:32 2014] <kishoreg>: yes [Wed Mar 19 06:54:32 2014] <osgigeek>: well flat does not get us the benefits of the event abstraction [Wed Mar 19 06:54:39 2014] <kanakb>: and we don't really want people writing their own controllers if possible [Wed Mar 19 06:55:09 2014] <kanakb>: it gets us one benefit: single callback [Wed Mar 19 06:55:22 2014] <kanakb>: and base class for all events [Wed Mar 19 06:55:24 2014] <osgigeek>: let me rephrase that it does not give us the entire benefit of decoupling the subscriber from the changes in events [Wed Mar 19 06:55:41 2014] <osgigeek>: by changes I mean adding events [Wed Mar 19 06:55:49 2014] <kishoreg>: the problem is heirarchy [Wed Mar 19 06:56:11 2014] <kishoreg>: one way to look at heirarchy is model -> type [Wed Mar 19 06:56:18 2014] <kishoreg>: other way is type -> model [Wed Mar 19 06:56:26 2014] <osgigeek>: what is type? [Wed Mar 19 06:56:34 2014] <kishoreg>: attribute type [Wed Mar 19 06:56:46 2014] <kishoreg>: for example statechangeevent [Wed Mar 19 06:57:05 2014] <kishoreg>: the one you explained is type -> model [Wed Mar 19 06:57:24 2014] <kishoreg>: u subscribe for a type of change and you get events across all models [Wed Mar 19 06:57:46 2014] <osgigeek>: remember you can subscribe for specific model level events [Wed Mar 19 06:57:51 2014] <osgigeek>: you dont have to subscribe for all [Wed Mar 19 06:57:58 2014] <kishoreg>: right [Wed Mar 19 06:58:06 2014] <osgigeek>: you can subscribe for only ParticipantConfigChangeEvent [Wed Mar 19 06:58:11 2014] <kishoreg>: correct [Wed Mar 19 06:58:15 2014] <osgigeek>: if that is all you care about [Wed Mar 19 06:58:33 2014] <kishoreg>: but another way to look at it is [Wed Mar 19 06:58:39 2014] <kishoreg>: subcribe for ResourceChange [Wed Mar 19 06:58:54 2014] <kishoreg>: so anything changes for that Resource I want to know [Wed Mar 19 06:59:02 2014] <kishoreg>: it can be config, state etc [Wed Mar 19 06:59:39 2014] <kishoreg>: what i am trying to say is the heirarchy can be either completely useful or completely useless [Wed Mar 19 07:00:02 2014] <kishoreg>: in the latter, user will have to subscribe to individual event types [Wed Mar 19 07:00:22 2014] <kishoreg>: which is equivalent of flat structure [Wed Mar 19 07:00:51 2014] <osgigeek>: to me (and its just me maybe) its more intutive to I am interested in config change events for this model or maybe all models [Wed Mar 19 07:00:59 2014] <osgigeek>: *to say [Wed Mar 19 07:02:11 2014] <osgigeek>: but maybe the usage you have seen is more model based [Wed Mar 19 07:02:29 2014] <osgigeek>: lets do what is most common use-cases [Wed Mar 19 07:02:37 2014] <osgigeek>: i.e. most commonly sought out today [Wed Mar 19 07:02:43 2014] <osgigeek>: we can take that approach [Wed Mar 19 07:03:04 2014] <osgigeek>: if people are doing give me all events at model lets do that [Wed Mar 19 07:03:18 2014] <kanakb>: depends on what you're trying to accomplish [Wed Mar 19 07:03:32 2014] <kanakb>: it's totally possible to deploy helix and listen for nothing except state transition callbacks [Wed Mar 19 07:03:40 2014] <kanakb>: at least to start [Wed Mar 19 07:03:51 2014] <kishoreg>: correct [Wed Mar 19 07:04:35 2014] <kanakb>: then there are use cases where participants listen on their own configs [Wed Mar 19 07:04:36 2014] <osgigeek>: so do we want to keep the events as a hierarchy and add the ability to register filters which are model entities? [Wed Mar 19 07:05:00 2014] <kanakb>: i don't follow [Wed Mar 19 07:05:02 2014] <kishoreg>: nope [Wed Mar 19 07:05:24 2014] <kishoreg>: i am suggesting no heirarchy [Wed Mar 19 07:05:49 2014] <kishoreg>: we can keep inner classes to group them [Wed Mar 19 07:06:06 2014] <osgigeek>: so they go registerListener(ConfigChangeEvent, listener, Filter(Partition)) [Wed Mar 19 07:06:42 2014] <osgigeek>: a hypothetical example but the way I explian this is I want config change event delivered to this listener and give me only for Partition [Wed Mar 19 07:06:44 2014] <kishoreg>: i was thinking more of registerListner(PartitionConfigChangeEvent, listner) [Wed Mar 19 07:07:14 2014] <kishoreg>: or PartitionEvent.ConfigChange [Wed Mar 19 07:07:47 2014] <osgigeek>: ok well you said no hierarchy [Wed Mar 19 07:07:49 2014] <kishoreg>: PartitionEvent.StateChange [Wed Mar 19 07:08:08 2014] <kishoreg>: its inner class [Wed Mar 19 07:08:21 2014] <osgigeek>: if we want that might as well flatten it out [Wed Mar 19 07:08:27 2014] <osgigeek>: PartitionStateChangeEvent [Wed Mar 19 07:08:30 2014] <kishoreg>: we cant say registerListener(partitionevent, listener) [Wed Mar 19 07:08:32 2014] <osgigeek>: why have the dot notation [Wed Mar 19 07:08:43 2014] <osgigeek>: what does the dot notation buy us? [Wed Mar 19 07:08:48 2014] <kanakb>: reduces number of files [Wed Mar 19 07:08:51 2014] <kishoreg>: just to group them [Wed Mar 19 07:09:00 2014] <osgigeek>: you still have the same number at runtime [Wed Mar 19 07:09:28 2014] <kishoreg>: easier to know the event types per model [Wed Mar 19 07:09:32 2014] <kishoreg>: while coding [Wed Mar 19 07:10:07 2014] <osgigeek>: ok I buy that [Wed Mar 19 07:10:32 2014] <osgigeek>: but you could do that with EventTypes [Wed Mar 19 07:11:03 2014] <osgigeek>: The way I see it is Event is a concrete realization of what happened [Wed Mar 19 07:11:12 2014] <osgigeek>: its not what the subscriber uses to subscribe [Wed Mar 19 07:11:28 2014] <osgigeek>: because subscriber does not have an event, the publisher generates one and it gets delivered to the subscriber [Wed Mar 19 07:11:29 2014] <kanakb>: PartitionEventTypes.ConfigChange [Wed Mar 19 07:11:31 2014] <kanakb>: ? [Wed Mar 19 07:11:33 2014] <osgigeek>: yea [Wed Mar 19 07:11:47 2014] <kishoreg>: yep [Wed Mar 19 07:11:58 2014] <osgigeek>: So I think event is still flat then [Wed Mar 19 07:12:06 2014] <osgigeek>: everything derives from HelixEvent [Wed Mar 19 07:12:13 2014] <kishoreg>: correct [Wed Mar 19 07:12:15 2014] <osgigeek>: and you have PartitionStateChangeEvent [Wed Mar 19 07:12:55 2014] <osgigeek>: or in kanakb: example PartitionConfigChangeEvent [Wed Mar 19 07:13:10 2014] <kishoreg>: correct [Wed Mar 19 07:13:32 2014] <osgigeek>: ok sounds good enough [Wed Mar 19 07:13:36 2014] <kishoreg>: there is implicit hierarchy at some levels [Wed Mar 19 07:14:11 2014] <kishoreg>: sorry what i mean is [Wed Mar 19 07:14:40 2014] <kishoreg>: we need partitionchange for all resources and all partitions [Wed Mar 19 07:15:07 2014] <kishoreg>: so event will have a constructor that takes resourceid and partitionid as optional [Wed Mar 19 07:15:11 2014] <kishoreg>: arguments [Wed Mar 19 07:15:35 2014] <kishoreg>: some kind of filter based on values [Wed Mar 19 07:15:48 2014] <kishoreg>: we can probably take Filter as input ? [Wed Mar 19 07:16:23 2014] <kishoreg>: let me think about it and will get some inputs [Wed Mar 19 07:16:28 2014] <kanakb>: is filter doing anything more than Scope? [Wed Mar 19 07:16:44 2014] <kishoreg>: some times it can be more than scope [Wed Mar 19 07:16:58 2014] <osgigeek>: you could extend it to say give me the event only if this property changed [Wed Mar 19 07:17:05 2014] <kishoreg>: for example listen to resources with a specific statemodel [Wed Mar 19 07:17:16 2014] <kishoreg>: yeah [Wed Mar 19 07:17:24 2014] <kishoreg>: something like that [Wed Mar 19 07:17:31 2014] <kishoreg>: its mainly for performance [Wed Mar 19 07:17:32 2014] <osgigeek>: yes, like give me state transitions where state is only 'xyz' state [Wed Mar 19 07:17:43 2014] <osgigeek>: *toState [Wed Mar 19 07:17:59 2014] <kishoreg>: yes [Wed Mar 19 07:18:23 2014] <kishoreg>: sometimes its for performance but it can be convenient as well [Wed Mar 19 07:18:47 2014] <kishoreg>: ok [Wed Mar 19 07:18:54 2014] <kishoreg>: i think i got enough info [Wed Mar 19 07:19:09 2014] <osgigeek>: awesome, I think I need to start winding down anyway [Wed Mar 19 07:19:16 2014] <kanakb>: alright [Wed Mar 19 07:19:42 2014] <kanakb>: ASFBot: meeting end Meeting ended at Wed Mar 19 07:19:42 2014
