Thanks for the summary Kishore!
One note about the new design of emitters: before, emitters used be notified of updates to the cluster, but this is not the case anymore: the senders are the ones receiving these updates, so they can decide where to send messages. One implication is that the mapping destination/node in the emitter is not updated automatically. Illustration: Suppose a mapping destination1->node1. Node 1 fails. Node 2 starts and replaces node1. Suppose we have a new mapping destination 1 -> node2. In the emitter, in the absence of notifications, this mapping has not been updated yet. This happens with the first message to destination 1 - which does not reach destination, but clears the old mapping. Next message for destination 1 will setup the new channels to node 2. Once we get the integration complete, we can improve that with something like keeping the mappings of the emitters in a cache with a TTL or something like that. The other pending tasks will be porting existing examples (the echo and the twitter examples) to illustrate the new perspectives offered by this integration. Regards, Matthieu On Feb 11, 2013, at 11:07 , kishore g wrote: > Hi, > > This is an update on S4 and Helix integration happening on S4-110-new > branch. Last week Matthieu and I made good progress. We have the initial > set of changes that allow us to configure the cluster management mode ( > existing zk based or helix based). We had to do some changes to the > existing code to facilitate this. > > Significant changes was that Emitters no longer hold the mapping of > partition to node. The logic of deciding the destination for an event is > moved to Sender. This allowed us to have clear separation > of responsibility between Sender and Emitter. > > In order to add new helix implementations, we also made changes in some of > the bindings like S4Bootstrap, DeploymentManager, cluster, Emittter etc. > > We still have the following pending > > > - Tools are broken. they currently support only Helix based mode. We > need to support additional configuration that indicates the mode(zk or > helix). > - Need to add more test cases/documentation for helix related changes. > - Fix broken test cases for s4 mode. > - Other minor code clean up/refactoring. > > Matthieu, please add items to this list( I am sure I have missed quite a > few) :-). The code is not yet ready for review, but feel free to comment on > design and code structure. > > Thanks, > Kishore G
