On 11/11/11 6:11 AM, Karthik Kambatla wrote:
Hi
In last Tuesday's meeting, we agreed on decoupling partitions and nodes.
How should we go about it?
1. io.s4.comm.topology.ClusterNode currently has a field called
partition - it should be changed to a set of partitions.
2. Emitters need to maintain hashtables<partition, clusterNode>; send
will pick the right node from this table.
3. Listeners need to receive messages for each partition that they are
hosting.
Comments?
Thanks
Karthik
PS: The NettyEmitter code that I am trying to fix depends on how we decide
to resolve this.
Hi Karthik,
My understanding of ClusterNode is that it corresponds to a task that
was picked by a process: it maps a logical partition to a physical node.
The ensemble of ClusterNodes is reflected in the Topology, and updated
automatically upon any change (at least, for the Zookeeper version).
Therefore I wouldn't do anything about point 1.
I agree though that the documentation doesn't say much.
I added a few improvements to the comm layer in my latest commit
yesterday, that enables integration with Zookeeper. Maybe you could have
a look at that and see what else would be missing? Points 2 and 3 are
already implemented, as I see it.
Thanks,
Matthieu
