[
https://issues.apache.org/jira/browse/S4-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584578#comment-13584578
]
Matthieu Morel commented on S4-110:
-----------------------------------
A quick update on the status of this feature.
We've been doing several iterations with Kishore and Daniel, we had to refactor
the code in a few places, including the emitter, and we are now close to a
complete integration.
The code is available in branch S4-110-new.
This will provide means to have finely grained partition, be able to add
resources as needed and rebalance partitions with minimal disruption (i.e.
minimal number of partition migrations).
We can define the partitioning globally - as in the existing S4 cluster
management - or per stream.
There is still some work left but I would say we are almost there!
Regarding Aimee's comment, here is how I see it through an example. Streams S1
and S2 are partitioned differently. If PE1 subscribes to S1 and S2, but S1 and
S2 are mapped to different nodes, PE1 instances won't be able to receive events
from both S1 and S2 and do some kind of join operation. As far as I understand,
currently this has to be considered in the design of the partitioning.
> Enhance cluster management features in S4
> -----------------------------------------
>
> Key: S4-110
> URL: https://issues.apache.org/jira/browse/S4-110
> Project: Apache S4
> Issue Type: New Feature
> Reporter: kishore gopalakrishna
> Assignee: kishore gopalakrishna
> Fix For: 0.6
>
>
> In S4 the number of partition is fixed for all streams and is dependent on
> the number of nodes in the cluster. Adding new nodes to S4 cluster causes
> the number of partitions to change. This results in lot of data movement. For
> example if there are 4 nodes and you add another node then nearly all keys
> will be remapped which result is huge data movement where as ideally only 20%
> of the data should move.
> By using Helix, every stream can be partitioned differently and independent
> of the number of nodes. Helix distributes the partitions evenly among the
> nodes. When new nodes are added, partitions can be migrated to new nodes
> without changing the number of partitions and minimizes the data movement.
>
> In S4 handles failures by having stand by nodes that are idle most of the
> time and become active when a node fails. Even though this works, its not
> ideal in terms of efficient hardware usage since the stand by nodes are idle
> most of the time. This also increases the fail over time since the PE state
> has to be transfered to only one node.
> Helix allows S4 to have Active and Standby nodes at a partition level so that
> all nodes can be active but some partitions will be Active and some in stand
> by mode. When a node fails, the partitions that were Active on that node
> will be evenly distributed among the remaining nodes. This provides automatic
> load balancing and also improves fail over time, since PE state can be
> transfered to multiple nodes in parallel.
> I have a prototype implementation here
> https://github.com/kishoreg/incubator-s4
> Instructions to build it and try it out are in the Readme.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
