[
https://issues.apache.org/jira/browse/S4-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13760936#comment-13760936
]
kishore gopalakrishna commented on S4-141:
------------------------------------------
Great write up Kurtt. Definitely the right direction
Agree with Matthieu that we should first get one version relying on distributed
storage like NFS, HDFS, S3, Hbase etc.
However each node having local storage is definitely more appealing. The only
problem would be handling failure which will then require replicating the
state. We can definitely do this by storing the checkpoint data on more than
one node. For example we can have a node in standby state in helix and the
leader node when it checkpoints, sends the checkpoint data over to in standby
node as well. This will allow us to handle failures as well.
For elasticity, we can do something like you have outlined but with ability to
save the checkpoint data on multiple nodes we can handle failures even during
elasticity.
You are right that Helix does not have any rebalancer that moves partitions
around based on health/performance metrics. For small clusters writing health
metrics to zookeeper per partition works but for larger cluster/partitions zk
wont scale.But the rebalancer can be agnostic of where it gets the metrics.
We have a contribution(patch) that uses google-or tools to periodically
recompute the mapping but we should probably do this in next phase.
S4-110 had to clean up some of the existing code in S4. We can start again from
trunk. The main piece of code in S4 that needed change was emitter and
listener. Emitter and listener kind of own the partition routing but ideally it
should be the Sender and Receiver that should keep track of topology changes.
As Flavio suggested, we can break it up into multiple subtasks. I can take a
stab at it.
Excited to see this happening!!!
> Enhance S4 with the elasticity feature
> --------------------------------------
>
> Key: S4-141
> URL: https://issues.apache.org/jira/browse/S4-141
> Project: Apache S4
> Issue Type: New Feature
> Reporter: Kurtt.Lin
> Attachments: S4-141.pdf
>
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Attached is a design documentation draft, which aims at enhancing S4 with the
> elasticity feature.
> Issues discussed are:
> * introduction to elasticity
> * current status with S4
> * possible roadmap
> * the design
> * implementation with Helix
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
