[
https://issues.apache.org/jira/browse/S4-141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13760318#comment-13760318
]
Matthieu Morel commented on S4-141:
-----------------------------------
That looks quite interesting thanks for formalizing the proposal!
S4-110 focuses on elasticity when adding or removing nodes. It does not
consider automatic load balancing, although being able to repartition
efficiently can be seen as a first step.
in S4-110 Helix is used to distribute and re-distribute PEs across nodes. We
actually gather keys into partitions (buckets), with nb partitions >> nb nodes
(You also mention this approach in your document). Helix computes a routing
table (partition->node mapping). When nodes are added or removed, Helix
efficiently updates the routing table in order to minimize reallocations
Regarding the transfer of state and messages:
- the simplest approach is to rely on checkpointing for moving PEs. We may lose
some state or some messages, but in many cases that's quite acceptable.
- if more guarantees are required then something like what you propose could be
envisaged, i.e. proactively migrating state and buffering messages. But it's a
lot more complex to implement.
Regarding the status of S4-110:
this is not fully complete. Mostly because the approach requires a lot of
refactoring - ideally there should be a less intrusive way to integrate Helix -
and because the 0.6 codebase evolved in parallel with many refactorings as
well. But it's worth having a look.
> Enhance S4 with the elasticity feature
> --------------------------------------
>
> Key: S4-141
> URL: https://issues.apache.org/jira/browse/S4-141
> Project: Apache S4
> Issue Type: New Feature
> Reporter: Kurtt.Lin
> Attachments: S4-141.pdf
>
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Attached is a design documentation draft, which aims at enhancing S4 with the
> elasticity feature.
> Issues discussed are:
> * introduction to elasticity
> * current status with S4
> * possible roadmap
> * the design
> * implementation with Helix
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
