Hi Ken, The maximum we have seen in few thousands of nodes in one cluster and around 30-40k partitions. I don't think we have tried 10 thousand nodes in production. Here are few things you might want to deal with the scale.
1. Need 5 or 7 zookeepers, each participant of Helix creates one connection to Zookeeper. Assuming even distribution this means 2k connections per zookeeper server. 2. For large number of partitions you can use the new compression support added in 0.6.5 3. Configure the resource to use group message feature, this will allow Helix to automatically group multiple state transitions messages into 1 There are more optimizations possible but that depends on the exact use case. thanks, Kishore G On Mon, Apr 13, 2015 at 7:06 AM, 梁成辉 <[email protected]> wrote: > Dear all, > > > I'm working on a project which support up to 10 thousand physical nodes(or > 100 thousand partitions). > All of this nodes will became the Participant of Helix. > Can Helix manage such a big cluster? > > How to use Helix to deal with it? > > > Best regards, > > Ken > > > >
