On Fri, Nov 20, 2009 at 2:35 PM, Steven Dake <[email protected]> wrote: >> The definition of PROCESSOR_COUNT_MAX is 384 in totem.h, >> and this value looks changeable according to the manpage of >> corosync.conf (in the explanation of vsftype parameter). >> What does this value depend on? > > The value depends on a few things like avoiding overflow of the input > and output ethernet buffers during the membership protocol operation. > Also latency to delivery is 1/2 the token rotation time. Each token > posession takes approximately 300 usec so with hundreds of nodes, the > latency might be somewhat high (latency scales linearly with node > count). > > We have tested 16 nodes very heavily and find larger node configurations > to be somewhat unstable. I have some minor changes I am working on that > allow us to scale to at least 48 nodes reliably (the largest physical > cluster I have access to). > > After I finish those changes I'm going to attempt to create a larger > virtual machine cluster and see if corosync can scale further then 48 > nodes and try to find the actual limits for forming a configuration with > the new changes. For the moment the upper bound is somewhere around 16 > nodes. > > Longer term, we may do multiring (totemmrp) work which allows further > scaling to thousands of nodes, although I'm not sure how much sense it > makes to run a fully replicated state machine over thousands of nodes. > Your input would be welcome here. We may focus on other features like > better diags and management if we can scale the single ring protocol > (totemsrp) to meet most people's needs.
I'm looking forward to a patch to stablize corosync in the large cluster environment. If I do not care about the latency and the amount of messages is small, corosync may be scale to several hundreds nodes without multiring, yes? >> What about CPG_MEMBERS_MAX (= 128) in cpg.h? > > There are upper limits that are very large dependent on the ipc layer > and totem fragmentation layer (1mb per message, which may be changed at > compile time). I'm not sure you would hit these limits with even > CPG_MEMBERS_MAX set to 10000. There is no measured upper bound on > CPG_MEMBERS_MAX, but the algorithms that execute on the data structures > are in some cases O(n^2) or O(n) so performance at large CPG_MEMBERS > count may be poor. Okay, I understood. > What is your use case and environment your trying to target? I'm planning to use corosync in my project, a distributed storage for KVM. http://www.osrg.net/sheepdog/ The target of the storage scalability is at most one thousand nodes. I'll use cpg library to implement lock service of VM images and to monitor which nodes are alive. I think the traffic of the lock service is small because it is used only when VM starts, new nodes are joined, existing nodes are left, and snapshots of VM images are taken. Is corosync suitable for this use? Thanks, MORITA Kazutaka _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
