On Fri, 2009-11-20 at 14:25 +0900, MORITA Kazutaka wrote: > I have questions about corosync scalability. > > The definition of PROCESSOR_COUNT_MAX is 384 in totem.h, > and this value looks changeable according to the manpage of > corosync.conf (in the explanation of vsftype parameter). > What does this value depend on?
The value depends on a few things like avoiding overflow of the input and output ethernet buffers during the membership protocol operation. Also latency to delivery is 1/2 the token rotation time. Each token posession takes approximately 300 usec so with hundreds of nodes, the latency might be somewhat high (latency scales linearly with node count). We have tested 16 nodes very heavily and find larger node configurations to be somewhat unstable. I have some minor changes I am working on that allow us to scale to at least 48 nodes reliably (the largest physical cluster I have access to). After I finish those changes I'm going to attempt to create a larger virtual machine cluster and see if corosync can scale further then 48 nodes and try to find the actual limits for forming a configuration with the new changes. For the moment the upper bound is somewhere around 16 nodes. Longer term, we may do multiring (totemmrp) work which allows further scaling to thousands of nodes, although I'm not sure how much sense it makes to run a fully replicated state machine over thousands of nodes. Your input would be welcome here. We may focus on other features like better diags and management if we can scale the single ring protocol (totemsrp) to meet most people's needs. > Can I define PROCESSOR_COUNT_MAX to be more than a thousand > if I configure corosync.conf properly? > > What about CPG_MEMBERS_MAX (= 128) in cpg.h? > I changed the value of CPG_MEMBERS_MAX to 1024 and tested cpg library. > It seemed to work well on 32 machines (each machine has 32 processes). > Can I set more than 128 to CPG_MEMBERS_MAX? > > What will happen if I set too large number to these definition? > There are upper limits that are very large dependent on the ipc layer and totem fragmentation layer (1mb per message, which may be changed at compile time). I'm not sure you would hit these limits with even CPG_MEMBERS_MAX set to 10000. There is no measured upper bound on CPG_MEMBERS_MAX, but the algorithms that execute on the data structures are in some cases O(n^2) or O(n) so performance at large CPG_MEMBERS count may be poor. What is your use case and environment your trying to target? Regards -steve > Thanks, > > MORITA Kazutaka > > _______________________________________________ > Openais mailing list > [email protected] > https://lists.linux-foundation.org/mailman/listinfo/openais _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
