On 10/26/2010 08:33 AM, dan clark wrote: > Hi Steve, > Thank you for your insight. I am not sure that the degradation measured > is linear in nature based on the buckets measured by the test application. > > How would you characterize the nature of the system under moderate load > under the default settings? Would you recommend an alternative to the > previously submitted test application? >
One thing that you will notice is smaller messages have less peformance. It may depend on your message size. Totem packs each message into a MTU frame as possible, but it may not be possible to pack "max_messages" during one rotation. Another factor > Would modifying any of the configuration options from the defaults have > an impact on the performance characterization under a specified load? > For example: > For batches of messages > 50, should window_size be dialed up? The purpose of window size is to limit the total bandwidth consumed on the network to a avoid totem from consuming the network under heavy load. If that is not a concern, this value could be increased. > Why would max_messages be set to 17? Is this tuned for some particular > number of nodes or a particular set of applications running over corosync? The maximum "max_messages" can be is the transmit multicast socket send buffer size. On Linux, we set this as follows: Oct 26 08:52:03 corosync [TOTEM ] Transmit multicast socket send buffer size (262142 bytes). At one time, the value allowed by kernel.org was different, and it was tuned to the maximum mtu we support (9000) which worked out at the time to 17 messages. The key is to take 262142/your mtu and that sets you max_messages size. Increasing this or window_size could increase latency if your hitting the limits on these flow control mechanisms. The number one thing you can do to increase throughput is increase mtu to 9000. Other then that, there isn't much way to increase throughput, but latecny could be reduced by tuning down max_messages and window_size. > Is there a way to modify the aforementioned 'time to send messages > during one rotation'? Is there some tuning value for the rotation time? No this is a factor of your hardware and network. > Would this be processor, node count related? > One thing that I suppose would be interesting is to block origination of new messages if the token rotation exceeds a certain time. This would allow for example, setting a max token rotation time. After this time was reached, new messages would not be originated in that rotation which may reduce latency in small bursty messaging. Can you provide more info about your performance goals? Interested in throughput or latency? Regards -steve > Thanks for your insights! > dan > > > On Fri, Oct 22, 2010 at 2:28 PM, Steven Dake <[email protected] > <mailto:[email protected]>> wrote: > > On 10/22/2010 02:05 PM, dan clark wrote: > > Hi Gentle Readers! > > Is round trip latency expected to degrade linearly as more > receivers are > added to a system given an equivalent number of messages sent? > > > Yes, latency from origination to delivery for totem is 1/2 token > rotation time on average. Token rotation time is mainly dependent > on number of nodes and partially dependent on the time needed to use > the udp services of the kernel to send messages in a token rotation. > These can be generalized into a time of > > latency = 1/2 * node count * token processing time per node + time > to send all messages during one rotation. > > Not exact, but pretty close if calculating for RT deadline purposes. > > Regards > -steve > > The attached test application attempts to capture the situation. > The > basic setup is to start N receivers on 1-N different nodes, then > run a > single sender which tracks the results in a log file on the system > distributing delays across multiple buckets. Note when changing the > number of receivers to result in a similar number of messages it is > necessary to modify the message count of the sender. > > For example: > Starting receivers with a simple script: > #!/bin/sh > cnt=1 > maxcnt=16 > host=`hostname -s` > echo $host > while (($cnt <= $maxcnt)) > do > corestart -s -c 1 -i 1 -j 2 -p 30000000 > ((cnt++)) > done > > And then start a single sender: > corestart -c 100 -i 1 -j 2 -p 1000000 > > By varying the number of receivers (and the associated message > count) > the round trip latency degrades with the number of receivers in a > non-linear relationship. > > How are your results for 1-8 node systems? > > Any insights as to what may cause this anomaly, including a poorly > written test application :)! > thanks, > dan > ps. there are several other parameters that allow this test > application > to exercise some interesting aspects of the library., such as > running > the number if iterations up (-i) to test initialize/finalize logic. > > > > _______________________________________________ > Openais mailing list > [email protected] > <mailto:[email protected]> > https://lists.linux-foundation.org/mailman/listinfo/openais > > > _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
