proton engine performance: two strong credit management effects
conclusion = Using the proton engine interface (in C) I am seeing two aspects of credit management that can strongly affect throughput. The first is credit stall. If you frequently allow the credit available to the sender to drop to zero, so that he has to cool his heels waiting for more, that can have a very strong effect even in my simple test case, in which the receiver is granting new credit as quickly as possible, and is serving only 1 sender. The second effect is credit replenishment amortization. It looks like the granting of new credit is kind of an expensive operation. If you do it too frequently, that will also have a noticeable effect on throughput. These tests use the C clients, written against the engine/driver level, that I recently put at https://github.com/mick-goulish/proton_c_clients.git test setup = * single sender, single receiver, on laptop. * sender only sends, receiver only receives. * one link * sender locked onto CPU core 1, receiver locked onto CPU core 2 * system is otherwise quiet - only OS and XFCE running. no browser, no internet. * sender sends as fast as possible, receiver receives as fast as possible. * each test consists of 5,000,000 messages, about 50 bytes of payload each. * each test is repeated 3 times, and the results averaged to make the number that is graphed. stall test result = scenario: start out with 200 credits. Every time we get down to X, add 100 to credit level. X axis: point at which credit gets refilled. Y axis: messages received per second. Note: When we let credit go to 10 or less before replenishment, throughput falls off a cliff. amortization test result = scenario: start out with 200 credits. Every time we get down to 200-X, add X to credit level. X axis: credit increment Y axis: messages received per second. Note: In this case, if X has a value of 5, that means we add 5 new units of credit every time we see that it has fallen by 5. The smaller the X value, the more frequently we replenish credit. If we replenish too frequently, throughput is affected.
Re: proton engine performance: two strong credit management effects
On Wed, 2014-08-27 at 13:39 -0400, Michael Goulish wrote: conclusion = Using the proton engine interface (in C) I am seeing two aspects of credit management that can strongly affect throughput. The first is credit stall. If you frequently allow the credit available to the sender to drop to zero, so that he has to cool his heels waiting for more, that can have a very strong effect even in my simple test case, in which the receiver is granting new credit as quickly as possible, and is serving only 1 sender. The second effect is credit replenishment amortization. It looks like the granting of new credit is kind of an expensive operation. If you do it too frequently, that will also have a noticeable effect on throughput. So my conclusion looking at the graphs for these tests would be that you will get the vast majority of benefit by using an initial credit of 40 and replenishing with 20 at 20. Of this only applies under the exact circumstances of these tests, until we have more data points. I'd suspect it must also depend on the message size too. Andrew
Re: proton engine performance: two strong credit management effects
On Wed, 2014-08-27 at 14:55 -0400, Andrew Stitcher wrote: ... Of this only applies under the exact circumstances of these tests, until we have more data points. I'd suspect it must also depend on the message size too. Just to reply to myself... Keeping the credit window at around the MTU size is going to give you the most efficient networking transfers. if looks like we get maximum throughput with at least 20 messages in flight at once. That is something over 1KiB and will pack them into few transfer units. For regular ethernet the MTU is 1500, and for loopback on Linux it is 64KiB. So I'd guess that when the message is big enough it makes less difference. And it looks like once you can hit around 1KiB of content for each packet then you get small performance gain after that. Andrew
Re: proton engine performance: two strong credit management effects
On 08/27/2014 02:55 PM, Andrew Stitcher wrote: On Wed, 2014-08-27 at 13:39 -0400, Michael Goulish wrote: conclusion = Using the proton engine interface (in C) I am seeing two aspects of credit management that can strongly affect throughput. The first is credit stall. If you frequently allow the credit available to the sender to drop to zero, so that he has to cool his heels waiting for more, that can have a very strong effect even in my simple test case, in which the receiver is granting new credit as quickly as possible, and is serving only 1 sender. The second effect is credit replenishment amortization. It looks like the granting of new credit is kind of an expensive operation. If you do it too frequently, that will also have a noticeable effect on throughput. So my conclusion looking at the graphs for these tests would be that you will get the vast majority of benefit by using an initial credit of 40 and replenishing with 20 at 20. Funny, those are the exact numbers I chose for the default in Dispatch (based on Mick's conclusion). I think the 'amortization' effect will be fairly independent of local conditions whereas the outstanding credit value will be primarily tied to network latency (i.e. you will want a larger window for networks with higher latency). Of this only applies under the exact circumstances of these tests, until we have more data points. I'd suspect it must also depend on the message size too. Andrew
