I just want to mention this to the list in case anyone has an immediate brilliant idea.
Something spooky is happening. I have a 2-process test, one sender, one receiver, written at the proton engine level. This is what I've been using for my nightly performance measurements lately, which are here: http://people.apache.org/~mgoulish/protonics/performance/results/nightly.svg Recently changed this test to be perpetual, and the receiver now only reports the timing for the most recent 5 million messages. So ... the weird result is that, over many messages, the test is slowing down ... and it is doing this without the RSS memory of either process growing! Arg! The virtual mem of the sender increased, but only slightly. It might very well have fallen again if I let the test keep running. ( That happened in the receiver. ) The effect is very gradual, but after 500 million messages it is taking about 50% more time to get each batch of 5 million messages received !! And it looks like the effect is accelerating. Also -- the receiver CPU usage is slowly going down. CPU usage on the sender is constant. For the receiver, CPU usage started out around 77% at the beginning of the test, and after 500 million msgs has fallen to 64% or so. My plan is to use callgrind to get a snapshot of sender behavior (I suspect sender is culprit) at the beginning of a run, and a separate snapshot later after the slowdown has started. But i wanted to just mention it here, just in case anybody has a Great Idea, or in case I get hit by a truck or something.
