I just want to mention this to the list in case
anyone has an immediate brilliant idea.
Something spooky is happening.
I have a 2-process test, one sender, one receiver,
written at the proton engine level.
This is what I've been using for my nightly performance
measurements lately, which are here:
Recently changed this test to be perpetual, and the
receiver now only reports the timing for the most
recent 5 million messages.
So ... the weird result is that, over many messages,
the test is slowing down ... and it is doing this
without the RSS memory of either process growing! Arg!
The virtual mem of the sender increased, but only slightly.
It might very well have fallen again if I let the test
keep running. ( That happened in the receiver. )
The effect is very gradual, but after 500 million
messages it is taking about 50% more time to get
each batch of 5 million messages received !!
And it looks like the effect is accelerating.
Also -- the receiver CPU usage is slowly going down.
CPU usage on the sender is constant. For the receiver,
CPU usage started out around 77% at the beginning of the
test, and after 500 million msgs has fallen to 64% or so.
My plan is to use callgrind to get a snapshot of
sender behavior (I suspect sender is culprit)
at the beginning of a run, and a separate snapshot
later after the slowdown has started.
But i wanted to just mention it here, just in case
anybody has a Great Idea, or in case I get hit by a
truck or something.