Bozo -- i see you're right. The size of my delivery list -- i.e. the list at connection->work_head -- is slowly increasing, and that's the problem. There is something I'm not handling, which gets a lot worse under heavy system load, and my sender never digs itself out.
But -- what I want here is a canonical example of how to get high throughput with proton-c at the engine level -- and what I am hearing is that I should be using the event-collector interface which Dispatch uses. That's what we want to steer new users toward. So far the simple fixes I've tried have all resulted in zero speed... So! Rather than spending more time fixing these examples I will switch to the event collector model and see if I can get a nice little example working that way. (And from what I hear, it will be much nicer and much littler.) At least I know better now what I am trying to do: Get a send/receive example at the lowest level, that we want to direct new users toward, that has throughput as high as possible, and that can run flat-out on a time scale of days or weeks without crashing, without growing, and without slowing down. ----- Original Message ----- On 28. 10. 14 20:18, Michael Goulish wrote: > I have gotten callgrind call-graph pictures of my proton/engine sender > and receiver when the test is running fast and when it slows down. > > The difference is in the sender -- when running fast, it is spending > most of its time in the subtree of pn_connector_process() .. like 71%. > > When it slows down it is instead spending 47% in pn_delivery_writeable(), > and only 17% in pn_connector_process(). > > > Since it is still not instantly obvious to me what has happened, > I thought I would share with you-all. > > > Please see cool pictures at: > > > http://people.apache.org/~mgoulish/protonics/performance/slowdown/2014_10_28/svg/psend_fast.svg > > > http://people.apache.org/~mgoulish/protonics/performance/slowdown/2014_10_28/svg/psend_slow.svg > > > > > To recap -- I can trigger this condition by getting the box busy while > my proton/engine test is running. I.e. by doing a build. > Even though I stop the build, and all 6 other processors on > the box go back to being idle -- the test never recovers. > > The receiver goes down to 50% CPU or worse -- but these pictures > show that the behavior change is in the sender. > > > look at call counts, for pn_connector_process() and pn_delivery_writable() fast : ratio 1 : 5 slow : ratio 1 : 244.5 (!) The iteration over connection work list gets really expensive, which means the connection thinks it has to work on other stuff than what psend.c wants to work on. I still think that the call to pn_delivery() in psend.c is in a really unfortunate spot. btw, why do you iterate over connection work list at all, you could just remember the delivery when calling pn_delivery()? Bozzo
