Earlier I reported a very gradual slowdown in the performance of my simple 1-sender 1-receiver test, on RHEL 7.0 and Fedora 20 but not on RHEL 6.3 .
The slowdown caused the test to end up running at half speed after a billion or two billion messages. ( Which took hours to run. ) I now know how to cause this slowdown to happen any time, and it works just as well on RHEL 6.3 as it does on RHEL 7.0 . All I have to do is make the machine busy. Even though I do not swamp all processors -- in fact, I leave a couple processors idle -- my receiver program slows down when the machine becomes busy -- ***and it never recovers***. I have been doing qpid builds, for example. Even after I interrupt the build -- many minutes later, long after the box has become idle but for my sender and receiver, the receiver's CPU usage is still depressed, and performance has been cut to 1/2 or 1/3 of what it was at the beginning of the test. It never comes back. In fact, I can make it ratchet down again by running another build. This is nothing magic about builds -- that was just a convenient way of making the box busy. I will be making a Jira for this later today. I was able to make a callgrind picture for the receiver from when the test was fast and from when it was slow. I will attach all my info to the Jira.
