And finally, yesterday when I was really trying to stress the brokers, I managed to bring one down so that it was completely unresponsive. There were a lot of connection timeouts, as you'd expect since I was hitting it hard, but two things in the logs stood out.
There were a bunch of framing errors, but it was at a time when the brokers were still responsive. It was a couple hours away from cratering: 2015-04-14 14:37:48 [System] error qpid::sys::AsynchIOHandler::readbuff: illegal-argument: Not enough data for field table. (/pilot/dev/qpid_0.32_build/qpid/cpp/src/qpid/framing/FieldTable.cpp:302) Starting at the time that the broker became unresponsive, there were repeated (every 2 seconds) channel exceptions: 2015-04-14 16:30:32 [Broker] error qpid::broker::<unnamed>::DefaultErrorListener::channelException: Channel exception: not-attached: Channel 8 is not attached (/pilot/dev/qpid_0.32_build/qpid/cpp/src/qpid/amqp_0_10/SessionHandler.cpp:39)
