On Tue, 2010-01-26 at 12:05 -0500, Dale Worley wrote: > On a test system, we've been noticing problems with overflowing of > message queues for OsServerTask's. To help diagnose this problem, I've > added log messages to OsMsgQ and related classes: > > If a message can't be sent to a queue because it is full, this message > is logged: > > OsSysLog::add(FAC_KERNEL, PRI_CRIT, > "OsMsgQShared::doSendCore message send failed for > queue '%s' - no room, ret = %d", > mName.data(), ret); > > If a message is sent to a queue and the queue is over half full, this > message is logged: > > OsSysLog::add(FAC_KERNEL, PRI_CRIT, > "OsMsgQShared::doSendCore message queue '%s' is over > half full - count = %d, max = %d", > mName.data(), count, max);
PRI_CRIT is too severe a level for that one - it probably should only be a warning. We want it to be true that _every_ CRIT level log entry is some problem that must be tracked down, and while this may be rare, it might not be fatal or even a big problem. What is 'ret' going to be? > There are a number of considerations and questions: > > - These messages are not logged if the queue's name is "syslog". This > is to ensure that if the logging task's queue gets full, the system does > not lock up in an infinite loop. > > - Of course, the caller of ::send() should check the return code to see > if it succeeded, but in almost no situation, does the caller do so. And > in a lot of cases, there is nothing useful the caller could do, other > than log a message and give up. In almost no case, is there a way for > the caller to recover without a functional error. So it's useful to > have uniform logging of queue-full conditions. Agreed > - Logging when a queue is half-full helps discover problems before they > lead to functional errors. While a queue size of 100 seems "big enough" > intuitively, there are a number of batch-like operations in sipXecs that > can lead to rapid sequences of over 100 complex operations. > Systematically logging situations where queues are not nearly empty can > point out potential bottlenecks. ... or create them, if logging affects the timing... > - It's not clear that the concept of "queue size" is useful. sipXecs is > not limited by memory consumption, so there doesn't seem to be a reason > to worry if, say, 1000 requests get queued for a task. I'm not sure I agree with that... I've found some pretty serious design errors that first manifested as a queue overflow. The alternative may often be to let queues grow until virtual memory is exhausted, by which time log files may be so big that debugging is much more difficult. _______________________________________________ sipx-dev mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-dev Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev sipXecs IP PBX -- http://www.sipfoundry.org/
