> On a test system, we've been noticing problems with > overflowing of message queues for OsServerTask's. To help > diagnose this problem, I've added log messages to OsMsgQ and > related classes: > > If a message can't be sent to a queue because it is full, > this message is logged: > > OsSysLog::add(FAC_KERNEL, PRI_CRIT, > "OsMsgQShared::doSendCore message send > failed for queue '%s' - no room, ret = %d", > mName.data(), ret); > > If a message is sent to a queue and the queue is over half > full, this message is logged: > > OsSysLog::add(FAC_KERNEL, PRI_CRIT, > "OsMsgQShared::doSendCore message > queue '%s' is over half full - count = %d, max = %d", > mName.data(), count, max); > > There are a number of considerations and questions: > > - These messages are not logged if the queue's name is > "syslog". This is to ensure that if the logging task's queue > gets full, the system does not lock up in an infinite loop. > > - Of course, the caller of ::send() should check the return > code to see if it succeeded, but in almost no situation, does > the caller do so. And in a lot of cases, there is nothing > useful the caller could do, other than log a message and give > up. In almost no case, is there a way for the caller to > recover without a functional error. So it's useful to have > uniform logging of queue-full conditions. > > - Logging when a queue is half-full helps discover problems > before they lead to functional errors. While a queue size of > 100 seems "big enough" > intuitively, there are a number of batch-like operations in > sipXecs that can lead to rapid sequences of over 100 complex > operations. > Systematically logging situations where queues are not nearly > empty can point out potential bottlenecks. > > - It's not clear that the concept of "queue size" is useful. > sipXecs is not limited by memory consumption, so there > doesn't seem to be a reason to worry if, say, 1000 requests > get queued for a task. > > Comments?
I was faced with some queue-full-related issues of my own lately and it took me a while to converge on the problem due to absence of logging. I therefore think that the extra logging is a good idea (The 'syslog' avoidance was a good catch BTW ). Now, WRT queue sizes, they should probably be sized to be able to cope with the biggest transient we have in our system. If we know that a valid configuration can generate n messages in rapid succession where n > 100 then I believe that it would make sense to increase the queue size to a number larger than n. Also, it would be best if that could be done on a per-process basis, that way we do not have to increase the default to accommodate the worst-case process. _______________________________________________ sipx-dev mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-dev Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev sipXecs IP PBX -- http://www.sipfoundry.org/
