On a test system, we've been noticing problems with overflowing of
message queues for OsServerTask's. To help diagnose this problem, I've
added log messages to OsMsgQ and related classes:
If a message can't be sent to a queue because it is full, this message
is logged:
OsSysLog::add(FAC_KERNEL, PRI_CRIT,
"OsMsgQShared::doSendCore message send failed for queue
'%s' - no room, ret = %d",
mName.data(), ret);
If a message is sent to a queue and the queue is over half full, this
message is logged:
OsSysLog::add(FAC_KERNEL, PRI_CRIT,
"OsMsgQShared::doSendCore message queue '%s' is over
half full - count = %d, max = %d",
mName.data(), count, max);
There are a number of considerations and questions:
- These messages are not logged if the queue's name is "syslog". This
is to ensure that if the logging task's queue gets full, the system does
not lock up in an infinite loop.
- Of course, the caller of ::send() should check the return code to see
if it succeeded, but in almost no situation, does the caller do so. And
in a lot of cases, there is nothing useful the caller could do, other
than log a message and give up. In almost no case, is there a way for
the caller to recover without a functional error. So it's useful to
have uniform logging of queue-full conditions.
- Logging when a queue is half-full helps discover problems before they
lead to functional errors. While a queue size of 100 seems "big enough"
intuitively, there are a number of batch-like operations in sipXecs that
can lead to rapid sequences of over 100 complex operations.
Systematically logging situations where queues are not nearly empty can
point out potential bottlenecks.
- It's not clear that the concept of "queue size" is useful. sipXecs is
not limited by memory consumption, so there doesn't seem to be a reason
to worry if, say, 1000 requests get queued for a task.
Comments?
Dale
_______________________________________________
sipx-dev mailing list [email protected]
List Archive: http://list.sipfoundry.org/archive/sipx-dev
Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev
sipXecs IP PBX -- http://www.sipfoundry.org/
