2014-03-25 4:49 GMT+04:00 Jeff Thorn <[email protected]>: > Hi spamden, > > I just wanted to follow up on this issue. It appeared that the reason for > the sudden excessive queuing was because of another VM on the same VM > server that was doing excessive disk I/O. We moved the VM and now only the > kannel server is running and things seem much better. > > FYI....we are using store-type = file. And kannel would definitely start > swapping once queues got too full. >
Try using store-type = spool and increase maximum open files limits in /etc/security/limits.conf for kannel user. (e.g. kannel soft nofile 999999 kannel hard nofile 999999) spool store type worked for us better in highly loaded environment > > I do have one final question based on your previous reply. You said "Yes, > it's by design, new submitted messages are going right after they are > submitted". If new submitted messages go right away, even though there are > older messages queued, can you briefly explain the logic that kannel uses > to determine whether a message should be queued or sent right away? > There is a priority field: you can use it to make messages go as faster as they can. (i.e. pass cgi-bin/sendsms?....&priority=3 to your script) from doc: priority number Optional. Sets the Priority value (range 0-3 is allowed). However what I've said is based on my observations (I'm using kannel+sqlbox) and I saw several times queue of 1000-2000 MTs and if I submit a new MT it goes right after sqlbox submits it to bearerbox so from my practice sqlbox+bearerbox works in LIFO way (Last Input First Out). > > Thanks, > Jeff > > > > On Wed, Mar 12, 2014 at 2:02 AM, spameden <[email protected]> wrote: > >> >> >> >> 2014-03-11 23:48 GMT+04:00 Jeff Thorn <[email protected]>: >> >> Thanks spamden. 1 and 2 seem fine. We are working on 3 and 4. I don't >>> know what would have caused the sudden change in throughput when everything >>> was working fine. But I will send an update for the group if we figure it >>> out. >>> >>> I am noticing something strange though....our MT queues are starting to >>> deplete now. However, if I look at the kannel "store-status" page, I still >>> see messages queued from 4 hours ago. All new MT requests, however, are now >>> going out immediately. Is this by design? Why doesn't kannel send queued >>> messages before newer ones? >>> >> >> Yes, it's by design, new submitted messages are going right after they >> are submitted unless you use an sqlbox with custom SQL query to get only >> certain MT messages to send to. >> >> What do you use for store? Spool or just a single file? >> >> Also check any errors in the dmesg or syslog. And check if kannel is >> swapping or anything else (free -m). >> >>> >>> Thanks, >>> Jeff >>> >>> >>> >>> On Tue, Mar 11, 2014 at 3:05 PM, spameden <[email protected]> wrote: >>> >>>> >>>> >>>> >>>> 2014-03-11 22:46 GMT+04:00 Jeff Thorn <[email protected]>: >>>> >>>> Hi group, >>>>> >>>>> We currently have 7 different Tx binds setup to our SMSC. We have been >>>>> sending MT volumes of up to 3,000,000 messages per day at a rate of almost >>>>> 200 / second. We've been doing this for more than a year now with no >>>>> problems. >>>>> >>>>> Suddenly about a week and a half ago, we noticed our throughput drop >>>>> suddenly. The kannel status page shows we are only sending about 150 msgs >>>>> / >>>>> sec, but each of the 7 binds have over 50,000 MT messages queued. These >>>>> queues eventually empty out, but some messages get delivered hours later >>>>> than when they were sent by our platform. >>>>> >>>>> I am trying to troubleshoot this sudden drop in throughput and >>>>> excessive queuing. The only thing I can think of is network latency >>>>> between >>>>> our network and the SMSC network. I don't know how to go about >>>>> troubleshooting this though. Is there any other reason that might explain >>>>> the sudden need for excessive queuing? >>>>> >>>> >>>> There are many potential factors could be related to the actual sending >>>> speed. >>>> >>>> Also I don't find speed in the kannel's web interface very accurate. >>>> >>>> 1) set verbose mode to 0 for your SMSC logs and check if there are >>>> any throttling errors >>>> 2) check for maximum number of open files for kannel (ulimit under >>>> kannel user) and adjust if needed (this needed if you use kanel-store = >>>> spool) >>>> 3) check your backend speed (e.g. dlr-url script or MySQL db how much >>>> queries it can handle simultaneously). >>>> 4) contact your SMSC operator for troubleshooting (e.g. tcpdump the >>>> traffic and try to check if there are any network problems, massive tcp >>>> retransmits or anything else) >>>> >>>> >>>>> >>>>> Side question.....if kannel was restarted with so many MT messages >>>>> queued for delivery, would they all be lost? >>>>> >>>> >>>> No, if you're using kannel-spool or kannel-store mode. >>>> >>>>> >>>>> Any assistance or tips for troubleshooting why so many messages are >>>>> suddenly queuing would be extremely appreciated. >>>>> >>>>> Thanks, >>>>> Jeff >>>>> >>>> >>>> >>> >> >
