Dear All, Ultimately the issue is resolved. This is our SMSc who was blocking sms and resetting connections. So kannel Q was filing up rapidly and on stopping our sms application Q was not get released and sms stuck there for ever.
let me explain the use case so that if some other might have same issue. We populated DB, 2 million customers. Our application was sending sms rapidly @ 120~140 sms/sec. Q was also populating meanwhile but this was not a bottleneck because sms were not stuck there. Actually, our db engineer loaded some numbers (about 100) for 10000 times each :P. SMSc was filtering theses multiple sms attempts and was not allowing sms to pass through. Due to this reason SMS go into Q and stuck there for ever. We realized the problem and again uploaded the numbers after uniqueness check. Now we are again back to track :) We also setup store file for Q after your suggestions. Thank you for the list for reply. -- Regards, Abdul Basit | +92 32 1416 4196 2010/10/17 Nikos Balkanas <[email protected]> > Hi, > > This is your SMSc problem. They have oversold their service and exceeded > their connection limit. Talk to them, so that they either raise their > limits, or change SMSc. > > BR, > Nikos > > ----- Original Message ----- > *From:* Abdul Basit <[email protected]> > *To:* Nikos Balkanas <[email protected]> > *Cc:* Alvaro Cornejo <[email protected]> ; [email protected] > *Sent:* Saturday, October 16, 2010 11:08 PM > *Subject:* Re: queue full issue > > SMS delivery is too slow. only 17 sms/sec. Most of the sms go to queue and > Q size grow very rapidly. > When we stop sending sms Q size should reduce. But this dont happen. SMS > stay in Q for ever. > > Earlier,Β with same configs, we were able to send 120 sms/sec and Q size > was not growing that much rapidly. After we pause our application, Q count > keep on reducing. > > We didnt changed config at our end. Β > > Any advise will be helpful inΒ figuring outΒ thisΒ anomaly.Β > Β > > 2010/10/17 Nikos Balkanas <[email protected]> > >> What is your other issue? >> >> >> BR, >> Nikos >> ----- Original Message ----- From: Abdul Basit >> To: Alvaro Cornejo >> Cc: Nikos Balkanas ; [email protected] >> Sent: Friday, October 15, 2010 9:38 PM >> >> Subject: Re: queue full issue >> >> >> That is what exactly happening with my queue. I lost Q each time i restart >> kannel :( >> Let me define storage. Thank you for the tip. >> >> >> Any clue on other issue regarding smsc. >> >> >> >> >> >> On Fri, Oct 15, 2010 at 10:49 PM, Alvaro Cornejo < >> [email protected]> wrote: >> >> You have not defined a storage for your queue, thus it is being build >> in memory and as stated in your 1st mail, you are running out of >> memory. >> >> Try to setup the store type to file and see what happens. This will >> also let you keep the messages in queue if kannel or the server >> crashed or if you need to reset kannel before the queue is empty. As >> you have your setup now, if you do reset kannel you will lost all >> messages in its queue. >> >> Regards >> >> Alvaro >> >> >> >> |-----------------------------------------------------------------------------------------------------------------| >> EnvΞ½e y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier >> celular y Nextel >> en el PerΟ , MΞΉxico y en mas de 180 paises. Use aplicaciones 2 vias via >> >> SMS y GPRS online >> Β Β Β Β Β Β Visitenos en www.perusms.NET www.smsglobal.com.mx y >> >> www.pravcom.com >> >> >> >> >> On Fri, Oct 15, 2010 at 10:21 AM, Abdul Basit <[email protected]> >> wrote: >> >>> My SMSc increased window size at there end. Now we are not >>> getting Throttling errors. >>> But messages are still keep going in the Q. When we stop our sms sender >>> application this Q count should reduce and Sent count should increase. >>> This >>> was the actual behavior. >>> >>> Now after stopping sms feed, Q and Sent count don't reduce that mean sms >>> are >>> not get delivered as expected. >>> I am getting same (Message Queue Full) logs as i shared earlier. SMSc ppl >>> are saying that they are innocent. >>> I want to understand what is happening. >>> You advise/suggestions are valuable to me. >>> My kannel.conf is >>> group = core >>> dlr-storage = mysql >>> admin-port = 13000 >>> smsbox-port = 13001 >>> admin-password = adminpass >>> log-level = 1 >>> log-file = "/var/log/kannel/kannel.log" >>> box-deny-ip = "*.*.*.*" >>> box-allow-ip = "*.*.*.*" >>> access-log = "/var/log/kannel/access.log" >>> sms-resend-retry = 1 >>> sms-outgoing-queue-limit = -1 >>> sms-incoming-queue-limit = 0 >>> sms-resend-retry = 1 >>> sms-resend-freq = 1800 >>> # session 1 >>> group = smsc >>> smsc = smpp >>> log-level = 0 >>> smsc-id = LA_1768 >>> host = 10.200.18.116 >>> port = 5026 >>> smsc-username = CSMS1 >>> smsc-password = CSMS1 >>> system-type = CSMS1 >>> transceiver-mode = true >>> alt-charset = gsm >>> receive-port = 0 >>> allowed-smsc-id = LA_1768 >>> throughput = 88 >>> wait-ack = 1800 >>> # session 2 >>> group = smsc >>> smsc = smpp >>> log-level = 0 >>> smsc-id = LA_1768 >>> host = 10.200.18.116 >>> port = 5026 >>> smsc-username = CSMS1 >>> smsc-password = CSMS1 >>> system-type = CSMS1 >>> transceiver-mode = true >>> alt-charset = gsm >>> receive-port = 0 >>> allowed-smsc-id = LA_1768 >>> throughput = 88 >>> wait-ack = 1800 >>> group = smsbox >>> bearerbox-host = 172.27.103.19 >>> sendsms-port = 13013 >>> log-level = 1 >>> log-file = "/var/log/kannel/smsbox.log" >>> access-log = "/var/log/kannel/access.log" >>> group = sendsms-user >>> username = testuser >>> password = abc >>> user-allow-ip = "*.*.*.*" >>> max-messages = 4 >>> concatenation = 1 >>> forced-smsc = LA_1768 >>> group = sms-service >>> keyword = test >>> text = "HI Test Reply from GO" >>> group = sms-service >>> keyword = default >>> text = "Sorry i dont understand this" >>> omit-empty = true >>> get-url = >>> http://localhost:80/receivesms.php?q=%k&sender=%p&message=%a&to=%P >>> max-messages = 2 >>> concatenation = 1 >>> group = mysql-connection >>> id = mydlr >>> host = 127.0.0.1 >>> username = kannel >>> password = kannel123 >>> database = kannel >>> max-connections = 25 >>> group = dlr-db >>> id = mydlr >>> table = dlr >>> field-smsc = smsc >>> field-timestamp = ts >>> field-destination = destination >>> field-source = source >>> field-service = service >>> field-url = url >>> field-mask = mask >>> field-status = status >>> field-boxc-id = boxc >>> >>> On Fri, Oct 15, 2010 at 5:21 PM, Abdul Basit <[email protected]> >>> wrote: >>> >>>> >>>> we are sure that this is their end issue. But we have to tell them with >>>> some prove. >>>> I sent them tcpdump as well. >>>> They are saying that they found following messages at their end. >>>> >>>> SMPPSERV_APP_BIND_FAIL Β Application has failed to bind as transceiver >>>> to >>>> SMPP server system id CSMS1 reason TRX connection limit exceeded Β node >>>> ApplicationAccessGroup.SMPPServerGroup.smppServer >>>> SMPPSERV_APP_BIND_FAIL Β Application has failed to bind as transceiver >>>> to >>>> SMPP server system id CSMS1 reason TRX connection limit exceeded Β node >>>> ApplicationAccessGroup.SMPPServerGroup.smppServer >>>> SMPPSERV_APP_BIND_FAIL Β Application has failed to bind as transceiver >>>> to >>>> SMPP server system id CSMS1 reason TRX connection limit exceeded Β node >>>> >>>> ApplicationAccessGroup.SMPPServerGroup.smppServer >>>> But if we are exceeding connection limit then how come we were able to >>>> send sms in last 1 and half days with same configuration. >>>> One more observation is that messages are getting queued even we send 2 >>>> sms per sec. 1 sms delivers and 1 sms go in queue. >>>> This is strange behavior. >>>> >>>> >>>> >>>> >>>> 2010/10/15 Nikos Balkanas <[email protected]> >>>> >>>>> >>>>> Hi, >>>>> >>>>> Talk to your SMSc. They are the ones that reject your SMS, because >>>>> their >>>>> Queue is full. >>>>> >>>>> BR, >>>>> Nikos >>>>> ----- Original Message ----- From: Abdul Basit >>>>> To: [email protected] >>>>> Sent: Friday, October 15, 2010 9:42 AM >>>>> Subject: queue full issue >>>>> >>>>> >>>>> Dear list, >>>>> >>>>> >>>>> We are facing problem that most of the messages go in queue. >>>>> Following is the snapshot of what we are facing. >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ >>>>> Kannel bearerbox version `svn-r4837M'. Build `Sep 28 2010 18:53:41', >>>>> compiler `4.1.2 20080704 (Red Hat 4.1.2-48)'. System Linux, release >>>>> 2.6.18-194.el5, version #1 SMP Fri Apr 2 14:58:35 EDT 2010, machine >>>>> i686. >>>>> Hostname dbcsms, IP 127.0.0.1. Libxml version 2.6.26. Using OpenSSL >>>>> 0.9.8e-fips-rhel5 01 Jul 2008. Compiled with MySQL 5.0.77, using MySQL >>>>> 5.0.77. Using native malloc. >>>>> >>>>> >>>>> Status: running, uptime 0d 0h 55m 21s >>>>> >>>>> >>>>> WDP: received 0 (0 queued), sent 0 (0 queued) >>>>> >>>>> >>>>> SMS: received 207 (0 queued), sent 33731 (2 queued), store size -1 >>>>> SMS: inbound (0.05,0.05,0.06) msg/sec, outbound (0.09,1.97,10.16) >>>>> msg/sec >>>>> >>>>> >>>>> DLR: received 0, sent 0 >>>>> DLR: inbound (0.00,0.00,0.00) msg/sec, outbound (0.00,0.00,0.00) >>>>> msg/sec >>>>> DLR: 2 queued, using mysql storage >>>>> >>>>> >>>>> Box connections: >>>>> Β smsbox:(none), IP 192.168.13.19 (0 queued), (on-line 0d 0h 55m 13s) >>>>> SMSC connections: >>>>> Β LA_1768[LA_1768] Β Β SMPP:10.200.18.116:5026/5026:CSMS1:CSMS1(online >>>>> >>>>> 3321s, rcvd: sms 101 / dlr 0, sent: sms 16898 / dlr 0, failed 30, >>>>> queued >>>>> 47948 msgs) >>>>> Β LA_1768[LA_1768] Β Β SMPP:10.200.18.116:5026/5026:CSMS1:CSMS1(online >>>>> >>>>> 3321s, rcvd: sms 106 / dlr 0, sent: sms 16833 / dlr 0, failed 24, >>>>> queued >>>>> 47963 msgs) >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >>>>> >>>>> >>>>> We getting all messages through till yesterday morning and not residing >>>>> in >>>>> Q. We didn't changed any thing at our end. >>>>> I checked with svn-r4858M release as well and also check via other >>>>> servers >>>>> in production to confirm that this is not machine issue. >>>>> >>>>> >>>>> Most of the ram is in use. >>>>> [r...@sms ~]# free -m >>>>> Β Β Β Β Β Β Β Β total Β Β Β used Β Β Β free Β Β shared >>>>> Β Β buffers >>>>> cached >>>>> Mem: Β Β Β Β Β 3868 Β Β Β 3747 Β Β Β Β 120 Β Β Β Β Β 0 Β >>>>> Β Β Β 218 3185 >>>>> -/+ buffers/cache: Β Β Β Β 343 Β Β Β 3524 >>>>> Swap: Β Β Β Β 4094 Β Β Β Β Β 0 Β Β Β 4094 >>>>> >>>>> >>>>> >>>>> Below are the kannel log. >>>>> >>>>> >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: SMPP[LA_7786]: SMSC returned >>>>> error >>>>> code 0x00000014 (Message Queue Full) in response to submit_sm. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert string from >>>>> <UTF-8> >>>>> to <gsm> - probably broken type names. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert msgdata from >>>>> charset <UTF-8> to <gsm>, will send as is. >>>>> 2010-10-15 11:30:42 [17462] [7] WARNING: SMPP: PDU NULL terminated >>>>> string >>>>> (message_id) has no NULL. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: SMPP[LA_7786]: SMSC returned >>>>> error >>>>> code 0x00000014 (Message Queue Full) in response to submit_sm. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert string from >>>>> <UTF-8> >>>>> to <gsm> - probably broken type names. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert msgdata from >>>>> charset <UTF-8> to <gsm>, will send as is. >>>>> 2010-10-15 11:30:42 [17462] [6] WARNING: SMPP: PDU NULL terminated >>>>> string >>>>> (message_id) has no NULL. >>>>> 2010-10-15 11:30:42 [17462] [6] ERROR: SMPP[LA_7786]: SMSC returned >>>>> error >>>>> code 0x00000014 (Message Queue Full) in response to submit_sm. >>>>> 2010-10-15 11:30:42 [17462] [6] ERROR: Failed to convert string from >>>>> <UTF-8> >>>>> to <gsm> - probably broken type names. >>>>> 2010-10-15 11:30:42 [17462] [6] ERROR: Failed to convert msgdata from >>>>> charset <UTF-8> to <gsm>, will send as is. >>>>> 2010-10-15 11:30:42 [17462] [7] WARNING: SMPP: PDU NULL terminated >>>>> string >>>>> (message_id) has no NULL. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: SMPP[LA_7786]: SMSC returned >>>>> error >>>>> code 0x00000014 (Message Queue Full) in response to submit_sm. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert string from >>>>> <UTF-8> >>>>> to <gsm> - probably broken type names. >>>>> 2010-10-15 11:30:42 [17462] [7] ERROR: Failed to convert msgdata from >>>>> charset <UTF-8> to <gsm>, will send as is. >>>>> >>>>> You suggestions are highly appreciated. >>>>> >>>>> -- >>>>> >>>>> Regards, >>>>> >>>>> Abdul Basit | +92 32 1416 4196 >>>>> >>>> >> >> >> >> -- >> >> Regards, >> >> Abdul Basit | +92 32 1416 4196 >> > > > > -- > Regards, > > Abdul Basit | +92 32 1416 4196 > >
