Re: Problematic Kannel behaviour with large message stores
Hi, you can search for my patch (http://article.gmane.org/gmane.comp.mobile.kannel.devel/20690/match=spool) that replaces store file with store spool directory. This works with similar performance when you put your store dir on reiserfs or tmpfs under linux. It would be great to hear test results for this patch. Please note that patch is very old and could not apply that you should fix manually. Thanks, Alex Giulio Harding wrote: We've been doing some more stress-testing of Kannel, this time under high-load over long periods of time, an we've encountered some problems when the message store grows large, of the order of hundreds of thousands of messages (in our case, because we're delivering MTs slower than we are receiving MOs, and due to the use of synchronous responses to MO HTTP requests, the MTs are queuing at the bearerbox). We understand that this is an extreme situation, and unlikely to be encountered in normal usage, but we want to cover all our bases (worst-case scenarios) and make sure that Kannel behaves correctly/ predictably in bad situations. - It seems that reading from the message store during Kannel startup is single-threaded, and CPU bound - for large stores (several hundred thousand messages), this can result in very long start-up times, since Kannel seems to not process any messages in the store until it has finished reading them *all* into memory from disk. - Why can't they simply be processed from disk? (i.e. read message from disk, process message, read next message from disk) Given that the messages are in a persistent store, why do they need to be in memory as well? - The bearerbox memory usage grows by around 100MB per hundred thousand messages in the store, as the messages are read from the store, which leads to VM thrashing under extreme circumstances, as the bearerbox footprint grows beyond the RAM size of the server... For one test, after the initial loading of ~634,000 MTs from the store, and bearerbox was using ~780M of RAM. 1+ KB per message seems a lot, given that messages seems to be occupying more space in memory than on disk (that number of MTs was consuming about 255MB on disc) - Once messages are read into memory, they seem to be copied into another queue (this exacerbates the problem above) - why is this? [from kannel admin page] ... Status: running, uptime 0d 0h 12m 39s WDP: received 0 (0 queued), sent 0 (0 queued) v-- *** this number has stopped growing *** SMS: received 0 (329153 queued), sent 4 (0 queued), store size 634247 ^--- *** this number is growing *** SMS: inbound 0.00 msg/sec, outbound 0.01 msg/sec ... (the queue sizes for all the SMPP binds are also growing, so I assume their queues are related to this main queue somehow?) At this point, bearerbox's memory footprint is *still* growing (eventually reaching 1264M virtual, 932M resident) and the server is spending 60+ percent of CPU cycles on disk IO, as the VM thrashes. - Finally, Kannel's queue sizes etc. seem to stop growing: [kannel admin page] Kannel bearerbox version `cvs-20060830'. Build `Aug 31 2006 22:56:51', compiler `3.4.5 20051201 (Red Hat 3.4.5-2)'. System Linux, release 2.6.9-42.0.2.ELsmp, version #1 SMP Wed Aug 23 13:38:27 BST 2006, machine x86_64. Hostname grover, IP 127.0.0.1. Libxml version 2.6.16. Using native malloc. Status: running, uptime 0d 0h 25m 5s WDP: received 0 (0 queued), sent 0 (0 queued) SMS: received 0 (531104 queued), sent 4 (0 queued), store size 634247 SMS: inbound 0.00 msg/sec, outbound 0.00 msg/sec DLR: 0 queued, using internal storage No boxes connected SMSC connections: testSMPP:10.100.123.20:7011/7011:test:smpp (online 178s, rcvd 0, sent 1, failed 0, queued 25776 msgs) testSMPP:10.100.123.20:7012/7012:test:smpp (online 177s, rcvd 0, sent 1, failed 0, queued 25775 msgs) testSMPP:10.100.123.20:7013/7013:test:smpp (online 1505s, rcvd 0, sent 1, failed 0, queued 25775 msgs) testSMPP:10.100.123.20:7014/7014:test:smpp (online 1505s, rcvd 0, sent 1, failed 0, queued 25776 msgs) BUT, the queue size doesn't match the store size, and Kannel has NOT resumed sending those messages (apart from 1 per SMSC, not sure why) The bearerbox still seems to be busy doing something, even though it doesn't seem to be loading messages from store into queues, or sending them: [top output] Mem: 1024580k total, 1009916k used,14664k free, 248k buffers top - 01:46:39 up 1 day, 2:15, 4 users, load average: 1.68, 2.03, 1.81 Tasks: 82 total, 2 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 16.5% us, 1.3% sy, 0.0% ni, 49.8% id, 32.1% wa, 0.0% hi, 0.3% si Mem: 1024580k
Re: Logfiles above 2G
Hi Andreas, it handled fine in glibc with some additional flags defined. These were added by me to the current CVS HEAD and works fine for Linux. Please make sure you use latest CVS HEAD and rerun configure script. Thanks, Alex Andreas Fink wrote: There where days where harddisk had a limit of 2GB or 4GB There where days where partitions had a limit of 2GB or 4GB There where days where files had a limit of 2GB or 4GB Those days are long long gone. However if kannel's gwlib runs in full debug mode and the logfile hits 2GB, the application quits/stops working and when you relaunch it, it appends to the same 2GB logfile and quits again because it cant go beyond this 2GB limit. Now this is not a bug of Kannel but one of the operating system. This bug doesn't exist under MacOS X but it does exist in Linux Fedora 5 with a ext3 filesystem. From reading log.c, I can see that kannel does fopen(filename,a) and then vsfprintf or fprintf. As the file is opened append only, no seeks are used or anything fancy, I can not understand why the file is limited to 2GB even thought the filesystem for sure can handle files larger than 2GB. On our system it takes 2-3 days to hit this problem with a empty log file. As we mainly use MacOS X we never see this problem but having added a new Linux machine to the park, I'm puzzled to see this problem I've already spotted in 2001 and would have expected to have been long fixed in current linux kernels. Anyone have a hint here? Andreas Fink Fink Consulting GmbH --- Tel: +41-61-332 Fax: +41-61-331 Mobile: +41-79-2457333 Address: Clarastrasse 3, 4058 Basel, Switzerland E-Mail: [EMAIL PROTECTED] Homepage: http://www.finkconsulting.com --- ICQ: 8239353 MSN: [EMAIL PROTECTED] AIM: smsrelay Skype: andreasfink Yahoo: finkconsulting SMS: +41792457333 -- Thanks, Alex
Re: Sending long text messages using SMPP message_payload
Hi, because not all SMSCs support this feature. Raul Igrisan wrote: Hi, Can someone please explain if there is a reason why Kannel doesn't use smpp.message_payload for sending long text messages over SMPP instead of splitting it into 160-chars messages? Thank you, Raul Igrisan -- Thanks, Alex
Re: Problematic Kannel behaviour with large message stores
good ! - Original Message - From: Alexander Malysh [EMAIL PROTECTED] To: devel@kannel.org Sent: Friday, September 08, 2006 1:03 PM Subject: Re: Problematic Kannel behaviour with large message stores Hi, will try todo it next week. I have it up to 80% ready with possibility to select store file or store spool... Thanks, Alex Vincent CHAVANIS wrote: Alex, Could you please review you patch to fit the actual CVS tree, as we can test it. Then, how it comes this patch does not have been commited as it was ++2 regards Vincent -- Telemaque - 06200 NICE - (FR) Service Technique/Reseau - NOC Developpement SMS/MMS/Kiosques http://www.telemaque.fr/ [EMAIL PROTECTED] Tel : +33 4 93 97 71 64 (fax 68) - Original Message - From: Alexander Malysh [EMAIL PROTECTED] To: devel@kannel.org Sent: Friday, September 08, 2006 12:06 PM Subject: Re: Problematic Kannel behaviour with large message stores Hi, you can search for my patch (http://article.gmane.org/gmane.comp.mobile.kannel.devel/20690/match=spool) that replaces store file with store spool directory. This works with similar performance when you put your store dir on reiserfs or tmpfs under linux. It would be great to hear test results for this patch. Please note that patch is very old and could not apply that you should fix manually. Thanks, Alex Giulio Harding wrote: We've been doing some more stress-testing of Kannel, this time under high-load over long periods of time, an we've encountered some problems when the message store grows large, of the order of hundreds of thousands of messages (in our case, because we're delivering MTs slower than we are receiving MOs, and due to the use of synchronous responses to MO HTTP requests, the MTs are queuing at the bearerbox). We understand that this is an extreme situation, and unlikely to be encountered in normal usage, but we want to cover all our bases (worst-case scenarios) and make sure that Kannel behaves correctly/ predictably in bad situations. - It seems that reading from the message store during Kannel startup is single-threaded, and CPU bound - for large stores (several hundred thousand messages), this can result in very long start-up times, since Kannel seems to not process any messages in the store until it has finished reading them *all* into memory from disk. - Why can't they simply be processed from disk? (i.e. read message from disk, process message, read next message from disk) Given that the messages are in a persistent store, why do they need to be in memory as well? - The bearerbox memory usage grows by around 100MB per hundred thousand messages in the store, as the messages are read from the store, which leads to VM thrashing under extreme circumstances, as the bearerbox footprint grows beyond the RAM size of the server... For one test, after the initial loading of ~634,000 MTs from the store, and bearerbox was using ~780M of RAM. 1+ KB per message seems a lot, given that messages seems to be occupying more space in memory than on disk (that number of MTs was consuming about 255MB on disc) - Once messages are read into memory, they seem to be copied into another queue (this exacerbates the problem above) - why is this? [from kannel admin page] ... Status: running, uptime 0d 0h 12m 39s WDP: received 0 (0 queued), sent 0 (0 queued) v-- *** this number has stopped growing *** SMS: received 0 (329153 queued), sent 4 (0 queued), store size 634247 ^--- *** this number is growing *** SMS: inbound 0.00 msg/sec, outbound 0.01 msg/sec ... (the queue sizes for all the SMPP binds are also growing, so I assume their queues are related to this main queue somehow?) At this point, bearerbox's memory footprint is *still* growing (eventually reaching 1264M virtual, 932M resident) and the server is spending 60+ percent of CPU cycles on disk IO, as the VM thrashes. - Finally, Kannel's queue sizes etc. seem to stop growing: [kannel admin page] Kannel bearerbox version `cvs-20060830'. Build `Aug 31 2006 22:56:51', compiler `3.4.5 20051201 (Red Hat 3.4.5-2)'. System Linux, release 2.6.9-42.0.2.ELsmp, version #1 SMP Wed Aug 23 13:38:27 BST 2006, machine x86_64. Hostname grover, IP 127.0.0.1. Libxml version 2.6.16. Using native malloc. Status: running, uptime 0d 0h 25m 5s WDP: received 0 (0 queued), sent 0 (0 queued) SMS: received 0 (531104 queued), sent 4 (0 queued), store size 634247 SMS: inbound 0.00 msg/sec, outbound 0.00 msg/sec DLR: 0 queued, using internal storage No boxes connected SMSC connections: testSMPP:10.100.123.20:7011/7011:test:smpp (online 178s, rcvd 0, sent 1, failed 0, queued 25776 msgs) testSMPP:10.100.123.20:7012/7012:test:smpp (online 177s, rcvd 0, sent 1, failed 0, queued 25775 msgs) testSMPP:10.100.123.20:7013/7013:test:smpp (online 1505s, rcvd 0, sent 1, failed 0, queued 25775
Re: [PATCH] Re: XML Post results in Sender missing and no global set,rejected
Paul Keogh wrote:
+1 for this.
This patch fixes the OA handling in the XML POST.
It also fixes the double free of the POST body but incorrectly identifies
the second free.
The body should not be freed in get_x_kannel_from_xml () because it is
freed in the outer sendsms_thread () function.
It is not freed in the get_tag() as stated in the bug report.
yup, votes enough... commited to cvs.
Thanks guys for review.
Stipe
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
Re: Mutex Failure
Silver wrote:
Thanks,
I have both the csv version and kannel 1.4.0 on different machines. For the
csv version it the smsbox hangs as i try to run it as indicated bellow
$ ./smsbox sk_smskannel.conf
2006-09-07 12:15:27 [2824] [0] INFO: Debug_lvl = -1, log_file = none,
log_lvl = 0
2006-09-07 12:15:27 [2824] [0] PANIC: gwlib/thread.c:139: mutex_lock_real:
Mutex failure! (Called from gwlib/list.c:503:lock.)
Then while exiting from the bearerbox for kannel 1.4.0 its hangs as
indicated below
2006-09-07 09:21:18 [4780] [0] INFO: Total WDP messages: received 0, sent 0
2006-09-07 09:21:19 [4780] [0] INFO: Total SMS messages: received 0, sent 0
2006-09-07 09:21:20 [4780] [0] DEBUG: Immutable octet strings: 285.
2006-09-07 09:21:21 [4780] [0] PANIC: gwlib/thread.c:139:
mutex_lock_real:Mutex failure! (Called from gwlib/list.c:503:lock.)
could it be that the enviroment has a problem?
forst of all, cygwin 1.5.x is *NOT* considered fully pthreads safe. This means
cygwin has still major issues in the posix thread implementation. So Kannel
won't run in production stable quality on Cygwin/Win32 hosts.
I see the same behaviour on my local Cygwin 1.5.x hosts, but couldn't resolve
the reason, assuming that it is a bug internally in Cygwin's pthreads support.
Stipe
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
Re: Sending long text messages using SMPP message_payload
Alexander Malysh wrote:
Hi,
because not all SMSCs support this feature.
agree'ing here with Alex.
The issue is that SMPP is considered a standardized protocol, but in the
real-world we have vendors that implement certain feature (in case they are
considered optional) and others don't.
In this case some SMSCs may be capable to handle this, other's don't. We try to
abstract as much as possible.
Stipe
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
