RE: Queue in Store
yes, are you using a direct smpp connection ? if so, do you know what is the tps allowed by your provider ? De : Vitória Silva [mailto:rodrigofsan...@gmail.com] Envoyé : mercredi 13 décembre 2017 12:18 À : Vangelis Typaldos Cc : Robin C; info.ubichip; users@kannel.org Objet : Re: Queue in Store ok, but to help you better what kind of equipment do you use? Do you use 3g modems or some gsm gateway? How is your continuous sending configured? <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail> Livre de vírus. <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail> www.avast.com. 2017-12-13 8:09 GMT-03:00 Vangelis Typaldos <vty...@outlook.com>: As far as you have a working kannel environment you should check for latency, retransmission, retries... A tcpdump would help you identify the source of issue. BR Vangelis Sent from my BlackBerry 10 smartphone. From: Robin C Sent: Τετάρτη, 13 Δεκεμβρίου 2017 - 13:01 To: info.ubichip Cc: users@kannel.org Subject: Re: Queue in Store Ok... Please find the details below.: Kannel bearerbox version `1.4.4'. Build `Feb 10 2017 19:38:03', compiler `4.4.7 20120313 (Red Hat 4.4.7-17)'. System Linux, release 2.6.32-696.1.1.el6.x86_64, version #1 SMP Tue Apr 11 17:13:24 UTC 2017, machine x86_64. Hostname CentOS-67-64-minimal, .Libxml version 2.7.6. Using OpenSSL 1.0.1e-fips 11 Feb 2013. Compiled with MySQL 5.7.17, using MySQL 5.7.18. Using native malloc. On Wed, Dec 13, 2017 at 4:28 PM, info.ubichip <info.ubic...@free.fr> wrote: Hello, In order to give a chance to the community to answer you, please provide at least a description of what you are using (kannel versio, opensmppbox, databse...etc). Regards De : users [mailto:users-boun...@kannel.org] De la part de Robin C Envoyé : mercredi 13 décembre 2017 06:38 À : users@kannel.org Objet : Queue in Store Dear Team, When we send large volume of sms the queue is building up in the store. And it process very slowly. Only getting 16 tps. Can anyone help me to solve this issue.? -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. 7 Switch off as you go |q Recycle always | P Print only if absolutely necessary -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. 7 Switch off as you go |q Recycle always | P Print only if absolutely necessary -- Deus seja Louvado eternamente Amem !! (o_Rodrigo Ferreira Santos //\RapidoSMS o seu Site de SMS V_/_ rodrigofsan...@gmail.com Linux user number 372852 <https://www.linuxcounter.net/cert/372852.png>
Re: Queue in Store
As far as you have a working kannel environment you should check for latency, retransmission, retries... A tcpdump would help you identify the source of issue. BR Vangelis Sent from my BlackBerry 10 smartphone. From: Robin C Sent: Τετάρτη, 13 Δεκεμβρίου 2017 - 13:01 To: info.ubichip Cc: users@kannel.org Subject: Re: Queue in Store Ok... Please find the details below.: Kannel bearerbox version `1.4.4'. Build `Feb 10 2017 19:38:03', compiler `4.4.7 20120313 (Red Hat 4.4.7-17)'. System Linux, release 2.6.32-696.1.1.el6.x86_64, version #1 SMP Tue Apr 11 17:13:24 UTC 2017, machine x86_64. Hostname CentOS-67-64-minimal, .Libxml version 2.7.6. Using OpenSSL 1.0.1e-fips 11 Feb 2013. Compiled with MySQL 5.7.17, using MySQL 5.7.18. Using native malloc. On Wed, Dec 13, 2017 at 4:28 PM, info.ubichip <info.ubic...@free.fr<mailto:info.ubic...@free.fr>> wrote: Hello, In order to give a chance to the community to answer you, please provide at least a description of what you are using (kannel versio, opensmppbox, databse...etc). Regards De : users [mailto:users-boun...@kannel.org<mailto:users-boun...@kannel.org>] De la part de Robin C Envoyé : mercredi 13 décembre 2017 06:38 À : users@kannel.org<mailto:users@kannel.org> Objet : Queue in Store Dear Team, When we send large volume of sms the queue is building up in the store. And it process very slowly. Only getting 16 tps. Can anyone help me to solve this issue.? -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. 7 Switch off as you go |q Recycle always | P Print only if absolutely necessary -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. 7 Switch off as you go |q Recycle always | P Print only if absolutely necessary
Re: Queue in Store
Ok... Please find the details below.: Kannel bearerbox version `1.4.4'. Build `Feb 10 2017 19:38:03', compiler `4.4.7 20120313 (Red Hat 4.4.7-17)'. System Linux, release 2.6.32-696.1.1.el6.x86_64, version #1 SMP Tue Apr 11 17:13:24 UTC 2017, machine x86_64. Hostname CentOS-67-64-minimal, .Libxml version 2.7.6. Using OpenSSL 1.0.1e-fips 11 Feb 2013. Compiled with MySQL 5.7.17, using MySQL 5.7.18. Using native malloc. On Wed, Dec 13, 2017 at 4:28 PM, info.ubichip <info.ubic...@free.fr> wrote: > Hello, > > > > In order to give a chance to the community to answer you, please provide > at least a description of what you are using (kannel versio, opensmppbox, > databse...etc). > > > > Regards > > > > > > *De :* users [mailto:users-boun...@kannel.org] *De la part de* Robin C > *Envoyé :* mercredi 13 décembre 2017 06:38 > *À :* users@kannel.org > *Objet :* Queue in Store > > > > > Dear Team, > > > > When we send large volume of sms the queue is building up in the store. > And it process very slowly. Only getting 16 tps. Can anyone help me to > solve this issue.? > > -- > > > > > > Disclaimer: This message contains confidential information and is > intended only for the individual named. If you are not the named addressee > you should not disseminate, distribute or copy this e-mail. Please notify > the sender immediately by e-mail if you have received this e-mail by > mistake and delete this e-mail from your system. E-mail transmission > cannot be guaranteed to be secure or error-free as information could be > intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. The sender therefore does not accept liability for any > errors or omissions in the contents of this message, which arise as a > result of e-mail transmission. If verification is required please request > a hard-copy version. > > *7* Switch off as you go |*q *Recycle always | P Print only if absolutely > necessary > -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. *7* Switch off as you go |*q *Recycle always | P Print only if absolutely necessary
RE: Queue in Store
Hello, In order to give a chance to the community to answer you, please provide at least a description of what you are using (kannel versio, opensmppbox, databse...etc). Regards De : users [mailto:users-boun...@kannel.org] De la part de Robin C Envoyé : mercredi 13 décembre 2017 06:38 À : users@kannel.org Objet : Queue in Store Dear Team, When we send large volume of sms the queue is building up in the store. And it process very slowly. Only getting 16 tps. Can anyone help me to solve this issue.? -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. 7 Switch off as you go |q Recycle always | P Print only if absolutely necessary
Queue in Store
Dear Team, When we send large volume of sms the queue is building up in the store. And it process very slowly. Only getting 16 tps. Can anyone help me to solve this issue.? -- Disclaimer: This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. *7* Switch off as you go |*q *Recycle always | P Print only if absolutely necessary
Re: Removing an erroring message stuck in queue / file store
Nikos, No, never tried ufs. Anyway, the performance issues with ext3 in particular were supposed to happen on store _startup_ afaik. Regards, Alejandro 2008/12/8 Nikos Balkanas [EMAIL PROTECTED] Dear Alej, I just did a couple of runs on Solaris w/ ufs logging/no logging. 1000 threads x 1000 msgs each (100 MTs) with DLRs over ppg (fake smsc). Time quoted is time to get PAP response from ppg. logging: 17':35. 100 SMS + 100 DLRs sent. no-logging: 11':53. 22000 SMS + 22000 DLRs sent. It took over 1hr for the queue to empty. Overall seems logging is synchronous, no-logging is asynchronous and takes much longer to complete. Did you have similar experiences with no-logging? How was the queue at the end of the run? Thanx, Nikos - Original Message - *From:* Nikos Balkanas [EMAIL PROTECTED] *To:* Alejandro Guerrieri [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 2:52 AM *Subject:* Re: Removing an erroring message stuck in queue / file store It goes without saying. In a production system with large queues, every little detail must be addressed. A suitable indexed filesystem (i.e. ReiserFS) should be used for the spool. Cheers, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 2:44 AM *Subject:* Re: Removing an erroring message stuck in queue / file store Yes, most of the time. The only exception seems to be if you're using some journaled filesystems (e.g. ext3), and your spool holds a lot of messages. The system would take some time to come up if there's a lot of messages enqueued when starting. We're talking about tenths of thousands messages on the queue, not just a few hundreds. Using xfs or ext2 avoids that. Other than that, at least in my own experience the spool folder is quite easier to maintain imho. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 2:18 AM *Subject:* Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Tony Kirkham [EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Wednesday, December 03, 2008 7:14 PM *Subject:* Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr
Re: Removing an erroring message stuck in queue / file store
Dear Alej, I just did a couple of runs on Solaris w/ ufs logging/no logging. 1000 threads x 1000 msgs each (100 MTs) with DLRs over ppg (fake smsc). Time quoted is time to get PAP response from ppg. logging: 17':35. 100 SMS + 100 DLRs sent. no-logging: 11':53. 22000 SMS + 22000 DLRs sent. It took over 1hr for the queue to empty. Overall seems logging is synchronous, no-logging is asynchronous and takes much longer to complete. Did you have similar experiences with no-logging? How was the queue at the end of the run? Thanx, Nikos - Original Message - From: Nikos Balkanas To: Alejandro Guerrieri Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:52 AM Subject: Re: Removing an erroring message stuck in queue / file store It goes without saying. In a production system with large queues, every little detail must be addressed. A suitable indexed filesystem (i.e. ReiserFS) should be used for the spool. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:44 AM Subject: Re: Removing an erroring message stuck in queue / file store Yes, most of the time. The only exception seems to be if you're using some journaled filesystems (e.g. ext3), and your spool holds a lot of messages. The system would take some time to come up if there's a lot of messages enqueued when starting. We're talking about tenths of thousands messages on the queue, not just a few hundreds. Using xfs or ext2 avoids that. Other than that, at least in my own experience the spool folder is quite easier to maintain imho. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:18 AM Subject: Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: users@kannel.org Sent: Wednesday, December 03, 2008 7:14 PM Subject: Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture
Re: Removing an erroring message stuck in queue / file store
Thanks, guys. This is a lot of great information. I will be trying this and I will post my results. One other question though, are there any advantages to the file store over the spool store? Before this conversation, I had not understood what the spool store was. I had only heard the term mentioned in passing. I have been using older documentation (Kannel 1.4.1 User's Guide) that lists a parameter to the core configuration of store-file and that is what is currently in my config file and running on my system. The actual Kannel code I am using is a CVS pull of Kannel from June of this year. Now that I examine the latest, sort of, documentation (cvs-20081125) I no longer see the store-file parameter. Instead I see store-type and store-location. I am assuming, now, that the spool storage type is a recent addition to Kannel and was added because it worked more effeciently than trying to rewrite a single file containing many messages. Is this assumption correct? Now that I know about the spool storage type I think I will be switching to it. I can't currently see any advantage the file store has over the spool type and I see many disadvantages, the impetus of this entire conversation, to name one. I am now wondering if my June Kannel code has these parameters in it or if I will have to get a more recent code base and recompile. Any thoughts? Thanks again, -Tony 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Definitely. I have a test kannel, hooked to fakesmsc, and I would always try out a copy of the fixed store file before commiting it. There is no header from what I can see, however there is a footer that has a list of all message ids. This is a bit more complex, not much though. I presumed that there was no CRC or indexing. It seems that the message has to be removed both from the body and the footer. Tony if you are watching this, a simple cat at the end won't do it, but something similar should. The footer has to be removed and saved in a different file. Then do the cat and insert the saved footer just before the new footer, so that the right order is preserved. Of course it goes without saying that the procedure has to be tested first on a test environment with fakesmsc. I could do it, except that i don't have any real messages. BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 4:20 AM *Subject:* Re: Removing an erroring message stuck in queue / file store Hmm, I'm not 100% sure you can append messages at the end of the store and get away with it... I'd first try it on a staging server. You can surely merge new messages on a spool by copying (you need to restart kannel for it to notice the new messages, though) but I don't know if is there any header or footer information on the store files that would break the format if appended together. I'm not saying it's not possible, just that I've never tried it and since I'm not _that_ fluent on the file store internals I cannot tell for sure. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] BTW, Tony. It doesn't seem that you will have significant downtime. If Alej is right you can just stop kannel momentarily to grab the store files and start it again. Once the problematic store files are fixed you can just stop kannel momentarily and cat kannel.store.fixed kannel.store. Do the same for kannel.store.back and restart kannel. Just don't change from storefile to spool during the process. So you don't loose any messages and downtime can be minimal. Cool right? - Original Message - *From:* Nikos Balkanas [EMAIL PROTECTED] *To:* Alejandro Guerrieri [EMAIL PROTECTED] ; Tony Kirkham[EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Thursday, December 04, 2008 2:35 AM *Subject:* Re: Removing an erroring message stuck in queue / file store Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users
Re: Removing an erroring message stuck in queue / file store
Tony, Just a quick update on store file internals. I've been checking the code a little and the dumping of messages doesn't occur on each message change, but on a separate thread at given time intervals (something like every 30 seconds I think). That being said, I personally prefer the spool folder over the store file. It's more like a Maildir over Mailbox approach. The spool has a little more I/O stress (each message is updated/deleted at the moment of the change, compared to dumping the whole file at given intervals) but on the other side there's a lot less of locking going on. The only issue to keep in mind is the performance on journaling filesystems, but that only shows up on heavily loaded scenarios (maybe handling hundreds of thousands messages a day or more) so if you're not dealing with high traffic it won't be a problem at all. If you're planning your setup you can consider having a dedicated ext2 or xfs partition for this anyway. The spool folder was introduced time ago (more than a year I think) but it was after 1.4.1 was out, so you need a CVS version to take advantage of this. Anyway, I think a new stable release it's not far away. CVS code is as stable as 1.4.1 imho, I've been using it on production systems for years without a single problem. Regards, Alejandro Guerrieri On Thu, Dec 4, 2008 at 1:39 PM, Tony Kirkham [EMAIL PROTECTED] wrote: Thanks, guys. This is a lot of great information. I will be trying this and I will post my results. One other question though, are there any advantages to the file store over the spool store? Before this conversation, I had not understood what the spool store was. I had only heard the term mentioned in passing. I have been using older documentation (Kannel 1.4.1 User's Guide) that lists a parameter to the core configuration of store-file and that is what is currently in my config file and running on my system. The actual Kannel code I am using is a CVS pull of Kannel from June of this year. Now that I examine the latest, sort of, documentation (cvs-20081125) I no longer see the store-file parameter. Instead I see store-type and store-location. I am assuming, now, that the spool storage type is a recent addition to Kannel and was added because it worked more effeciently than trying to rewrite a single file containing many messages. Is this assumption correct? Now that I know about the spool storage type I think I will be switching to it. I can't currently see any advantage the file store has over the spool type and I see many disadvantages, the impetus of this entire conversation, to name one. I am now wondering if my June Kannel code has these parameters in it or if I will have to get a more recent code base and recompile. Any thoughts? Thanks again, -Tony 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Definitely. I have a test kannel, hooked to fakesmsc, and I would always try out a copy of the fixed store file before commiting it. There is no header from what I can see, however there is a footer that has a list of all message ids. This is a bit more complex, not much though. I presumed that there was no CRC or indexing. It seems that the message has to be removed both from the body and the footer. Tony if you are watching this, a simple cat at the end won't do it, but something similar should. The footer has to be removed and saved in a different file. Then do the cat and insert the saved footer just before the new footer, so that the right order is preserved. Of course it goes without saying that the procedure has to be tested first on a test environment with fakesmsc. I could do it, except that i don't have any real messages. BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 4:20 AM *Subject:* Re: Removing an erroring message stuck in queue / file store Hmm, I'm not 100% sure you can append messages at the end of the store and get away with it... I'd first try it on a staging server. You can surely merge new messages on a spool by copying (you need to restart kannel for it to notice the new messages, though) but I don't know if is there any header or footer information on the store files that would break the format if appended together. I'm not saying it's not possible, just that I've never tried it and since I'm not _that_ fluent on the file store internals I cannot tell for sure. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] BTW, Tony. It doesn't seem that you will have significant downtime. If Alej is right you can just stop kannel momentarily to grab the store files and start it again. Once the problematic store files are fixed you can just stop kannel momentarily and cat kannel.store.fixed kannel.store. Do the same for kannel.store.back and restart kannel. Just
Re: Removing an erroring message stuck in queue / file store
Interesting, Let's throw in a few numbers. Imagine that you have a large queue, storefile 100 MB. Now imagine that every 30 you get 100 MB I/O to maintain that. First of all, if you crash you may loose up to the last 30 of messsages. On the other hand, if in those 30 you send ~20,000 sms x 2250 B/msg = 43 MB, you even get less I/O with the spool directory! Using these numbers you can do your own calculations for your own system. The whole advantage to the spool is that the FS will provide indexing to the messages whereas it is not supported in the store-file. Like a database table without indeces. It is true that kannel doesn't need to match specific messages, just propably pulls from the top of the store. The I/O is more evenly spread over the period, your spool files are up to the second, and on top of that you can manage it better. Just a bit of care to use a dedicated, appropriate filesystem. Thanks a lot Alej, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: Nikos Balkanas ; users@kannel.org Sent: Thursday, December 04, 2008 6:18 PM Subject: Re: Removing an erroring message stuck in queue / file store Tony, Just a quick update on store file internals. I've been checking the code a little and the dumping of messages doesn't occur on each message change, but on a separate thread at given time intervals (something like every 30 seconds I think). That being said, I personally prefer the spool folder over the store file. It's more like a Maildir over Mailbox approach. The spool has a little more I/O stress (each message is updated/deleted at the moment of the change, compared to dumping the whole file at given intervals) but on the other side there's a lot less of locking going on. The only issue to keep in mind is the performance on journaling filesystems, but that only shows up on heavily loaded scenarios (maybe handling hundreds of thousands messages a day or more) so if you're not dealing with high traffic it won't be a problem at all. If you're planning your setup you can consider having a dedicated ext2 or xfs partition for this anyway. The spool folder was introduced time ago (more than a year I think) but it was after 1.4.1 was out, so you need a CVS version to take advantage of this. Anyway, I think a new stable release it's not far away. CVS code is as stable as 1.4.1 imho, I've been using it on production systems for years without a single problem. Regards, Alejandro Guerrieri On Thu, Dec 4, 2008 at 1:39 PM, Tony Kirkham [EMAIL PROTECTED] wrote: Thanks, guys. This is a lot of great information. I will be trying this and I will post my results. One other question though, are there any advantages to the file store over the spool store? Before this conversation, I had not understood what the spool store was. I had only heard the term mentioned in passing. I have been using older documentation (Kannel 1.4.1 User's Guide) that lists a parameter to the core configuration of store-file and that is what is currently in my config file and running on my system. The actual Kannel code I am using is a CVS pull of Kannel from June of this year. Now that I examine the latest, sort of, documentation (cvs-20081125) I no longer see the store-file parameter. Instead I see store-type and store-location. I am assuming, now, that the spool storage type is a recent addition to Kannel and was added because it worked more effeciently than trying to rewrite a single file containing many messages. Is this assumption correct? Now that I know about the spool storage type I think I will be switching to it. I can't currently see any advantage the file store has over the spool type and I see many disadvantages, the impetus of this entire conversation, to name one. I am now wondering if my June Kannel code has these parameters in it or if I will have to get a more recent code base and recompile. Any thoughts? Thanks again, -Tony 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Definitely. I have a test kannel, hooked to fakesmsc, and I would always try out a copy of the fixed store file before commiting it. There is no header from what I can see, however there is a footer that has a list of all message ids. This is a bit more complex, not much though. I presumed that there was no CRC or indexing. It seems that the message has to be removed both from the body and the footer. Tony if you are watching this, a simple cat at the end won't do it, but something similar should. The footer has to be removed and saved in a different file. Then do the cat and insert the saved footer just before the new footer, so that the right order is preserved. Of course it goes without saying that the procedure has to be tested first on a test environment with fakesmsc. I could do it, except that i don't have any real messages. BR
Re: Removing an erroring message stuck in queue / file store
Sorry, a correction in the sizing. Average message should be 400 B/msg not 2250 B/msg. Therefore in the example, I/O for spool would be 7,6 MB only! I imagine that's one of the reasons that sendmail and other mail programs use spools! BR, Nikos - Original Message - From: Nikos Balkanas To: Alejandro Guerrieri ; Tony Kirkham Cc: users@kannel.org Sent: Thursday, December 04, 2008 6:56 PM Subject: Re: Removing an erroring message stuck in queue / file store Interesting, Let's throw in a few numbers. Imagine that you have a large queue, storefile 100 MB. Now imagine that every 30 you get 100 MB I/O to maintain that. First of all, if you crash you may loose up to the last 30 of messsages. On the other hand, if in those 30 you send ~20,000 sms x 2250 B/msg = 43 MB, you even get less I/O with the spool directory! Using these numbers you can do your own calculations for your own system. The whole advantage to the spool is that the FS will provide indexing to the messages whereas it is not supported in the store-file. Like a database table without indeces. It is true that kannel doesn't need to match specific messages, just propably pulls from the top of the store. The I/O is more evenly spread over the period, your spool files are up to the second, and on top of that you can manage it better. Just a bit of care to use a dedicated, appropriate filesystem. Thanks a lot Alej, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: Nikos Balkanas ; users@kannel.org Sent: Thursday, December 04, 2008 6:18 PM Subject: Re: Removing an erroring message stuck in queue / file store Tony, Just a quick update on store file internals. I've been checking the code a little and the dumping of messages doesn't occur on each message change, but on a separate thread at given time intervals (something like every 30 seconds I think). That being said, I personally prefer the spool folder over the store file. It's more like a Maildir over Mailbox approach. The spool has a little more I/O stress (each message is updated/deleted at the moment of the change, compared to dumping the whole file at given intervals) but on the other side there's a lot less of locking going on. The only issue to keep in mind is the performance on journaling filesystems, but that only shows up on heavily loaded scenarios (maybe handling hundreds of thousands messages a day or more) so if you're not dealing with high traffic it won't be a problem at all. If you're planning your setup you can consider having a dedicated ext2 or xfs partition for this anyway. The spool folder was introduced time ago (more than a year I think) but it was after 1.4.1 was out, so you need a CVS version to take advantage of this. Anyway, I think a new stable release it's not far away. CVS code is as stable as 1.4.1 imho, I've been using it on production systems for years without a single problem. Regards, Alejandro Guerrieri On Thu, Dec 4, 2008 at 1:39 PM, Tony Kirkham [EMAIL PROTECTED] wrote: Thanks, guys. This is a lot of great information. I will be trying this and I will post my results. One other question though, are there any advantages to the file store over the spool store? Before this conversation, I had not understood what the spool store was. I had only heard the term mentioned in passing. I have been using older documentation (Kannel 1.4.1 User's Guide) that lists a parameter to the core configuration of store-file and that is what is currently in my config file and running on my system. The actual Kannel code I am using is a CVS pull of Kannel from June of this year. Now that I examine the latest, sort of, documentation (cvs-20081125) I no longer see the store-file parameter. Instead I see store-type and store-location. I am assuming, now, that the spool storage type is a recent addition to Kannel and was added because it worked more effeciently than trying to rewrite a single file containing many messages. Is this assumption correct? Now that I know about the spool storage type I think I will be switching to it. I can't currently see any advantage the file store has over the spool type and I see many disadvantages, the impetus of this entire conversation, to name one. I am now wondering if my June Kannel code has these parameters in it or if I will have to get a more recent code base and recompile. Any thoughts? Thanks again, -Tony 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Definitely. I have a test kannel, hooked to fakesmsc, and I would always try out a copy of the fixed store file before commiting it. There is no header from what I can see, however there is a footer that has a list of all message ids. This is a bit more complex, not much though. I presumed that there was no CRC or indexing. It seems
Re: Removing an erroring message stuck in queue / file store
How? By hand, in my favorite editor? I have tried this, with bad results. There are special characters, at least vi displays them that way, that I can't interpret so I do not know where, exactly, one message stops and the next starts. Is there a document that describes the store file in detail? -Tony On Tue, Dec 2, 2008 at 4:08 PM, Alejandro Guerrieri [EMAIL PROTECTED] wrote: And do it without kannel running, of course ;) If you're using the spool store file, you can remove the individual message (assuming that you can find out what's the message causing the error). Regards, Alejandro On Tue, Dec 2, 2008 at 5:49 PM, info.ubichip [EMAIL PROTECTED] wrote: Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards -- *From:* Tony Kirkham [mailto:[EMAIL PROTECTED] *Sent:* lundi 1 décembre 2008 21:32 *To:* users@kannel.org *Subject:* Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
Re: Removing an erroring message stuck in queue / file store
Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue. * On the particular case of the file store, a backup file is kept at all times in case the main file is deleted or corrupted. So, from the above, it's clear why you need to stop kannel before deleting the file(s). You can stop kannel, delete the store and store.bak files, or remove the whole spool tree and start again. It will remove ALL your pending messages. If you can figure out which message is causing the problem, you can remove it from the spool (if you're using the spool store) as well. With the file store, you'd need to figure out from the binary, which is near to impossible without some specially crafted tools imho. Regards, Alejandro On Wed, Dec 3, 2008 at 1:15 PM, Tony Kirkham [EMAIL PROTECTED] wrote: How? By hand, in my favorite editor? I have tried this, with bad results. There are special characters, at least vi displays them that way, that I can't interpret so I do not know where, exactly, one message stops and the next starts. Is there a document that describes the store file in detail? -Tony On Tue, Dec 2, 2008 at 4:08 PM, Alejandro Guerrieri [EMAIL PROTECTED] wrote: And do it without kannel running, of course ;) If you're using the spool store file, you can remove the individual message (assuming that you can find out what's the message causing the error). Regards, Alejandro On Tue, Dec 2, 2008 at 5:49 PM, info.ubichip [EMAIL PROTECTED]wrote: Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards -- *From:* Tony Kirkham [mailto:[EMAIL PROTECTED] *Sent:* lundi 1 décembre 2008 21:32 *To:* users@kannel.org *Subject:* Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
Re: Removing an erroring message stuck in queue / file store
Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: users@kannel.org Sent: Wednesday, December 03, 2008 7:14 PM Subject: Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue. * On the particular case of the file store, a backup file is kept at all times in case the main file is deleted or corrupted. So, from the above, it's clear why you need to stop kannel before deleting the file(s). You can stop kannel, delete the store and store.bak files, or remove the whole spool tree and start again. It will remove ALL your pending messages. If you can figure out which message is causing the problem, you can remove it from the spool (if you're using the spool store) as well. With the file store, you'd need to figure out from the binary, which is near to impossible without some specially crafted tools imho. Regards, Alejandro On Wed, Dec 3, 2008 at 1:15 PM, Tony Kirkham [EMAIL PROTECTED] wrote: How? By hand, in my favorite editor? I have tried this, with bad results. There are special characters, at least vi displays them that way, that I can't interpret so I do not know where, exactly, one message stops and the next starts. Is there a document that describes the store file in detail? -Tony On Tue, Dec 2, 2008 at 4:08 PM, Alejandro Guerrieri [EMAIL PROTECTED] wrote: And do it without kannel running, of course ;) If you're using the spool store file, you can remove the individual message (assuming that you can find out what's the message causing the error). Regards, Alejandro On Tue, Dec 2, 2008 at 5:49 PM, info.ubichip [EMAIL PROTECTED] wrote: Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards From: Tony Kirkham [mailto:[EMAIL PROTECTED] Sent: lundi 1 dιcembre 2008 21:32 To: users@kannel.org Subject: Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
Re: Removing an erroring message stuck in queue / file store
The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Tony Kirkham [EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Wednesday, December 03, 2008 7:14 PM *Subject:* Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue. * On the particular case of the file store, a backup file is kept at all times in case the main file is deleted or corrupted. So, from the above, it's clear why you need to stop kannel before deleting the file(s). You can stop kannel, delete the store and store.bak files, or remove the whole spool tree and start again. It will remove ALL your pending messages. If you can figure out which message is causing the problem, you can remove it from the spool (if you're using the spool store) as well. With the file store, you'd need to figure out from the binary, which is near to impossible without some specially crafted tools imho. Regards, Alejandro On Wed, Dec 3, 2008 at 1:15 PM, Tony Kirkham [EMAIL PROTECTED]wrote: How? By hand, in my favorite editor? I have tried this, with bad results. There are special characters, at least vi displays them that way, that I can't interpret so I do not know where, exactly, one message stops and the next starts. Is there a document that describes the store file in detail? -Tony On Tue, Dec 2, 2008 at 4:08 PM, Alejandro Guerrieri [EMAIL PROTECTED] wrote: And do it without kannel running, of course ;) If you're using the spool store file, you can remove the individual message (assuming that you can find out what's the message causing the error). Regards, Alejandro On Tue, Dec 2, 2008 at 5:49 PM, info.ubichip [EMAIL PROTECTED]wrote: Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards -- *From:* Tony Kirkham [mailto:[EMAIL PROTECTED] *Sent:* lundi 1 dιcembre 2008 21:32 *To:* users@kannel.org *Subject:* Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages
Re: Removing an erroring message stuck in queue / file store
Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:18 AM Subject: Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: users@kannel.org Sent: Wednesday, December 03, 2008 7:14 PM Subject: Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue. * On the particular case of the file store, a backup file is kept at all times in case the main file is deleted or corrupted. So, from the above, it's clear why you need to stop kannel before deleting the file(s). You can stop kannel, delete the store and store.bak files, or remove the whole spool tree and start again. It will remove ALL your pending messages. If you can figure out which message is causing the problem, you can remove it from the spool (if you're using the spool store) as well. With the file store, you'd need to figure out from the binary, which is near to impossible without some specially crafted tools imho. Regards, Alejandro On Wed, Dec 3, 2008 at 1:15 PM, Tony Kirkham [EMAIL PROTECTED] wrote: How? By hand, in my favorite editor
Re: Removing an erroring message stuck in queue / file store
BTW, Tony. It doesn't seem that you will have significant downtime. If Alej is right you can just stop kannel momentarily to grab the store files and start it again. Once the problematic store files are fixed you can just stop kannel momentarily and cat kannel.store.fixed kannel.store. Do the same for kannel.store.back and restart kannel. Just don't change from storefile to spool during the process. So you don't loose any messages and downtime can be minimal. Cool right? - Original Message - From: Nikos Balkanas To: Alejandro Guerrieri ; Tony Kirkham Cc: users@kannel.org Sent: Thursday, December 04, 2008 2:35 AM Subject: Re: Removing an erroring message stuck in queue / file store Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:18 AM Subject: Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: users@kannel.org Sent: Wednesday, December 03, 2008 7:14 PM Subject: Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue
Re: Removing an erroring message stuck in queue / file store
Yes, most of the time. The only exception seems to be if you're using some journaled filesystems (e.g. ext3), and your spool holds a lot of messages. The system would take some time to come up if there's a lot of messages enqueued when starting. We're talking about tenths of thousands messages on the queue, not just a few hundreds. Using xfs or ext2 avoids that. Other than that, at least in my own experience the spool folder is quite easier to maintain imho. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 2:18 AM *Subject:* Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Tony Kirkham [EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Wednesday, December 03, 2008 7:14 PM *Subject:* Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file, or remove an entire file from the spool tree. * If kannel crashes or is shutdown with messages pending, the messages are kept on the file/spool queue. During a shutdown, the in-memory queue is dumped to file again afaik. * When kannel starts, it first load the messages from the file/spool queue. * On the particular case of the file store, a backup file is kept at all times in case the main file is deleted or corrupted. So, from the above, it's clear why you need to stop kannel before deleting the file(s). You can stop kannel, delete the store and store.bak
Re: Removing an erroring message stuck in queue / file store
It goes without saying. In a production system with large queues, every little detail must be addressed. A suitable indexed filesystem (i.e. ReiserFS) should be used for the spool. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:44 AM Subject: Re: Removing an erroring message stuck in queue / file store Yes, most of the time. The only exception seems to be if you're using some journaled filesystems (e.g. ext3), and your spool holds a lot of messages. The system would take some time to come up if there's a lot of messages enqueued when starting. We're talking about tenths of thousands messages on the queue, not just a few hundreds. Using xfs or ext2 avoids that. Other than that, at least in my own experience the spool folder is quite easier to maintain imho. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:18 AM Subject: Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - From: Alejandro Guerrieri To: Tony Kirkham Cc: users@kannel.org Sent: Wednesday, December 03, 2008 7:14 PM Subject: Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid is used to keep track of messages on both places. * When kannel ends processing the message, it removes it from the in-memory queue and deletes the message from disk. Depending on the store type being used, this means removing it from the store file and rewriting the file
Re: Removing an erroring message stuck in queue / file store
Hmm, I'm not 100% sure you can append messages at the end of the store and get away with it... I'd first try it on a staging server. You can surely merge new messages on a spool by copying (you need to restart kannel for it to notice the new messages, though) but I don't know if is there any header or footer information on the store files that would break the format if appended together. I'm not saying it's not possible, just that I've never tried it and since I'm not _that_ fluent on the file store internals I cannot tell for sure. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] BTW, Tony. It doesn't seem that you will have significant downtime. If Alej is right you can just stop kannel momentarily to grab the store files and start it again. Once the problematic store files are fixed you can just stop kannel momentarily and cat kannel.store.fixed kannel.store. Do the same for kannel.store.back and restart kannel. Just don't change from storefile to spool during the process. So you don't loose any messages and downtime can be minimal. Cool right? - Original Message - *From:* Nikos Balkanas [EMAIL PROTECTED] *To:* Alejandro Guerrieri [EMAIL PROTECTED] ; Tony Kirkham[EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Thursday, December 04, 2008 2:35 AM *Subject:* Re: Removing an erroring message stuck in queue / file store Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Nikos Balkanas [EMAIL PROTECTED] *Cc:* Tony Kirkham [EMAIL PROTECTED] ; users@kannel.org *Sent:* Thursday, December 04, 2008 2:18 AM *Subject:* Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel, and repeat for both files. Cheers, Nikos - Original Message - *From:* Alejandro Guerrieri [EMAIL PROTECTED] *To:* Tony Kirkham [EMAIL PROTECTED] *Cc:* users@kannel.org *Sent:* Wednesday, December 03, 2008 7:14 PM *Subject:* Re: Removing an erroring message stuck in queue / file store Ah, you're using the file store, not spool maybe? AFAIK, there's no document on the storage format. It's mainly a dump from the Msg structure, but the Octstr objects can have nulls in the middle, so it's not as easy to parse and, it doesn't have a fixed length and (if you're still willing to venture at it) you'll need an hex editor if you don't want to mess the whole file (the nulls in the middle of the file will probably render any text editor unusable). If you use the spool store, each message is stored on an independent file on a directory tree. Removing the file removes the message. The only problem is that kannel only uses the files as a backup mechanism and it doesn't refresh it during runtime. To clarify: * When a message arrives (or is enqueued to be sent), kannel adds it to an in-memory queue and stores a copy on disk. An internal uuid
Re: Removing an erroring message stuck in queue / file store
Definitely. I have a test kannel, hooked to fakesmsc, and I would always try out a copy of the fixed store file before commiting it. There is no header from what I can see, however there is a footer that has a list of all message ids. This is a bit more complex, not much though. I presumed that there was no CRC or indexing. It seems that the message has to be removed both from the body and the footer. Tony if you are watching this, a simple cat at the end won't do it, but something similar should. The footer has to be removed and saved in a different file. Then do the cat and insert the saved footer just before the new footer, so that the right order is preserved. Of course it goes without saying that the procedure has to be tested first on a test environment with fakesmsc. I could do it, except that i don't have any real messages. BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 4:20 AM Subject: Re: Removing an erroring message stuck in queue / file store Hmm, I'm not 100% sure you can append messages at the end of the store and get away with it... I'd first try it on a staging server. You can surely merge new messages on a spool by copying (you need to restart kannel for it to notice the new messages, though) but I don't know if is there any header or footer information on the store files that would break the format if appended together. I'm not saying it's not possible, just that I've never tried it and since I'm not _that_ fluent on the file store internals I cannot tell for sure. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] BTW, Tony. It doesn't seem that you will have significant downtime. If Alej is right you can just stop kannel momentarily to grab the store files and start it again. Once the problematic store files are fixed you can just stop kannel momentarily and cat kannel.store.fixed kannel.store. Do the same for kannel.store.back and restart kannel. Just don't change from storefile to spool during the process. So you don't loose any messages and downtime can be minimal. Cool right? - Original Message - From: Nikos Balkanas To: Alejandro Guerrieri ; Tony Kirkham Cc: users@kannel.org Sent: Thursday, December 04, 2008 2:35 AM Subject: Re: Removing an erroring message stuck in queue / file store Thanks for pointing that out. Still it is easy to preview in vim -b (not hex) and can identify *any* of the fields Sender, recipient, text, messageID, so if anyone is missing there are a lot of matches in ASCII. The binary text is easy and straightforward. One can reasonably find the borders (patterns) of the previous and next message (presumably intact) and delete the in between. Tony, if the file is small enough to send through mail (zipped) and don't feel like trying it yourself, why don't you send it to me along with the message details to see what I can do about it. Alej from your words it seems that a spool directory is more efficient than a storefile. If anytime kannel sends a message has to delete and rewrite almost the whole store file, it is a big waste. Isn't so? BR, Nikos - Original Message - From: Alejandro Guerrieri To: Nikos Balkanas Cc: Tony Kirkham ; users@kannel.org Sent: Thursday, December 04, 2008 2:18 AM Subject: Re: Removing an erroring message stuck in queue / file store The 20 nulls are probably a lot of empty/unused fields, that doesn't necessary means that they'll be always empty... so don't assume that there should be that much. It's not a separator, just a pattern in your particular message queue set. When you edit, you have to make sure that you're actually cutting on the right places. Alas, if the message is in fact corrupted maybe it's not complete, so YMMV. Using the spool file is way easier, of course. Again, if you have a broken message (why is that it's another question) it's just a matter of determining which one is causing the problem and removing it. Regards, Alejandro 2008/12/3 Nikos Balkanas [EMAIL PROTECTED] Hi, Just out of curiosity I openned a store file and it doesn't seem too difficult. You can edit it easily with vim -b and there are a lot of ASCII fields (sender, recipient, messageid), so you can search for your message easily. Each message seems to be bordered by 20 '\0' (NULLS). Can't miss them. Find offending message and delete by hand (assuming that you know which message is the offending one). This again assumes that there is no CRC or indexing of the messages as discussed by Alejandro. In other words, you can safely remove any message. Make a backup, try it and let us know of the result. Stop kannel
RE: Removing an erroring message stuck in queue / file store
Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards _ From: Tony Kirkham [mailto:[EMAIL PROTECTED] Sent: lundi 1 décembre 2008 21:32 To: users@kannel.org Subject: Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
Re: Removing an erroring message stuck in queue / file store
And do it without kannel running, of course ;) If you're using the spool store file, you can remove the individual message (assuming that you can find out what's the message causing the error). Regards, Alejandro On Tue, Dec 2, 2008 at 5:49 PM, info.ubichip [EMAIL PROTECTED] wrote: Hello, when you trash the .store file, don't forget to remove the .store.bak as well otherwise, it will retry to send all the remaining message. regards -- *From:* Tony Kirkham [mailto:[EMAIL PROTECTED] *Sent:* lundi 1 décembre 2008 21:32 *To:* users@kannel.org *Subject:* Removing an erroring message stuck in queue / file store Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
Removing an erroring message stuck in queue / file store
Is there a way to remove a message from the file store that is erroring continuously allowing the other messages to be sent? If I simply restart Kannel sometimes the other messages get sent and sometimes they do not. Why is that? If they all get sent but the erroring message I can simply replace the file store with another file with a restart of Kannel, but I would like to know if there is a way to remove the message by some other means. Thanks, -Tony
