RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Fri, 2002-11-22 at 04:54, Alan McNatty wrote: Hi Oded, It was included - but as mentioned implied iconv was required not optional. No so - configure will check for it and set HAVE_ICONV only if the header file exists (which to me suggests that the library is installed - I don't know many people who hold on their build systems headers for libraries they don't have installed. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Don't think that's a correct approach. coding anything to latin1 regardless of the SMSC's preferences or the user's preferences just because they user did not specificly enable a feature which should be auto-detected does not sound like a good idea to me. -- Oded Arbel m-Wise mobile solutions __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sun, 2002-11-24 at 05:34, Stipe Tolj wrote: I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary yep, agreed, binary things should never be converted. Ahmm.. may I point to the fact that the patch Alan submitted already does these two last discussed issues: - convert using iconv when possible - warn when failing - allows setting binary to prevent encoding of default data coding The only thing it does not do is to warn if encoding is needed and iconv is not available. this could be easily fixed by adding an #else section to the charset_convert body's #ifdef, that try to do conversion using libxml and outputs an error if it fails. if no one will beat me to it, I can have a patch that does it in a couple of days. -- Oded Arbel m-Wise mobile solutions __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 04:35 Uhr, Alan McNatty wrote: On Fri, 2002-11-22 at 21:04, Andreas Fink wrote: Why gsm_to_latin1 if we know encoding? We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. Well the spec says that its GSM character set. We need to be able to specify via the config what encoding each smsc coonsiders to be it's default - they do vary. Ok. that would be a config variable telling which one to be considered. Currently all incomming are assumed to gsm hence gsm_to_latin1 encoding is performed on all incomming messages from all smsc via smpp driver. However, the SMSC default encoding could easily be ASCII, for example. yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set -> latin1 -> gsm character set as some characters would not pass this way. -- consider -- If iconv is not available or no default is set via config and the data coding is smsc default (0x00) - we assume gsm encoding so the default behaviour is the same as current. Alternatively, if iconv is not available and data coding requires iconv or data coding is set via config to something which would require iconv, we sould error and do no conversion (because we can't). Otherwise - we utilise Oded's work with iconv to do sensible decoding of incomming message. the idea is good but the SMPP driver is the wrong place do to this. the only thing we should do there is to say what is considered the default so we can convert it if needed when the message leaves kannel. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Hi At 12:57 PM 11/24/02 +0100, Andreas Fink wrote: We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. Well the spec says that its GSM character set. actually not, quote from smpp 3.4 specs 5.2.19 data_coding Bits 7 6 5 4 3 2 1 0 Meaning Notes 0 0 0 0 0 0 0 0 SMSC Default Alphabet But no mention of what the default alphabet of the smsc ought tio be. nisan
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set - latin1 - gsm character set as some characters would not pass this way. That's sounds about right.if you also encode from external characater set to GSM or UCS2 in smsbox on the way in (MT), then I'm all for moving all the charset conversions to smsbox. one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..May's Law:The quality of correlation is inversly proportional to the densityof control. (The fewer the data points, the smoother the curves.) __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 03:36 Uhr, Oded Arbel wrote: yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set -> latin1 -> gsm character set as some characters would not pass this way. That's sounds about right. if you also encode from external characater set to GSM or UCS2 in smsbox on the way in (MT), then I'm all for moving all the charset conversions to smsbox. one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through anSMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..Self-confidence isn't everything, its the only thing. __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 04:49 Uhr, Oded Arbel wrote: one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through an SMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? well for MT messages it has to be done in the driver or in bearerbox. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through anSMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? well for MT messages it has to be done in the driver or in bearerbox. That's fine, but if the SMSBox does all the character set encoding, then the driver has no clue as to what was the original encoding. of course it can simply try to encode in all the supported charsets to see what succeeds, but this is hardly an optimal solution. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::.."Only wimps use tape backups; real men put their software on ftp-serversand let the rest of the world mirror it." -- Linus Torvalds __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Fri, 2002-11-22 at 21:04, Andreas Fink wrote: Why gsm_to_latin1 if we know encoding? We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. We need to be able to specify via the config what encoding each smsc coonsiders to be it's default - they do vary. Currently all incomming are assumed to gsm hence gsm_to_latin1 encoding is performed on all incomming messages from all smsc via smpp driver. However, the SMSC default encoding could easily be ASCII, for example. -- consider -- If iconv is not available or no default is set via config and the data coding is smsc default (0x00) - we assume gsm encoding so the default behaviour is the same as current. Alternatively, if iconv is not available and data coding requires iconv or data coding is set via config to something which would require iconv, we sould error and do no conversion (because we can't). Otherwise - we utilise Oded's work with iconv to do sensible decoding of incomming message. Sound reasonable? Cheers, Alan
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary yep, agreed, binary things should never be converted. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Freitag, November 22, 2002, at 03:54 Uhr, Alan McNatty wrote: Hi Oded, It was included - but as mentioned implied iconv was required not optional. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Why gsm_to_latin1 if we know encoding? I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Hi Oded, It was included - but as mentioned implied iconv was required not optional. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Given that even libxml requires iconv for full support - it would be nice to at least 'recommend' iconv when compiling kannel esp considering things may not work as they should in smpp world for some. On Fri, 2002-11-22 at 03:23, Oded Arbel wrote: The original charset_convert patch had a patch for configure. it should have been in the current patch - I'll look into it again. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. There is no spoon. (only annoying little kids) ! -Original Message- From: Stipe Tolj [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 21, 2002 2:10 AM To: Oded Arbel Cc: Andreas Fink; [EMAIL PROTECTED] Subject: Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?! the patch tests on the header files, not on the library itself. BTW, the patch does not work when linking the executables for me, because configure has not added any lib dependecy statements for ld. Ok, I'd propose the following behaviour: configure tests on iconv presence. If available sets #define HAVE_ICONV in config.h and the iconv routines are used directly in the code. If not available use the old behaviour. What about that approach? Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 272555332, DDI: +64 4 9167203, Office: +64 4 4992267
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is works for me(tm). ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Mittwoch, November 20, 2002, at 03:42 Uhr, Stipe Tolj wrote: Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is "works for me(tm)". ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
At 03:55 PM 11/20/02 +0100, you Andreas Fink: On Mittwoch, November 20, 2002, at 03:42 Uhr, Stipe Tolj wrote: Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is works for me(tm). ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? I dont think so.. We ended up using it for a proprietary SMSC we connect to. Libxml's replacement functions seem only to handle character encodings needed for XML. Libiconv is also pretty std on most systems I have worked with. Iconv supports many charsets, so another advantage of using it is we have support for charsets that are not yet required or have not been encountered by the list yet. I vote +1 for rolling this patch. We allready use it and it works for us, and its a real pain to keep our source base up to date when there are these differences. Nisan
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Yep, that wouldbe a problem. though current implementation means that if iconv isn't there, it will simply not encode- it won't break. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? iconv is GNU's character conversion library and it supports every imaginable encoding - unlike libXML who's interface only exposes a very llimited set of character encodings, and its use is also far from being trivial. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..People are flexible enough to make any theory look good for a while. -- Jaron Lanier, in his Karma Vertigo essay
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Mittwoch, November 20, 2002, at 06:52 Uhr, Oded Arbel wrote: My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Yep, that would be a problem. though current implementation means that if iconv isn't there, it will simply not encode- it won't break. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? iconv is GNU's character conversion library and it supports every imaginable encoding - unlike libXML who's interface only exposes a very llimited set of character encodings, and its use is also far from being trivial. I would go for a ./configure --with-iconv where the default is without. So people who need it can use it and others who dont, wont. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I would go for a ./configure --with-iconv where the default is without. So people who need it can use it and others who dont, wont. After the patch, configure will automaticly detect if iconv exist and will use it. if it does not exist, the charset_convert function will still be defined, but with empty body, which behaves the same except that the text will not be encoded. Its possible to strap some "iconv replacement" logic instead of the empty body so that if iconv exists it will be used, otherwise - the replacement logic. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..malpractice, n.:The reason surgeons wear masks.
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I vote +1 for rolling this patch. We allready use it and it works for us, and its a real pain to keep our source base up to date when there are these differences. Sync'ing private cvs trees is not our main intention. Basically we at Wapme do this in some sense too, but we always focus that the officual cvs branch keeps sementically intact. (just to my 2ct in regards to private cvs sync'ing). Ok, if iconv is anyway used in libxml2 (by default?!) then I'd vote +0 for this too and I'd like to hear some more votes from the developers before dropping this in. At least someone may veto to add this, if so, please argument on the vetos. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
the patch tests on the header files, not on the library itself. BTW, the patch does not work when linking the executables for me, because configure has not added any lib dependecy statements for ld. Ok, I'd propose the following behaviour: configure tests on iconv presence. If available sets #define HAVE_ICONV in config.h and the iconv routines are used directly in the code. If not available use the old behaviour. What about that approach? Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
[PATCH] was Re: SMPP: data_coding (deliver_sm)
Hi any reason why this didnt make it to the cvs? Also why not apply it to the data_coding in submit_sm? nisan At 08:42 PM 11/6/02 +1300, Alan McNatty wrote: Here's Oded's patch against current cvs. Is working fine for me ... On Sat, 2002-11-02 at 11:21, Alan McNatty wrote: In progress - am building a patch against cvs for Oded's posts, and re-post for testign etc ... Thanks again Oded - your patches look great. On Sat, 2002-11-02 at 00:04, Stipe Tolj wrote: remeber guys to have a [PATCH] post to the mailing list if you want to have that in the general cvs tree. Just a reminder. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 21-312136, DDI: +64 4 9167203, Office: +64 4 4992267 ... error accessing whit Segmentation fault (core dumped)
Re: [PATCH] was Re: SMPP: data_coding (deliver_sm)
Hi Nisan, I guess waiting for confirmation from others that all is good. Good idea to apply same approach to submit_sm, will have a look at that. Cheers, Alan On Mon, 2002-11-18 at 10:14, Nisan Bloch wrote: Hi any reason why this didnt make it to the cvs? Also why not apply it to the data_coding in submit_sm? nisan At 08:42 PM 11/6/02 +1300, Alan McNatty wrote: Here's Oded's patch against current cvs. Is working fine for me ... On Sat, 2002-11-02 at 11:21, Alan McNatty wrote: In progress - am building a patch against cvs for Oded's posts, and re-post for testign etc ... Thanks again Oded - your patches look great. On Sat, 2002-11-02 at 00:04, Stipe Tolj wrote: remeber guys to have a [PATCH] post to the mailing list if you want to have that in the general cvs tree. Just a reminder. Stipe
Re: SMPP: data_coding (deliver_sm)
Here's Oded's patch against current cvs. Is working fine for me ...
On Sat, 2002-11-02 at 11:21, Alan McNatty wrote:
In progress - am building a patch against cvs for Oded's posts, and
re-post for testign etc ...
Thanks again Oded - your patches look great.
On Sat, 2002-11-02 at 00:04, Stipe Tolj wrote:
remeber guys to have a [PATCH] post to the mailing list if you want to
have that in the general cvs tree.
Just a reminder.
Stipe
[EMAIL PROTECTED]
---
Wapme Systems AG
Vogelsanger Weg 80
40470 Düsseldorf
Tel: +49-211-74845-0
Fax: +49-211-74845-299
E-Mail: [EMAIL PROTECTED]
Internet: http://www.wapme-systems.de
---
wapme.net - wherever you are
--
Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz
Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ
Mob: +64 21-312136, DDI: +64 4 9167203, Office: +64 4 4992267
... error accessing whit
Segmentation fault (core dumped)
? charset.patch
? data_coding-patch
? data_coding.patch
? migrate.sh
? smsc_smpp.c-charset
? smsc_smpp.c-diff
? smsc_smpp.c-patch
? smsc_smpp.c-tcl
? smsc_smpp.patch
? doc/arch/arch.aux
? doc/arch/arch.dvi
? doc/arch/arch.log
? doc/arch/arch.tex
? doc/userguide/userguide.xml-patch
? gw/diff
? gw/smsc_at2.c.diff
Index: config.h.in
===
RCS file: /home/cvs/gateway/config.h.in,v
retrieving revision 1.42
diff -u -r1.42 config.h.in
--- config.h.in 9 Sep 2002 10:49:28 - 1.42
+++ config.h.in 6 Nov 2002 07:35:37 -
@@ -113,6 +113,9 @@
/* Define if you have syslog.h. */
#undef HAVE_SYSLOG_H
+/* Define if you have iconv.h. */
+#undef HAVE_ICONV_H
+
/* Define if you have and want to use the ssl library (-lssl) */
#undef HAVE_LIBSSL
Index: configure
===
RCS file: /home/cvs/gateway/configure,v
retrieving revision 1.105
diff -u -r1.105 configure
--- configure 9 Sep 2002 10:49:28 - 1.105
+++ configure 6 Nov 2002 07:35:38 -
@@ -1781,7 +1781,7 @@
fi
done
-for ac_hdr in pthread.h getopt.h syslog.h
+for ac_hdr in pthread.h getopt.h syslog.h iconv.h
do
ac_safe=`echo $ac_hdr | sed 'y%./+-%__p_%'`
echo $ac_n checking for $ac_hdr... $ac_c 16
Index: configure.in
===
RCS file: /home/cvs/gateway/configure.in,v
retrieving revision 1.107
diff -u -r1.107 configure.in
--- configure.in9 Sep 2002 10:49:28 - 1.107
+++ configure.in6 Nov 2002 07:35:39 -
@@ -95,7 +95,7 @@
AC_HEADER_STDC
AC_CHECK_HEADERS(sys/ioctl.h sys/time.h sys/types.h unistd.h sys/poll.h)
-AC_CHECK_HEADERS(pthread.h getopt.h syslog.h)
+AC_CHECK_HEADERS(pthread.h getopt.h syslog.h iconv.h)
dnl Checks for typedefs, structures, and compiler characteristics.
Index: gw/smsc/smsc_smpp.c
===
RCS file: /home/cvs/gateway/gw/smsc/smsc_smpp.c,v
retrieving revision 1.14
diff -u -r1.14 smsc_smpp.c
--- gw/smsc/smsc_smpp.c 17 Oct 2002 08:28:10 - 1.14
+++ gw/smsc/smsc_smpp.c 6 Nov 2002 07:35:39 -
@@ -102,6 +102,7 @@
time_t throttling_err_time;
int smpp_msg_id_type; /* msg id in C string, hex or decimal */
int autodetect_addr;
+Octstr* alt_charset;
SMSCConn *conn;
} SMPP;
@@ -115,7 +116,8 @@
int alt_dcs, int enquire_link_interval,
int max_pending_submits, int reconnect_delay,
int version, int priority, Octstr *my_number,
- int smpp_msg_id_type, int autodetect_addr)
+ int smpp_msg_id_type, int autodetect_addr,
+Octstr* alt_charset)
{
SMPP *smpp;
@@ -147,6 +149,7 @@
smpp-quitting = 0;
smpp-version = version;
smpp-priority = priority;
+smpp-alt_charset = octstr_duplicate(alt_charset);
smpp-conn = conn;
smpp-throttling_err_time = 0;
smpp-smpp_msg_id_type = smpp_msg_id_type;
@@ -169,6 +172,8 @@
octstr_destroy(smpp-system_type);
octstr_destroy(smpp-address_range);
octstr_destroy(smpp-our_host);
+octstr_destroy(smpp-my_number);
+octstr_destroy(smpp-alt_charset);
gw_free(smpp);
}
}
@@ -223,7 +228,7 @@
}
-static Msg *pdu_to_msg(SMPP_PDU *pdu)
+static Msg *pdu_to_msg(SMPP *smpp, SMPP_PDU *pdu)
{
Msg *msg;
@@ -236,9 +241,44 @@
pdu-u.deliver_sm.destination_addr = NULL;
msg-sms.msgdata = pdu-u.deliver_sm.short_message;
pdu-u.deliver_sm.short_message = NULL;
-charset_gsm_to_latin1(msg-sms.msgdata);
+dcs_to_fields(msg, pdu-u.deliver_sm.data_coding);
+
+switch (pdu-u.deliver_sm.data_coding) {// handle default data coding
+
RE: SMPP: data_coding (deliver_sm)
Hi I would vote +1 for this to be rolled into CVS. We have somethng similar but it is not as general nor as complete. Nisan At 01:24 PM 10/31/02 +0200, Oded Arbel wrote: -Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] ok - my mistake, according to the specs, it should be if ( pdu-u.deliver_sm.data_coding == 0x00 ) charset_gsm_to_latin1(msg-sms.msgdata); yes - I'm aware that this does not solve your problem, but SMPP servers send GSM 7 bit alphabet encoded messages using data_coding 0x00. Yes - this is what I thought. What's your opionion on having an additional config directive stating what the SMSC default alphabet. I don't know how in the world anyone is suppost to know in advance what the SMSC default alphabet is... ie: default_alphabet = 0x01 I hate putting in more configuration paremeters. how about this rewrite of data_coding handling (see attached patch file). a similar version is running on our system for about half a year now, I only now had the incentive to generalize it, but I changed some stuff, to treat this code as untested. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. Let's suppose you have a table with 2^n cups... Wait a second - is n a natural number?
Re: SMPP: data_coding (deliver_sm)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 áéåí çîéùé, 31 áàå÷èåáø 2002, 23:50, Alan McNatty ëúá òì 'RE: SMPP: data_coding (deliver_sm)': Thanks Oded, looks good - can you post your gwlib/charset.c (I'm guessing) you're referencing a charset_convert function which doesn't exist in cvs. cheers, Oh, sorry about that. Unfortunatly I'm not in the office right now so it will take me some time to regenerate it, but I have already submitted that patch to the list and it can be found on the mailing list archive - here's a direct link : http://www.mail-archive.com/devel;kannel.3glab.org/msg04859/charset.patch - -- Oded Arbel m-Wise mobile solutions ::.. -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE9wnpnR2fhD3+UKoQRAvxjAJ9E+2PwqenJhpse6HXbmw+vrwaDuACfVJYT HMHIya30sUQr2HYxggjLLJA= =FLFS -END PGP SIGNATURE-
Re: SMPP: data_coding (deliver_sm)
In progress - am building a patch against cvs for Oded's posts, and re-post for testign etc ... Thanks again Oded - your patches look great. On Sat, 2002-11-02 at 00:04, Stipe Tolj wrote: remeber guys to have a [PATCH] post to the mailing list if you want to have that in the general cvs tree. Just a reminder. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
RE: SMPP: data_coding (deliver_sm)
-Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] this sound like smsc weirdness? Yes.. :-/ actually it does. The null character is apparently optional and can go away easily so that's not a problem. I'm not sure that tryig to detect if it exists before truncating the string is a good idea - what if someone sends binary content that ends with a 0 byte ? The subsequent sms push problem I was having was a user issued involving experimental coding parameters ... Oded, are you suggesting that gsm_to_latin1 conversion take place only when data_coding == 0x01 or when != 0x01 (Am I readin the specs wrong or 0x01 Ascii?). ie: ok - my mistake, according to the specs, it should be if ( pdu-u.deliver_sm.data_coding == 0x00 ) charset_gsm_to_latin1(msg-sms.msgdata); yes - I'm aware that this does not solve your problem, but SMPP servers send GSM 7 bit alphabet encoded messages using data_coding 0x00. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. They're unfriendly, which is fortunate, really. They'd be difficult to like. -- Avon
RE: SMPP: data_coding (deliver_sm)
On Thu, 2002-10-31 at 22:18, Oded Arbel wrote: Oded, are you suggesting that gsm_to_latin1 conversion take place only when data_coding == 0x01 or when != 0x01 (Am I readin the specs wrong or 0x01 Ascii?). ie: ok - my mistake, according to the specs, it should be if ( pdu-u.deliver_sm.data_coding == 0x00 ) charset_gsm_to_latin1(msg-sms.msgdata); yes - I'm aware that this does not solve your problem, but SMPP servers send GSM 7 bit alphabet encoded messages using data_coding 0x00. Yes - this is what I thought. What's your opionion on having an additional config directive stating what the SMSC default alphabet. I don't know how in the world anyone is suppost to know in advance what the SMSC default alphabet is... ie: default_alphabet = 0x01
RE: SMPP: data_coding (deliver_sm)
On Thu, 2002-10-31 at 22:18, Oded Arbel wrote: -Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] this sound like smsc weirdness? Yes.. :-/ actually it does. The null character is apparently optional and can go away easily so that's not a problem. I'm not sure that tryig to detect if it exists before truncating the string is a good idea - what if someone sends binary content that ends with a 0 byte ? Sorry I was unclear - the SMSC is appending a null to the end of each message. Apparently this is a configurable option with SMSC so they can turn it off (hence it will go away). No code change required.
RE: SMPP: data_coding (deliver_sm)
-Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] ok - my mistake, according to the specs, it should be if ( pdu-u.deliver_sm.data_coding == 0x00 ) charset_gsm_to_latin1(msg-sms.msgdata); yes - I'm aware that this does not solve your problem, but SMPP servers send GSM 7 bit alphabet encoded messages using data_coding 0x00. Yes - this is what I thought. What's your opionion on having an additional config directive stating what the SMSC default alphabet. I don't know how in the world anyone is suppost to know in advance what the SMSC default alphabet is... ie: default_alphabet = 0x01 I hate putting in more configuration paremeters. how about this rewrite of data_coding handling (see attached patch file). a similar version is running on our system for about half a year now, I only now had the incentive to generalize it, but I changed some stuff, to treat this code as untested. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. Let's suppose you have a table with 2^n cups... Wait a second - is n a natural number? smsc_smpp.patch Description: smsc_smpp.patch
RE: SMPP: data_coding (deliver_sm)
Thanks Oded, looks good - can you post your gwlib/charset.c (I'm guessing) you're referencing a charset_convert function which doesn't exist in cvs. cheers, Alan On Fri, 2002-11-01 at 00:24, Oded Arbel wrote: -Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] ok - my mistake, according to the specs, it should be if ( pdu-u.deliver_sm.data_coding == 0x00 ) charset_gsm_to_latin1(msg-sms.msgdata); yes - I'm aware that this does not solve your problem, but SMPP servers send GSM 7 bit alphabet encoded messages using data_coding 0x00. Yes - this is what I thought. What's your opionion on having an additional config directive stating what the SMSC default alphabet. I don't know how in the world anyone is suppost to know in advance what the SMSC default alphabet is... ie: default_alphabet = 0x01 I hate putting in more configuration paremeters. how about this rewrite of data_coding handling (see attached patch file). a similar version is running on our system for about half a year now, I only now had the incentive to generalize it, but I changed some stuff, to treat this code as untested. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. Let's suppose you have a table with 2^n cups... Wait a second - is n a natural number?
RE: SMPP: data_coding (deliver_sm)
-Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] and how do you know that the last char is a null ? I don't think it is. Just a guess - I have \0 at the end of each message comming through without doing the ocstr_truncate on last char. Or does this sound like smsc weirdness? Yes.. :-/ actually it does. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. Yacc owes much to a most stimulating collection of users, who have goaded me beyond my inclination, and frequently beyond my ability in their endless search for one more feature. Their irritating unwillingness to learn how to do things my way has usually led to my doing things their way; most of the time, they have been right. -- S. C. Johnson, Yacc guide acknowledgements
RE: SMPP: data_coding (deliver_sm)
On Thu, 2002-10-31 at 00:15, Oded Arbel wrote: -Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] and how do you know that the last char is a null ? I don't think it is. Just a guess - I have \0 at the end of each message comming through without doing the ocstr_truncate on last char. Or does this sound like smsc weirdness? Yes.. :-/ actually it does. The null character is apparently optional and can go away easily so that's not a problem. The subsequent sms push problem I was having was a user issued involving experimental coding parameters ... Oded, are you suggesting that gsm_to_latin1 conversion take place only when data_coding == 0x01 or when != 0x01 (Am I readin the specs wrong or 0x01 Ascii?). ie: if ( pdu-u.deliver_sm.data_coding == 0x01 ) charset_gsm_to_latin1(msg-sms.msgdata);
RE: SMPP: data_coding (deliver_sm)
As you suggested Oded I simply need not perform the gsm_to_latin1 translation (but also remove trailing null character). To impliment this sensibly I was hoping the data_coding would be set to something other than 0x00 - SMSC default alphabet. Unfortunately this is not the case which would suggest I need another configurable parameter to state what the SMSC default data_coding is ... ie: extra smsc config directive would look something like default_aphabet = 0x03. Corresponding translation can be implimented in smsc_smpp.c, pdu_to_msg - although parsing the smpp structure would also be required. I also noted a tread started about whether or not payload translation should be done at bearerbox level - is there any work being done on this? Cheers, Alan Index: gw/smsc/smsc_smpp.c === RCS file: /home/cvs/gateway/gw/smsc/smsc_smpp.c,v retrieving revision 1.14 diff -r1.14 smsc_smpp.c 239c239,244 charset_gsm_to_latin1(msg-sms.msgdata); --- if ( pdu-u.deliver_sm.data_coding == 0x00 ) octstr_truncate(msg-sms.msgdata, octstr_len(msg-sms.msgdata) - 1); else charset_gsm_to_latin1(msg-sms.msgdata); On Wed, 2002-10-23 at 08:42, Alan McNatty wrote: Yep - the logic as you described it seems to be how Kannel handles things. I have not had a chance to recieve unicode data through an SMPP SMSC, but that would seem to be problematic. we don't need to transcode from UCS2, as Kannel should send unicode text as UCS2 to the application (as all other drivers do), we simply should not use gsm_to_latin1 on UCS2 encoded message as that can mangle some characters.
RE: SMPP: data_coding (deliver_sm)
Alternatively we could reuse the alt-dcs used in submit_sm. On Wed, 2002-10-30 at 12:58, Alan McNatty wrote: As you suggested Oded I simply need not perform the gsm_to_latin1 translation (but also remove trailing null character). To impliment this sensibly I was hoping the data_coding would be set to something other than 0x00 - SMSC default alphabet. Unfortunately this is not the case which would suggest I need another configurable parameter to state what the SMSC default data_coding is ... ie: extra smsc config directive would look something like default_aphabet = 0x03. Corresponding translation can be implimented in smsc_smpp.c, pdu_to_msg - although parsing the smpp structure would also be required. I also noted a tread started about whether or not payload translation should be done at bearerbox level - is there any work being done on this? Cheers, Alan Index: gw/smsc/smsc_smpp.c === RCS file: /home/cvs/gateway/gw/smsc/smsc_smpp.c,v retrieving revision 1.14 diff -r1.14 smsc_smpp.c 239c239,244 charset_gsm_to_latin1(msg-sms.msgdata); --- if ( pdu-u.deliver_sm.data_coding == 0x00 ) octstr_truncate(msg-sms.msgdata, octstr_len(msg-sms.msgdata) - 1); else charset_gsm_to_latin1(msg-sms.msgdata); On Wed, 2002-10-23 at 08:42, Alan McNatty wrote: Yep - the logic as you described it seems to be how Kannel handles things. I have not had a chance to recieve unicode data through an SMPP SMSC, but that would seem to be problematic. we don't need to transcode from UCS2, as Kannel should send unicode text as UCS2 to the application (as all other drivers do), we simply should not use gsm_to_latin1 on UCS2 encoded message as that can mangle some characters. -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 21-312136, DDI: +64 4 9167203, Office: +64 4 4992267 ... error accessing whit Segmentation fault (core dumped)
RE: SMPP: data_coding (deliver_sm)
interestingly enough - leaving the message unencoded works for incomming but somehow breaks outgoing (via http interface). The message payload is visable in bearerbox pdu dump logs but the message arrives empty - requires further investigation. On Wed, 2002-10-30 at 12:58, Alan McNatty wrote: As you suggested Oded I simply need not perform the gsm_to_latin1 translation (but also remove trailing null character). To impliment this sensibly I was hoping the data_coding would be set to something other than 0x00 - SMSC default alphabet. Unfortunately this is not the case which would suggest I need another configurable parameter to state what the SMSC default data_coding is ... ie: extra smsc config directive would look something like default_aphabet = 0x03. Corresponding translation can be implimented in smsc_smpp.c, pdu_to_msg - although parsing the smpp structure would also be required. I also noted a tread started about whether or not payload translation should be done at bearerbox level - is there any work being done on this? Cheers, Alan Index: gw/smsc/smsc_smpp.c === RCS file: /home/cvs/gateway/gw/smsc/smsc_smpp.c,v retrieving revision 1.14 diff -r1.14 smsc_smpp.c 239c239,244 charset_gsm_to_latin1(msg-sms.msgdata); --- if ( pdu-u.deliver_sm.data_coding == 0x00 ) octstr_truncate(msg-sms.msgdata, octstr_len(msg-sms.msgdata) - 1); else charset_gsm_to_latin1(msg-sms.msgdata); On Wed, 2002-10-23 at 08:42, Alan McNatty wrote: Yep - the logic as you described it seems to be how Kannel handles things. I have not had a chance to recieve unicode data through an SMPP SMSC, but that would seem to be problematic. we don't need to transcode from UCS2, as Kannel should send unicode text as UCS2 to the application (as all other drivers do), we simply should not use gsm_to_latin1 on UCS2 encoded message as that can mangle some characters. -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 21-312136, DDI: +64 4 9167203, Office: +64 4 4992267 ... error accessing whit Segmentation fault (core dumped)
RE: SMPP: data_coding (deliver_sm)
Thanks Oded, I'll put something together for this. On Mon, 2002-10-21 at 01:45, Oded Arbel wrote: -Original Message- From: Alan McNatty [mailto:alan;catalyst.net.nz] If I receive a message from a smsc in say ucs2, from what I can see in smsc_smpp.c pdu_to_msg the msg data always gets converted from gsm to latin1 regardless of source charset. Although the data coding is recorded it doesn't seem to make any difference to the msg decoding. In this way there is potential for some characters from the source message to be lost. Is this an accurate assessment? Anyone else experiencing this problem (ie chars that vanish from MO messages)? What about having a translation scheme for each data_coding (usc2_to_latin, etc with gsm_to_latin1 the default obviously. Potentially a smsc configuration directive could be useful to force a data_coding for broken smsc. Yep - the logic as you described it seems to be how Kannel handles things. I have not had a chance to recieve unicode data through an SMPP SMSC, but that would seem to be problematic. we don't need to transcode from UCS2, as Kannel should send unicode text as UCS2 to the application (as all other drivers do), we simply should not use gsm_to_latin1 on UCS2 encoded message as that can mangle some characters. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. If you lie to the compiler, it will get its revenge. -- Henry Spencer -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 21-312136, DDI: +64 4 9167203, Office: +64 4 4992267 ... error accessing whit Segmentation fault (core dumped)
SMPP: data_coding (deliver_sm)
Hi All, Just trying to clear up my understanding regarding how kannel handles smsc message charset translations. Just been through code no pdu debugging yet. If I receive a message from a smsc in say ucs2, from what I can see in smsc_smpp.c pdu_to_msg the msg data always gets converted from gsm to latin1 regardless of source charset. Although the data coding is recorded it doesn't seem to make any difference to the msg decoding. In this way there is potential for some characters from the source message to be lost. Is this an accurate assessment? Anyone else experiencing this problem (ie chars that vanish from MO messages)? What about having a translation scheme for each data_coding (usc2_to_latin, etc with gsm_to_latin1 the default obviously. Potentially a smsc configuration directive could be useful to force a data_coding for broken smsc. Cheers, Alan
