Hi, Giulio Harding wrote:
> It works! Thanks again Alex, I think we're almost due to send you > some more 'thankyou' beer ;) it's always welcome ;) > > Interestingly, the carriage returns make it to the handset fine, but > they're still output incorrectly (or at least, differently) as '.' > characters in the kannel traffic logs: > > 2008-01-24 20:13:02 SMS HTTP-request sender:XXX request: 'Azure > hotspot PIN^Mfrom Vodafone^MXXXXXXXX' url: 'http://localhost/kannel? > msgData=Test&sourceAddr=XXX&channel=mbloxpsms&destinationAddr=19996736&t > lv=%3Fsmpp%3FMBbilling%3D55%26MBoperator%3D50503%26' reply: 200 '<< > successful >>' > 2008-01-24 20:13:04 Sent SMS [SMSC:mbloxpsms] [SVC:kannel_gw] [ACT:] > [BINF:] [from:19996736] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:41:Azure > hotspot PIN.from Vodafone.XXXXXXXX.] [udh:0:] > [id:b6fdd2f0-30e3-49d0-8e68-28c07bcf68d1] > > Anyway, that's a minor irritation that I can have a look at later - > in the meantime, the problem's fixed, and I get to put the TLV- > enabled Kannel back in production :) > > On 24/01/2008, at 4:19 AM, Alexander Malysh wrote: > >> Hi, >> >> the fix is pretty simple and bug only occurs if split-chars defined in >> config, therefore I was unable to reproduce this bug for sooo long :) >> >> Please try attached patch that fixes it for me. >> >> Giulio Harding wrote: >> >>> I think this is the HTTP service involved (we have a few): >>> >>> group = sms-service >>> name = kannel_gw >>> keyword = default >>> #get-url = http://10.110.123.101:8404/?msgData=%a&sourceAddr= >>> %p&channel=%i&destinationAddr=%P >>> get-url = http://localhost/kannel?msgData=%a&sourceAddr=%p&channel= >>> %i&destinationAddr=%P&tlv=%D >>> max-messages = 1 >>> catch-all = true >>> concatenation = false >>> split-chars = " " >>> omit-empty = true >>> >>> This is a configuration that I inherited many years back - I didn't >>> know what the split-chars parameter was for until now, though I still >>> don't fully understand how it's used. >>> >>> On 24/01/2008, at 12:25 AM, Alexander Malysh wrote: >>> >>>> Hi, >>>> >>>> did you define 'split-chars' in your config and if so please post it >>>> here, >>>> so I can test it. >>>> >>>> Giulio Harding wrote: >>>> >>>>> I'm kind of stumped with this problem, so I figured I'd write up a >>>>> summary of my investigation so far, in case someone else might have >>>>> an idea... >>>>> >>>>> On 23/01/2008, at 12:34 AM, Giulio Harding wrote: >>>>> >>>>>> I've actually had to back out of using the meta-data Kannel, as we >>>>>> spotted what looks to be a problem in the way Kannel handles >>>>>> carriage return characters (ctrl-M) in synchronous responses to MO >>>>>> HTTP requests - the carriage returns arrive at smsbox fine, but by >>>>>> the time they leave the bearerbox in an SMPP PDU, they somehow end >>>>>> up actually mangling the MT message body (text after the carriage >>>>>> returns is missing, subsequent text segments overwrite the >>>>>> start of >>>>>> the message body, etc.). Asynchronous MT HTTP requests aren't >>>>>> affected... >>>>>> >>>>>> Not 100% sure whether it's a Kannel problem or something else >>>>>> (site- >>>>>> specific) - I'll do a bit more investigation tomorrow morning and >>>>>> get back to you ASAP. >>>>> >>>>> >>>>> When we tested the patched TLV-enabled branch of Kannel (hereby >>>>> referred to as *NEW* Kannel) a few days ago, everything worked >>>>> fine, >>>>> except that some messages coming from one particular application >>>>> appeared to be being mangled by Kannel - from our kannel traffic >>>>> log: >>>>> >>>>> ... >>>>> 2008-01-22 03:10:27 SMS HTTP-request sender:XXX request: 'Azure >>>>> hotspot PIN^Mfrom Vodafone^MXXXXXXXX' url: 'http:// >>>>> 10.110.123.101:8404/? >>>>> msgData >>>>> =Go&sourceAddr=XXX&channel=vodafonechmt&destinationAddr=19929873' >>>>> reply: 200 '<< successful >>' >>>>> 2008-01-22 03:10:28 Sent SMS [SMSC:vodafonechmt] [SVC:kannel_gw] >>>>> [ACT:] [BINF:] [from:19929873] [to:XXX] [flags:-1:0:-1:-1:-1] [msg: >>>>> 23:Azure hotspot PIN.from ] [udh:0:] [id: >>>>> 7936719e-1c7c-4e90-8161-6f14af4f23f7] >>>>> ... >>>>> >>>>> So, going in, the message is intact, but going out, it's had a '.' >>>>> character inserted, and seems to be truncated. >>>>> >>>>> This wasn't a problem with our original Kannel instance (hereby >>>>> referred to as *OLD* Kannel) - Kannel bearerbox version >>>>> `cvs-20060727'. Build `Apr 12 2007 17:01:36', compiler `3.4.6 >>>>> 20060404 (Red Hat 3.4.6-3)'. System Linux, release >>>>> 2.6.9-55.0.2.ELsmp, version #1 SMP Tue Jun 26 14:14:47 EDT 2007, >>>>> machine x86_64. Hostname smsgw2.appgw.mnetcorporation.com, IP >>>>> 10.110.123.31. Libxml version 2.6.16. Using native malloc. >>>>> >>>>> ... >>>>> 2008-01-22 00:43:57 SMS HTTP-request sender:XXX request: 'Azure >>>>> hotspot PIN^Mfrom Optus^MXXXXXXXX' url: 'http:// >>>>> 10.110.123.101:8404/? >>>>> msgData=Go&sourceAddr=XXX&channel=optuschmt&destinationAddr=1992987 >>>>> 3' >>>>> reply: 200 '<< successful >>' >>>>> 2008-01-22 00:43:57 Sent SMS [SMSC:optuschmt] [SVC:kannel_gw] >>>>> [ACT:] >>>>> [BINF:] [from:19929873] [to:XXX] [flags:-1:0:-1:-1:-1] [msg: >>>>> 37:Azure >>>>> hotspot PIN^Mfrom Optus^MXXXXXXXX] [udh:0:] [id:86dfd47e-0106-4700- >>>>> ab48-47381c43047c] >>>>> ... >>>>> >>>>> Also, part of the problem seems to be triggered by the presence of >>>>> the carriage return characters (^M) and part seems to be their >>>>> presence in the HTTP response to an MO request. >>>>> >>>>> In *OLD* Kannel, a separate MT request with the same text works >>>>> fine: >>>>> >>>>> ... >>>>> 2008-01-23 23:21:12 send-SMS request added - sender:kannelpush:test >>>>> 10.110.123.101 target:XXX request: 'Azure hotspot PIN^Mfrom >>>>> Optus^MXXXXXXXXX' >>>>> 2008-01-23 23:21:13 Sent SMS [SMSC:mbloxdirect] [SVC:kannelpush] >>>>> [ACT:] [BINF:] [from:test] [to:XXX] [flags:-1:0:-1:-1:-1] [msg: >>>>> 37:Azure hotspot PIN^Mfrom Optus^MXXXXXXXX] [udh:0:] [id:09ba2907- >>>>> f013-46ff-9ec0-de006422d0b4] >>>>> ... >>>>> >>>>> But in *NEW* Kannel, a separate MT request with the same text still >>>>> has problems: it's not truncated any more, but it has '.' >>>>> characters >>>>> instead of carriage returns: >>>>> >>>>> ... >>>>> 2008-01-22 13:49:55 send-SMS request added - sender:kannelpush: >>>>> 19996737 127.0.0.1 target:XXXX request: 'Azure hotspot PIN^Mfrom >>>>> Optus^MXXXXXXXX' >>>>> 2008-01-22 13:49:56 Sent SMS [SMSC:mbloxpsms] [SVC:kannelpush] >>>>> [ACT:] >>>>> [BINF:] [from:19996737] [to:XXX] [flags:-1:0:-1:-1:-1] [msg: >>>>> 37:Azure >>>>> hotspot PIN.from Optus.XXXXXXXX] [udh:0:] >>>>> [id:d9308e85-4790-4714-9ac9-48a72159ac31] >>>>> ... >>>>> >>>>> So, with a few extra debug/trace statements, I was able to narrow >>>>> down the mangling as possibly occurring in the >>>>> extract_msgdata_part_by_coding function, in gw/sms.c: >>>>> >>>>> >>>>> >>>>> static Octstr *extract_msgdata_part_by_coding(Msg *msg, Octstr >>>>> *split_chars, >>>>> int max_part_len) >>>>> { >>>>> Octstr *temp = NULL, *temp_utf; >>>>> >>>>> if (msg->sms.coding == DC_8BIT || msg->sms.coding == DC_UCS2) { >>>>> /* nothing to do here, just call the original >>>>> extract_msgdata_part */ >>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars, >>>>> max_part_len); >>>>> } >>>>> >>>>> /* convert to and the from gsm, so we drop all non GSM chars */ >>>>> charset_utf8_to_gsm(msg->sms.msgdata); >>>>> charset_gsm_to_utf8(msg->sms.msgdata); >>>>> >>>>> /* >>>>> * else we need to do something special. I'll just get >>>>> charset_gsm_truncate to >>>>> * cut the string to the required length and then count real >>>>> characters. >>>>> */ >>>>> temp = octstr_duplicate(msg->sms.msgdata); >>>>> charset_utf8_to_gsm(temp); >>>>> charset_gsm_truncate(temp, max_part_len); >>>>> >>>>> /* calculate utf-8 length */ >>>>> temp_utf = octstr_duplicate(temp); >>>>> charset_gsm_to_utf8(temp_utf); >>>>> max_part_len = octstr_len(temp_utf); >>>>> >>>>> octstr_destroy(temp); >>>>> octstr_destroy(temp_utf); >>>>> >>>>> /* now just call the original extract_msgdata_part with the >>>>> new >>>>> length */ >>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars, >>>>> max_part_len); >>>>> } >>>>> >>>>> >>>>> This has changed noticeably from our *OLD* Kannel: >>>>> >>>>> >>>>> static Octstr *extract_msgdata_part_by_coding(Msg *msg, Octstr >>>>> *split_chars, >>>>> int max_part_len) >>>>> { >>>>> Octstr *temp = NULL; >>>>> int pos, esc_count; >>>>> >>>>> if (msg->sms.coding == DC_8BIT || msg->sms.coding == DC_UCS2) { >>>>> /* nothing to do here, just call the original >>>>> extract_msgdata_part */ >>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars, >>>>> max_part_len); >>>>> } >>>>> >>>>> /* >>>>> * else we need to do something special. I'll just get >>>>> charset_gsm_truncate to >>>>> * cut the string to the required length and then count real >>>>> characters. >>>>> */ >>>>> temp = octstr_duplicate(msg->sms.msgdata); >>>>> charset_latin1_to_gsm(temp); >>>>> charset_gsm_truncate(temp, max_part_len); >>>>> >>>>> pos = esc_count = 0; >>>>> >>>>> while ((pos = octstr_search_char(temp, 27, pos)) != -1) { >>>>> ++pos; >>>>> ++esc_count; >>>>> } >>>>> >>>>> octstr_destroy(temp); >>>>> >>>>> /* now just call the original extract_msgdata_part with the >>>>> new >>>>> length */ >>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars, >>>>> max_part_len - esc_count); >>>>> } >>>>> >>>>> >>>>> The biggest change is the use of UTF-8 as the internal encoding, I >>>>> think - however, one part of the mangling, the truncation, >>>>> appears to >>>>> be happening in the extract_msgdata_part function. I jammed in some >>>>> extra traces, like so: >>>>> >>>>> >>>>> >>>>> static Octstr *extract_msgdata_part(Octstr *msgdata, Octstr >>>>> *split_chars, >>>>> int max_part_len) >>>>> { >>>>> debug("sms", 0, "X1: %s", octstr_get_cstr(msgdata)); >>>>> long i, len; >>>>> Octstr *part; >>>>> >>>>> debug("sms", 0, "split: %s", octstr_get_cstr(split_chars)); >>>>> debug("sms", 0, "max_part_len: %i", max_part_len); >>>>> len = max_part_len; >>>>> if (split_chars != NULL) { >>>>> debug("sms", 0, "split_chars not null"); >>>>> for (i = max_part_len; i > 0; i--) { >>>>> debug("sms", 0, "working back from max_part_len: %i", i); >>>>> if (octstr_search_char(split_chars, >>>>> octstr_get_char(msgdata, i - 1), >>>>> 0) != -1) { >>>>> debug("sms", 0, "something something, at char %i ->%c<-", i, >>>>> octstr_get_char(msgdata, i - 1)); >>>>> len = i; >>>>> break; >>>>> } >>>>> } >>>>> } >>>>> part = octstr_copy(msgdata, 0, len); >>>>> debug("sms", 0, "len: %i", len); >>>>> debug("sms", 0, "X2: %s", octstr_get_cstr(part)); >>>>> octstr_delete(msgdata, 0, len); >>>>> return part; >>>>> } >>>>> >>>>> >>>>> And got the following output: >>>>> >>>>> >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: X1: Azure hotspot PIN^Mfrom >>>>> Vodafone^MXXXXXXXX >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: split: >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: max_part_len: 41 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: split_chars not null >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 41 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 40 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 39 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 38 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 37 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 36 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 35 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 34 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 33 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 32 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 31 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 30 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 29 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 28 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 27 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 26 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 25 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 24 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from >>>>> max_part_len: 23 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: something something, at char >>>>> 23 -> <- >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: len: 23 >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: X2: Azure hotspot PIN^Mfrom >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: Extract: Azure hotspot >>>>> PIN^Mfrom >>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: message length 41, sending 1 >>>>> messages >>>>> >>>>> >>>>> So, that loop appears to be truncating the message text - but >>>>> I'm not >>>>> sure why. Can someone explain that code? Is that behaviour >>>>> incorrect? >>>>> (It looks like it to me) >>>>> >>>>> As for the carriage returns becoming '.' characters, I haven't >>>>> located where that's occurring, but I'm wondering whether that >>>>> might >>>>> be due to the switch to UTF8 internally? >>>>> >>>>> Anyway, like I said, I'm a bit out of my depth here - does anyone >>>>> have any ideas? Is anyone else able to replicate this strange >>>>> behaviour with carriage returns in the meta_data branch? (or >>>>> even the >>>>> latest CVS snapshot, since this doesn't seem to be TLV-specific...) >>>>> >>>>> Thanks, >>>>> >>>>> -- >>>>> Giulio Harding >>>>> Systems Administrator >>>>> >>>>> m.Net Corporation >>>>> Level 2, 8 Leigh Street >>>>> Adelaide SA 5000, Australia >>>>> >>>>> Tel: +61 8 8210 2041 >>>>> Fax: +61 8 8211 9620 >>>>> Mobile: 0432 876 733 >>>>> Yahoo: giulio.harding >>>>> MSN: [EMAIL PROTECTED] >>>>> >>>>> http://www.mnetcorporation.com >>>> >>>> -- >>>> Thanks, >>>> Alex >>>> >>>> >>> >>> -- >>> Giulio Harding >>> Systems Administrator >>> >>> m.Net Corporation >>> Level 2, 8 Leigh Street >>> Adelaide SA 5000, Australia >>> >>> Tel: +61 8 8210 2041 >>> Fax: +61 8 8211 9620 >>> Mobile: 0432 876 733 >>> MSN: [EMAIL PROTECTED] >>> >>> http://www.mnetcorporation.com >> >> -- >> Thanks, >> Alex<truncate-fix.diff> > > -- > Giulio Harding > Systems Administrator > > m.Net Corporation > Level 2, 8 Leigh Street > Adelaide SA 5000, Australia > > Tel: +61 8 8210 2041 > Fax: +61 8 8211 9620 > Mobile: 0432 876 733 > Yahoo: giulio.harding > MSN: [EMAIL PROTECTED] > > http://www.mnetcorporation.com -- Thanks, Alex
