Re: Kannel left open connections on provider's side
Stipe Tolj wrote: Manuel Fernando Aller schrieb: First of all, I don't think it was problem of kannel, but searching in the source code I can't find the specific code where kannel closes connection whith the smsc after sending an sms. As you can imagine, I'm not a c++ developer :-P you don't have to. Kannel itself is C with POSIX.1 threads (pthreads), so no object orientation at all :p You've catched me again, I'm not a C developer :-S I agree with Falko, we need more details. In fact the list would need to review a snippet of your bearerbox.log to verify what is going on exactly at the time you have experienced the connection misbehavior. When we connect to this new operator, we configure Kannel as usual. Couple of days later, operator is saying that we are leaving open connections with the SMSC. I then tried to understand why this happens (with my limited knowledge of smpp protocol), And I figure out that 'kannel opens an maintains ONE and ONLY connection with the SMSC, so I don't know where or when the connections are left open', as the operator is saying that the only content aggregator with this problem is us... Another possibility is the so-called silent TCP teardrop effect, which means: a router node in the middle of the IP connection route drops the connected silently. Which means both TCP communication endpoints still think the connection is valid, but the router has cut it in the middle, without signaling the TCP connect dropping to any of the endpoints. This was my first feeling to this situation. Thanks! -- Manuel Fernando Aller
Re: Kannel left open connections on provider's side
Did I get this right: TCP not SMPP connections are left open? Well, anyway, on both protocols this can happen anytime to anyone with misconfigured or defect routers or whatever, that should be no problem to any SMSC or currently available TCP stack... that's why the good and awesome god OSI invented timeouts ;-) I administrate a a SMSC and a couple of VSMSCs with some hundred clients connected to them. There are always some not connected or the connection has broken down without any TCP FIN or SMPP UNBIND. The connection hangs around in an undefined state for some seconds and returns to Listen/Disconnected and awaits a new connection. I really don't get the problem... But maybe it would be very helpful if you have a bearerbox.log from such an event. Regards Falko Am 17.02.2009 um 17:33 schrieb Manuel Fernando Aller: Stipe Tolj wrote: Manuel Fernando Aller schrieb: First of all, I don't think it was problem of kannel, but searching in the source code I can't find the specific code where kannel closes connection whith the smsc after sending an sms. As you can imagine, I'm not a c++ developer :-P you don't have to. Kannel itself is C with POSIX.1 threads (pthreads), so no object orientation at all :p You've catched me again, I'm not a C developer :-S I agree with Falko, we need more details. In fact the list would need to review a snippet of your bearerbox.log to verify what is going on exactly at the time you have experienced the connection misbehavior. When we connect to this new operator, we configure Kannel as usual. Couple of days later, operator is saying that we are leaving open connections with the SMSC. I then tried to understand why this happens (with my limited knowledge of smpp protocol), And I figure out that 'kannel opens an maintains ONE and ONLY connection with the SMSC, so I don't know where or when the connections are left open', as the operator is saying that the only content aggregator with this problem is us... Another possibility is the so-called silent TCP teardrop effect, which means: a router node in the middle of the IP connection route drops the connected silently. Which means both TCP communication endpoints still think the connection is valid, but the router has cut it in the middle, without signaling the TCP connect dropping to any of the endpoints. This was my first feeling to this situation. Thanks! -- Manuel Fernando Aller
Re: Kannel left open connections on provider's side
Manuel Fernando Aller schrieb:
I then tried to understand why this happens (with my limited knowledge
of smpp protocol), And I figure out that 'kannel opens an maintains ONE
and ONLY connection with the SMSC, so I don't know where or when the
connections are left open', as the operator is saying that the only
content aggregator with this problem is us...
my suggetion:
let the operator provide you logs which proof that. And you may have the
natural arrogance to tell them We use Kannel, the most used SMS
gateway in the world!. Maybe they will re-think if it's up to the SMPP
cleint. Which is most likely not, but rather a network (TCP) related
issue.. ;)
Stipe
--
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
Re: Kannel left open connections on provider's side
Falko Ziemann schrieb:
Did I get this right: TCP not SMPP connections are left open?
he said open connection, which leaves the interpretation room to both.
That's why my most cited sentence should be BE PRECISE ;) ... this is
all complex things we're doing, if you boil it from the I send a SMS
down to the TCP/IP connection stage.
Well, anyway, on both protocols this can happen anytime to anyone with
misconfigured or defect routers or whatever, that should be no problem
to any SMSC or currently available TCP stack... that's why the good and
awesome god OSI invented timeouts ;-)
yeah... but that's still existent. There are routers outside, and I
don't take the name C...o into my mouth, that can be configured the way
to do silent TCP tear dropping.
Stipe
--
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
Re: Kannel left open connections on provider's side
yeah... but that's still existent. There are routers outside, and I don't take the name C...o into my mouth, that can be configured the way to do silent TCP tear dropping. Sure, and even if they are configured to do so, or a 3..m-router is broken, or the apprentice fell over an ethernetcable, still any TCP stack should sooner or later recognise that something is wrong with the connection and reset it. By the way: I first thought I didn't do my homework, but what is TCP tear dropping? The only thing I know sounding similar is TCP-TEARDROP and that is cracking a TCP stack by overloading the payload and I am quite shure that you cannot configure any router from C...o to do DoS- attacks and silent is not the first word I think of when I hear DoS. It would be really nice if you could send me any link to an article explaining this effect... Regards Falko
Re: Kannel left open connections on provider's side
Stipe Tolj wrote: Manuel Fernando Aller schrieb: I then tried to understand why this happens (with my limited knowledge of smpp protocol), And I figure out that 'kannel opens an maintains ONE and ONLY connection with the SMSC, so I don't know where or when the connections are left open', as the operator is saying that the only content aggregator with this problem is us... my suggetion: let the operator provide you logs which proof that. And you may have the natural arrogance to tell them We use Kannel, the most used SMS gateway in the world!. Maybe they will re-think if it's up to the SMPP cleint. Which is most likely not, but rather a network (TCP) related issue.. ;) Stipe Ok, the operator sent the output of netstat -n to me: tcp0 0 192.168.1.55:3000 10.10.2.10:55740 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:41852 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:40574 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45049 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:43960 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45496 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:56315 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:57205 CLOSE_WAIT . . where 192.168.1.55 is the local ip for his smsc server, and 10.10.2.10 is my ip (yes, I've chage that!). So, the operator says I left TCP connections open. I do not connect with 192.168.1.55, I connect with a valid public IP, and evidently the firewall/router/whatever the operator have, redirects the connection to 192.168.1.55. I believe the problem is in the firewall/router/whatever. -- Manuel Fernando Aller
Re: Kannel left open connections on provider's side
Ok, the operator sent the output of netstat -n to me: tcp0 0 192.168.1.55:3000 10.10.2.10:55740 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:41852 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:40574 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45049 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:43960 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45496 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:56315 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:57205 CLOSE_WAIT . . where 192.168.1.55 is the local ip for his smsc server, and 10.10.2.10 is my ip (yes, I've chage that!). That's NAT, that's normal. But, I admit, I have forgot a lot about the TCP stack since my apprentienceship, but doesn't CLOSE_WAIT mean, the foreign side (you) closed the connection and the stack is waiting for the LOCAL software to acknowledge the FIN? http://en.wikipedia.org/wiki/File:Tcp_state_diagram_fixed.svg Regards Falko
Re: Kannel left open connections on provider's side
Well, actually it shows that the local SMPP server (192.x) is waiting on a FIN from 10.10.x. There is an issue why 10.10 is not sending a fin (brute disconnection?), but the provider should fine-tune his tcp-close-wait-interval, nevertheless. In Solaris: ndd /dev/tcp tcp_time_wait_interval 6 (for 1' CLOSE_WAIT interval) BR, Nikos - Original Message - From: Falko Ziemann fal...@gmail.com To: Manuel Fernando Aller manuel.al...@gmail.com Cc: users@kannel.org Sent: Tuesday, February 17, 2009 8:14 PM Subject: Re: Kannel left open connections on provider's side Ok, the operator sent the output of netstat -n to me: tcp0 0 192.168.1.55:3000 10.10.2.10:55740 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:41852 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:40574 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45049 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:43960 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45496 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:56315 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:57205 CLOSE_WAIT . . where 192.168.1.55 is the local ip for his smsc server, and 10.10.2.10 is my ip (yes, I've chage that!). That's NAT, that's normal. But, I admit, I have forgot a lot about the TCP stack since my apprentienceship, but doesn't CLOSE_WAIT mean, the foreign side (you) closed the connection and the stack is waiting for the LOCAL software to acknowledge the FIN? http://en.wikipedia.org/wiki/File:Tcp_state_diagram_fixed.svg Regards Falko
Re: Kannel left open connections on provider's side
Hi, Just to add that this is not a kannel issue, but network. Kannel, like any application, doesn't control the 3-way TCP handshake or the 3-way closing. It just opens and closes a connection. The underlying OS is taking care of the rest, which it is safe to assume it is not at fault. I see 2 problems: 1) Network: Something is hanging up connections to dry. Since the SMSc says that no other client has this issue, it could be a device on your side. If you don't get any hang connections with other servers, it could be something in the middle. 2) Server: The SMSc server needs to configure tcp_close_timeouts on the tcp driver. I am surprised if it is the only time they noticed. BR, Nikos - Original Message - From: Nikos Balkanas nbalka...@gmail.com To: Falko Ziemann fal...@gmail.com; Manuel Fernando Aller manuel.al...@gmail.com Cc: users@kannel.org Sent: Tuesday, February 17, 2009 8:56 PM Subject: Re: Kannel left open connections on provider's side Well, actually it shows that the local SMPP server (192.x) is waiting on a FIN from 10.10.x. There is an issue why 10.10 is not sending a fin (brute disconnection?), but the provider should fine-tune his tcp-close-wait-interval, nevertheless. In Solaris: ndd /dev/tcp tcp_time_wait_interval 6 (for 1' CLOSE_WAIT interval) BR, Nikos - Original Message - From: Falko Ziemann fal...@gmail.com To: Manuel Fernando Aller manuel.al...@gmail.com Cc: users@kannel.org Sent: Tuesday, February 17, 2009 8:14 PM Subject: Re: Kannel left open connections on provider's side Ok, the operator sent the output of netstat -n to me: tcp0 0 192.168.1.55:3000 10.10.2.10:55740 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:41852 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:40574 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45049 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:43960 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45496 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:56315 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:57205 CLOSE_WAIT . . where 192.168.1.55 is the local ip for his smsc server, and 10.10.2.10 is my ip (yes, I've chage that!). That's NAT, that's normal. But, I admit, I have forgot a lot about the TCP stack since my apprentienceship, but doesn't CLOSE_WAIT mean, the foreign side (you) closed the connection and the stack is waiting for the LOCAL software to acknowledge the FIN? http://en.wikipedia.org/wiki/File:Tcp_state_diagram_fixed.svg Regards Falko
Re: Kannel left open connections on provider's side
Dears, I had similar problem, I find workaround (not efficient but work for me), that I increase the reconnect-delay so the TCP connections (from client side) closed. Hope that work for you. BR, Hafez On Tue, Feb 17, 2009 at 9:53 PM, Nikos Balkanas nbalka...@gmail.com wrote: Hi, Just to add that this is not a kannel issue, but network. Kannel, like any application, doesn't control the 3-way TCP handshake or the 3-way closing. It just opens and closes a connection. The underlying OS is taking care of the rest, which it is safe to assume it is not at fault. I see 2 problems: 1) Network: Something is hanging up connections to dry. Since the SMSc says that no other client has this issue, it could be a device on your side. If you don't get any hang connections with other servers, it could be something in the middle. 2) Server: The SMSc server needs to configure tcp_close_timeouts on the tcp driver. I am surprised if it is the only time they noticed. BR, Nikos - Original Message - From: Nikos Balkanas nbalka...@gmail.com To: Falko Ziemann fal...@gmail.com; Manuel Fernando Aller manuel.al...@gmail.com Cc: users@kannel.org Sent: Tuesday, February 17, 2009 8:56 PM Subject: Re: Kannel left open connections on provider's side Well, actually it shows that the local SMPP server (192.x) is waiting on a FIN from 10.10.x. There is an issue why 10.10 is not sending a fin (brute disconnection?), but the provider should fine-tune his tcp-close-wait-interval, nevertheless. In Solaris: ndd /dev/tcp tcp_time_wait_interval 6 (for 1' CLOSE_WAIT interval) BR, Nikos - Original Message - From: Falko Ziemann fal...@gmail.com To: Manuel Fernando Aller manuel.al...@gmail.com Cc: users@kannel.org Sent: Tuesday, February 17, 2009 8:14 PM Subject: Re: Kannel left open connections on provider's side Ok, the operator sent the output of netstat -n to me: tcp0 0 192.168.1.55:3000 10.10.2.10:55740 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:41852 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:40574 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45049 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:43960 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:45496 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:56315 CLOSE_WAIT tcp0 0 192.168.1.55:3000 10.10.2.10:57205 CLOSE_WAIT . . where 192.168.1.55 is the local ip for his smsc server, and 10.10.2.10 is my ip (yes, I've chage that!). That's NAT, that's normal. But, I admit, I have forgot a lot about the TCP stack since my apprentienceship, but doesn't CLOSE_WAIT mean, the foreign side (you) closed the connection and the stack is waiting for the LOCAL software to acknowledge the FIN? http://en.wikipedia.org/wiki/File:Tcp_state_diagram_fixed.svg Regards Falko -- Hafez A.Ahmad Amman-Jordan mobile:962-785259011 962-795708728 http://blog.hafezadnan.com
Kannel left open connections on provider's side
Hi! First of all, I don't think it was problem of kannel, but searching in the source code I can't find the specific code where kannel closes connection whith the smsc after sending an sms. As you can imagine, I'm not a c++ developer :-P Can anybody help me find this code? Thanks a lot! -- Manolo
Re: Kannel left open connections on provider's side
Hi, could you give some more details? What protocol you're talking about for example? In general it is absolutly normal for standard protocols like SMPP or EMI/UCP to NOT close the connection. Normally you even sent keep alives, so that the connection is not shut down... The connection should only be closed when you stop the bearerbox or terminate a smsc connection and that is done by kannel. To say it with the words of M$: it's not a bug, it's a feature. Regards Falko Am 16.02.2009 um 22:43 schrieb Manuel Fernando Aller: Hi! First of all, I don't think it was problem of kannel, but searching in the source code I can't find the specific code where kannel closes connection whith the smsc after sending an sms. As you can imagine, I'm not a c++ developer :-P Can anybody help me find this code? Thanks a lot! -- Manolo
Re: Kannel left open connections on provider's side
Manuel Fernando Aller schrieb:
First of all, I don't think it was problem of kannel, but searching in
the source code I can't find the specific code where kannel closes
connection whith the smsc after sending an sms.
As you can imagine, I'm not a c++ developer :-P
you don't have to. Kannel itself is C with POSIX.1 threads (pthreads),
so no object orientation at all :p
I agree with Falko, we need more details. In fact the list would need to
review a snippet of your bearerbox.log to verify what is going on
exactly at the time you have experienced the connection misbehavior.
Another possibility is the so-called silent TCP teardrop effect, which
means: a router node in the middle of the IP connection route drops the
connected silently. Which means both TCP communication endpoints still
think the connection is valid, but the router has cut it in the
middle, without signaling the TCP connect dropping to any of the endpoints.
Typically this leads at some point to a re-connection establishment of
the client, and if the server has a limit in terms of allowed
connections, you run into trouble. I hope you got the picture.
So, the reasons can be various. And typically it's NOT Kannel's fault.
So, please send some details to the list, so people can shade some light
on your issue.
Stipe
--
---
Kölner Landstrasse 419
40589 Düsseldorf, NRW, Germany
tolj.org system architecture Kannel Software Foundation (KSF)
http://www.tolj.org/ http://www.kannel.org/
mailto:st_{at}_tolj.org mailto:stolj_{at}_kannel.org
---
