Re: Problem with GET HTTP

[Stipe Tolj]

Am 19.10.2017 15:56, schrieb Stipe Tolj:

Am 24.08.2017 18:24, schrieb Giulio Giovannini:

Hello Stipe.

Thanks for the interest.

I can not confirm 100% that but I can see the smsbox.log flowing without
interruption. So I assume that the lines below are written as a DLR is
picked up by the smsbox and if a new line gets printed the previous DLR
has been processed.

2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 

Nonethless, while that log flows fluid, a tcpdump on the machine shows
that HTTP GET are delayed, like accumulated in batches. That is
confirmed on the other end where the HTTP GETs are received in batches.

When traffic is low, there are no losses. When traffic gets intense I
noticed losses in the DLRs on the HTTP receiving end.


Hi Giulio,

now, the only reason I see at the moment for something that "looks" like
a batched HTTP call processing, is the limit that Kannel's HTTP client
sets for how many concurrent calls are allowed to be in a "open state".

Which is controlled via:

group = smsbox
...
max-pending-requests = x

BTW, default is 512 if not configured.


BTW, this was introduced as "protection" towards the HTTP servers, as we 
had various cases in the real-world where Kannel was loading the HTTP 
server side so heavily,that it was basically becoming a DOS attack.


--
Best Regards,
Stipe Tolj

---
Düsseldorf, NRW, Germany

Kannel Foundation tolj.org system architecture
http://www.kannel.org/http://www.tolj.org/

stolj at kannel.org   st at tolj.org
---

Re: Problem with GET HTTP

[Stipe Tolj]

Am 24.08.2017 18:24, schrieb Giulio Giovannini:

Hello Stipe.

Thanks for the interest.

I can not confirm 100% that but I can see the smsbox.log flowing without
interruption. So I assume that the lines below are written as a DLR is
picked up by the smsbox and if a new line gets printed the previous DLR
has been processed.

2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
 from 

Nonethless, while that log flows fluid, a tcpdump on the machine shows
that HTTP GET are delayed, like accumulated in batches. That is
confirmed on the other end where the HTTP GETs are received in batches.

When traffic is low, there are no losses. When traffic gets intense I
noticed losses in the DLRs on the HTTP receiving end.


Hi Giulio,

now, the only reason I see at the moment for something that "looks" like 
a batched HTTP call processing, is the limit that Kannel's HTTP client 
sets for how many concurrent calls are allowed to be in a "open state".


Which is controlled via:

  group = smsbox
  ...
  max-pending-requests = x

BTW, default is 512 if not configured.

So, what COULD happen is that we load the HTTP server in a high load 
situation, the HTTP server may take a bit longer to respond per request, 
yielding this "open response" count to 512, and while this is the case 
the HTTP client side layer would "block" the further calling towards the 
HTTP server. Then HTTP server flushes out a bunch of responses, the 
counter drops again for some figure (i.e. 50) and then Kannel's HTTP 
client writes again 50, to block again since 512 are again "open". As 
long as the input flow is high, this could result in such an effect you see.


But it doesn't explain the differences in the revisions you have been using.

Any chance you can create a test scenario with can be reproduced 
deterministically on our side, so we see the differences for both revisions?


Reviewing the changesets from r5180 to r5186 I see only ONE thing 
touching gwlib/http.c, which is the HTTP/1.1 keep-alive correction I 
added, but this doesn't touch any of the HTTP client side code, as this 
is what we do as HTTP server.


So we simply can't qualify any change to be the cause and rather assume 
my above scenario to be the case.


--
Best Regards,
Stipe Tolj

---
Düsseldorf, NRW, Germany

Kannel Foundation tolj.org system architecture
http://www.kannel.org/http://www.tolj.org/

stolj at kannel.org   st at tolj.org
---

Re: Problem with GET HTTP

[Giulio Giovannini]

Sorry, In my previous email I provided the svn info of one of my kannel
instance that HASN'T GOT the problem.

The svn info of the kannel with the problem is this one:

Path: .
URL: https://svn.kannel.org/gateway/trunk
Relative URL: ^/trunk
Repository Root: https://svn.kannel.org/gateway
Repository UUID: a7d65f57-29cd-40c9-b71e-ebca51ad1b1d
Revision: 5186
Node Kind: directory
Schedule: normal
Last Changed Author: stolj
Last Changed Rev: 5186
Last Changed Date: 2017-04-11 14:36:14 +0200 (Tue, 11 Apr 2017)





[image: Mobyt S.p.A.]  *Giulio Giovannini*
*Devops Mobyt*
giulio.giovann...@mobyt.it
*Mobyt S.p.A.*
Via Aldighieri 10
44121 Ferrara - Italy tel. + 39 0532 207296
fax. + 39 0532 242952
*www.mobyt.it* 

2017-08-24 18:24 GMT+02:00 Giulio Giovannini :

> Hello Stipe.
>
> Thanks for the interest.
>
> I can not confirm 100% that but I can see the smsbox.log flowing without
> interruption. So I assume that the lines below are written as a DLR is
> picked up by the smsbox and if a new line gets printed the previous DLR has
> been processed.
>
> 2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
>  from 
> 2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
>  from 
> 2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report
>  from 
>
> Nonethless, while that log flows fluid, a tcpdump on the machine shows
> that HTTP GET are delayed, like accumulated in batches. That is confirmed
> on the other end where the HTTP GETs are received in batches.
>
> When traffic is low, there are no losses. When traffic gets intense I
> noticed losses in the DLRs on the HTTP receiving end.
>
> This is the output of the svn info that produced the executable:
>
> Path: .
> URL: https://svn.kannel.org/gateway/trunk
> Relative URL: ^/trunk
> Repository Root: https://svn.kannel.org/gateway
> Repository UUID: a7d65f57-29cd-40c9-b71e-ebca51ad1b1d
> Revision: 5180
> Node Kind: directory
> Schedule: normal
> Last Changed Author: amalysh
> Last Changed Rev: 5180
> Last Changed Date: 2017-01-10 18:34:44 +0100 (Tue, 10 Jan 2017)
>
> Best regards,
> Giulio
>
>
>
> [image: Mobyt S.p.A.]  *Giulio Giovannini*
> *Devops Mobyt*
> giulio.giovann...@mobyt.it
> *Mobyt S.p.A.*
> Via Aldighieri 10
> 44121 Ferrara - Italy tel. + 39 0532 207296 <+39%200532%20207296>
> fax. + 39 0532 242952 <+39%200532%20242952>
> *www.mobyt.it* 
>
> 2017-08-24 17:51 GMT+02:00 Stipe Tolj :
>
>> Am 10.08.2017 12:38, schrieb Giulio Giovannini:
>>
>>> Hi,
>>>
>>> nobody has anything to say about this issue?
>>>
>>> More details: I am seeing a difference of up to 8 seconds between the
>>> logging of  DLR on smsbox log and the actual sending of the HTTP GET on
>>> the network interface.
>>>
>>> The server the application is running on does not seem to have any
>>> problem with the networking.
>>>
>>
>> can you confirm that the DLRs are passed from bearerbox to smsbox in a
>> reasonable time, and that the "delaying" is from smsbox's HTTP call
>> perspective?
>>
>> BTW, no, there was no "caching" added, as we always keep the premises to
>> transport ASAP and as fast as possible.
>>
>> --
>> Best Regards,
>> Stipe Tolj
>>
>> ---
>> Düsseldorf, NRW, Germany
>>
>> Kannel Foundation tolj.org system architecture
>> http://www.kannel.org/http://www.tolj.org/
>>
>> stolj at kannel.org   st at tolj.org
>> ---
>>
>>
>

Re: Problem with GET HTTP

[Giulio Giovannini]

Hello Stipe.

Thanks for the interest.

I can not confirm 100% that but I can see the smsbox.log flowing without
interruption. So I assume that the lines below are written as a DLR is
picked up by the smsbox and if a new line gets printed the previous DLR has
been processed.

2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report 
from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report 
from 
2017-08-24 18:18:27 [27072] [5] INFO: Starting delivery report 
from 

Nonethless, while that log flows fluid, a tcpdump on the machine shows that
HTTP GET are delayed, like accumulated in batches. That is confirmed on the
other end where the HTTP GETs are received in batches.

When traffic is low, there are no losses. When traffic gets intense I
noticed losses in the DLRs on the HTTP receiving end.

This is the output of the svn info that produced the executable:

Path: .
URL: https://svn.kannel.org/gateway/trunk
Relative URL: ^/trunk
Repository Root: https://svn.kannel.org/gateway
Repository UUID: a7d65f57-29cd-40c9-b71e-ebca51ad1b1d
Revision: 5180
Node Kind: directory
Schedule: normal
Last Changed Author: amalysh
Last Changed Rev: 5180
Last Changed Date: 2017-01-10 18:34:44 +0100 (Tue, 10 Jan 2017)

Best regards,
Giulio



[image: Mobyt S.p.A.]  *Giulio Giovannini*
*Devops Mobyt*
giulio.giovann...@mobyt.it
*Mobyt S.p.A.*
Via Aldighieri 10
44121 Ferrara - Italy tel. + 39 0532 207296
fax. + 39 0532 242952
*www.mobyt.it* 

2017-08-24 17:51 GMT+02:00 Stipe Tolj :

> Am 10.08.2017 12:38, schrieb Giulio Giovannini:
>
>> Hi,
>>
>> nobody has anything to say about this issue?
>>
>> More details: I am seeing a difference of up to 8 seconds between the
>> logging of  DLR on smsbox log and the actual sending of the HTTP GET on
>> the network interface.
>>
>> The server the application is running on does not seem to have any
>> problem with the networking.
>>
>
> can you confirm that the DLRs are passed from bearerbox to smsbox in a
> reasonable time, and that the "delaying" is from smsbox's HTTP call
> perspective?
>
> BTW, no, there was no "caching" added, as we always keep the premises to
> transport ASAP and as fast as possible.
>
> --
> Best Regards,
> Stipe Tolj
>
> ---
> Düsseldorf, NRW, Germany
>
> Kannel Foundation tolj.org system architecture
> http://www.kannel.org/http://www.tolj.org/
>
> stolj at kannel.org   st at tolj.org
> ---
>
>

Re: Problem with GET HTTP

[Stipe Tolj]

Am 10.08.2017 12:38, schrieb Giulio Giovannini:

Hi,

nobody has anything to say about this issue?

More details: I am seeing a difference of up to 8 seconds between the
logging of  DLR on smsbox log and the actual sending of the HTTP GET on
the network interface.

The server the application is running on does not seem to have any
problem with the networking.


can you confirm that the DLRs are passed from bearerbox to smsbox in a 
reasonable time, and that the "delaying" is from smsbox's HTTP call 
perspective?


BTW, no, there was no "caching" added, as we always keep the premises to 
transport ASAP and as fast as possible.


--
Best Regards,
Stipe Tolj

---
Düsseldorf, NRW, Germany

Kannel Foundation tolj.org system architecture
http://www.kannel.org/http://www.tolj.org/

stolj at kannel.org   st at tolj.org
---

Re: Problem with GET HTTP

[Giulio Giovannini]

Hi,

nobody has anything to say about this issue?

More details: I am seeing a difference of up to 8 seconds between the
logging of  DLR on smsbox log and the actual sending of the HTTP GET on the
network interface.

The server the application is running on does not seem to have any problem
with the networking.

Best regards,
Giulio



[image: Mobyt S.p.A.]  *Giulio Giovannini*
*Devops Mobyt*
giulio.giovann...@mobyt.it
*Mobyt S.p.A.*
Via Aldighieri 10
44121 Ferrara - Italy tel. + 39 0532 207296
fax. + 39 0532 242952
*www.mobyt.it* 

2017-08-08 12:01 GMT+02:00 Giulio Giovannini :

> Hi all,
>
> am using a recent version of kannel and have problems with HTTP GETs
> executed after receiving a DLR.
>
> I got the code from svn but the VERSION file does not contain any version
> number. The first line only read: 'svn'. The HTTP status page says: Kannel
> bearerbox version `svn-runknown'. Build `May 8 2017 12:07:25'
>
> The problem is as follows: it looks like kannel is buffering or
> aggregating the HTTP GET that I have asked it to execute for each DLR. In
> doing so, DLRs can be delayed for various seconds (up to 5-6 seconds) or
> lost when traffic is high.
>
> This behaviour is different from other kannel instances I have build on Jan
> 23 2017 10:23:30. All instances share the same mysql DB for DLR storage,
> so, that can not be the problem.
>
> So, my question is: has any queue or caching system been introduced into
> smsbox HTTP GETs towards external resources? That would explain that
> behaviour and overflowed buffer could explain the lost DLRs.
>
> Best regards,
> Giulio Giovannnini
>

Problem with GET HTTP

[Giulio Giovannini]

Hi all,

am using a recent version of kannel and have problems with HTTP GETs
executed after receiving a DLR.

I got the code from svn but the VERSION file does not contain any version
number. The first line only read: 'svn'. The HTTP status page says: Kannel
bearerbox version `svn-runknown'. Build `May 8 2017 12:07:25'

The problem is as follows: it looks like kannel is buffering or aggregating
the HTTP GET that I have asked it to execute for each DLR. In doing so,
DLRs can be delayed for various seconds (up to 5-6 seconds) or lost when
traffic is high.

This behaviour is different from other kannel instances I have build on Jan
23 2017 10:23:30. All instances share the same mysql DB for DLR storage,
so, that can not be the problem.

So, my question is: has any queue or caching system been introduced into
smsbox HTTP GETs towards external resources? That would explain that
behaviour and overflowed buffer could explain the lost DLRs.

Best regards,
Giulio Giovannnini