Hi Diego,
I forgot to tell that opening large number of tcp sockets means opening large
amount of file descriptors.
echo 128000 > /proc/sys/fs/inode-max
echo 64000 > /proc/sys/fs/file-max
ulimit -n 64000
// Binan
________________________________
Från: Diego Barberio <[email protected]>
Till: Binan AL Halabi <[email protected]>; OpenSIPS users mailling list
<[email protected]>
Skickat: fredag, 12 oktober 2012 1:38
Ämne: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response under
high load
Hi Binnan,
Thanks for all your help.
I've applied all you suggestions and nothing changed.
I generated a trace of OpenSIPs connections using TCP dump (attached). I
noticed that the send queues have a lot of data between 6 pm and 8 pm (when is
the period the problem appears).
As far as I know this means opensips is calling send funcion but the OS is not
actually sending data, am I right?
What can be causing this problem? Do you have any other ideas?
Again, thanks a lot for your help.
Regards
Diego
On Tue, Oct 9, 2012 at 7:20 PM, Binan AL Halabi <[email protected]> wrote:
>
>Do the following to increase TCP performance:
>
>
>1- Increase the number of available local ports:
>
> echo 1024 65000 > /proc/sys/net/ipv4/ip_local_port_range
>
>
>2- Increase the amount of memory associated with socket buffers (socket input
>and output queues):
>For input queues:
>
> echo 262143 > /proc/sys/net/core/rmem_max
>
> echo 262143 > /proc/sys/net/core/rmem_default
>And wmem_max , wmem_default for output queues.
>
>
>3- TCP stack
>
> echo 0 > /proc/sys/net/ipv4/tcp_sack
>
> echo 0 > /proc/sys/net/ipv4/tcp_timestamps
>
>
>4- opensips memory pool is well configured :
>http://www.opensips.org/Resources/DocsTsMem
>
>
>// Binan
>
>
>________________________________
> Från: Diego Barberio <[email protected]>
>Till: OpenSIPS users mailling list <[email protected]>
>Skickat: tisdag, 9 oktober 2012 20:08
>Ämne: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK response
>under high load
>
>
>
>Hi Binan,
>
>Thank's to Bogdan's patch I was able to run opensips 1.8.1. However, the
>problem with TCP under highload is still present.
>
>Do you have any other ideas?
>
>Thanks
>Diego
>
>
>On Mon, Sep 10, 2012 at 4:59 PM, Diego Barberio
><[email protected]> wrote:
>
>Hi Binan,
>>
>>I tried upgrading to 1.8.1 but I'm having some issues I'm duscussin with
>>Bogdan on the following thread:
>>
>>http://lists.opensips.org/pipermail/users/2012-August/022764.html
>>
>>Thanks
>>Diego
>>
>>
>>
>>On Mon, Sep 10, 2012 at 4:47 PM, Binan AL Halabi <[email protected]>
>>wrote:
>>
>>hi Diego,
>>>
>>>1- As you dont have state in database so you dont need to check that.
>>>2- Why you dont upgrade to opensips 1.8.1 ? since it contains TCP fix.
>>>
>>>Regards.
>>>//Binan
>>>
>>>--- On Mon, 9/10/12, Binan AL Halabi <[email protected]> wrote:
>>>
>>>
>>>>From: Binan AL Halabi <[email protected]>
>>>>
>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
>>>>response under high load
>>>>To: "OpenSIPS users mailling list" <[email protected]>
>>>>Date: Monday, September 10, 2012, 12:19 PM
>>>>
>>>>
>>>>
>>>>If you have
>>>>
>>>>--- On Mon, 9/10/12, Diego Barberio <[email protected]>
>>>>wrote:
>>>>
>>>>
>>>>>From: Diego Barberio <[email protected]>
>>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
>>>>>response under high load
>>>>>To: "OpenSIPS users mailling list" <[email protected]>
>>>>>Date: Monday, September 10, 2012, 10:51 AM
>>>>>
>>>>>
>>>>>Hi Binan,
>>>>>
>>>>>I understand what do you say with "play around TCP connection lifetime"
>>>>>however I don't get the part you say "check the database". I don't have
>>>>>any database, opensips is working without any DB.
>>>>>Which database are you talking about?
>>>>>
>>>>>Thanks
>>>>>Diego
>>>>>
>>>>>
>>>>>On Mon, Sep 10, 2012 at 1:37 PM, Binan AL Halabi <[email protected]>
>>>>>wrote:
>>>>>
>>>>>Hi Diego,
>>>>>>play around TCP connection lifetime, you could find something.
>>>>>>
>>>>>>one thing more check the database during the busy hour and see if
>>>>>>something expired.
>>>>>>
>>>>>>//Binan
>>>>>>
>>>>>>
>>>>>>--- On Mon, 9/10/12, Diego Barberio <[email protected]>
>>>>>>wrote:
>>>>>>
>>>>>>
>>>>>>>From: Diego Barberio <[email protected]>
>>>>>>>Subject: Re: [OpenSIPS-Users] FW: Opensips 1.6.4 doesn't send an OK
>>>>>>>response under high load
>>>>>>>To: "OpenSIPS users mailling list" <[email protected]>
>>>>>>>Date: Monday, September 10, 2012, 6:39 AM
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Hi All,
>>>>>>>
>>>>>>>I'm still having this issue. However, I went on with my investigation
>>>>>>>and I've discovered that this issue only happens with my SIP over TCP
>>>>>>>connections
(I have 10 TCP connections)
>>>>>>>I've upgraded to version:
>>>>>>>Server:: OpenSIPS (1.7.2-notls (i386/linux))
>>>>>>>
>>>>>>>And the issue is still there.
>>>>>>>
>>>>>>>
>>>>>>>Do you have any other ideas? (Disabling TCP is not an option)
>>>>>>>
>>>>>>>Thanks
>>>>>>>Diego
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: [email protected]
>>>>>>>>[mailto:[email protected]] On Behalf Of Ali Pey
>>>>>>>>Sent: viernes, 10 de agosto de 2012 01:30 p.m.
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I can't think of anything else.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Dear opensips team,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hope you had a good flight-terminator :)
>>>>>>>>
>>>>>>>>Can you take a look at this please?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>On Fri, Aug 10, 2012 at 12:05 PM, Diego Barberio
>>>>>>>><[email protected]> wrote:
>>>>>>>>
>>>>>>>>Hi Ali,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks again for rour feedbak.
>>>>>>>>
>>>>>>>>Unfortunately that's not the case. I'm attaching an example with the
>>>>>>>>same
>>>>>>>>Via that works perfectly.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Also, opensips works perfectly all day and only stops sending some OKs
>>>>>>>>during peak hour.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Do you have any other idea?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: [email protected]
>>>>>>>>[mailto:[email protected]] On Behalf Of Ali Pey
>>>>>>>>Sent: viernes, 10 de agosto de 2012 11:00 a.m.
>>>>>>>>
>>>>>>>>
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hi Diego,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I looked at your pcap file and I see that your second via has this:
>>>>>>>>
>>>>>>>>BC01.MBCSCR4.MSS.IMS.TIGO.COM.GT
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Is this a valid address? Maybe opensips can not resolve it so it
>>>>>>>>ignores the
>>>>>>>>via. Have you tried adding it to /etc/hosts.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Compare a good 200OK with one of these and see if there is any
>>>>>>>>difference.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>On Fri, Aug 10, 2012 at 9:20 AM, Diego Barberio
>>>>>>>><[email protected]> wrote:
>>>>>>>>
>>>>>>>>Hi Ali,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>I removed set the debug flag to 0 and added the "-" before the file
>>>>>>>>path,
>>>>>>>>and I sill have the same issue.
>>>>>>>>
>>>>>>>>Is there any other thing I can try?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>From: [email protected]
>>>>>>>>[mailto:[email protected]] On Behalf Of Ali Pey
>>>>>>>>Sent: jueves, 09 de agosto de 2012 03:30 p.m.
>>>>>>>>To: OpenSIPS users mailling list
>>>>>>>>Subject: Re: [OpenSIPS-Users] Opensips 1.6.4 doesn't send an OK response
>>>>>>>>under high load
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Hi Diego,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Have you tried disabling the logging? That's usually the problem. Syslog
>>>>>>>>slows down opensips.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Also in /etc/syslog.conf as '-' before the file path to make it
>>>>>>>>asynchronous.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>
>>>>>>>>Ali Pey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>On Thu, Aug 9, 2012 at 2:19 PM, Diego Barberio
>>>>>>>><[email protected]> wrote:
>>>>>>>>
>>>>>>>>Hello,
>>>>>>>>
>>>>>>>>I'm using opensips to forward incoming INVITEs to a set of 8
>>>>>>>>applications.
>>>>>>>>I'm using dispatcher module to achieve this.
>>>>>>>>During peak hour, opensips receives about 350 INVITEs per second, during
>>>>>>>>this period about 30 OK responses per second my application sends
>>>>>>>>correctly
>>>>>>>>aren't forwarded by opensips.
>>>>>>>>
>>>>>>>>I've recompiled opensips so it can use more memory by modifing the
>>>>>>>>config.h
>>>>>>>>file:
>>>>>>>>
>>>>>>>>#define PKG_MEM_POOL_SIZE 10*1024*1024 /*!< Used only if
>>>>>>>>PKG_MALLOC
>>>>>>>>is defined*/
>>>>>>>>#define SHM_MEM_SIZE 320 /*!< Used if SH_MEM is
>>>>>>>>defined*/
>>>>>>>>
>>>>>>>>
>>>>>>>>The incoming INVITEs come from UDP and TCP.
>>>>>>>>
>>>>>>>>
>>>>>>>>For each of the OK responses that aren't forwarded this entry appears
>>>>>>>>in the
>>>>>>>>messages file:
>>>>>>>>
>>>>>>>>usr/local/sbin/opensips[24554]: ERROR:core:forward_reply: no 2nd via
>>>>>>>>found
>>>>>>>>in reply
>>>>>>>>
>>>>>>>>According to my research on the web this message means that the OK only
>>>>>>>>has
>>>>>>>>one Via header, however this is not the case and the capture attached
>>>>>>>>(made
>>>>>>>>on server which runs opensips) shows it.
>>>>>>>>
>>>>>>>>I've also tried to increase the log level, but it made opensips to work
>>>>>>>>slow
>>>>>>>>even in low traffic hours.
>>>>>>>>
>>>>>>>>
>>>>>>>>I run some test in my lab and opensips worked perfectly even under more
>>>>>>>>than
>>>>>>>>350 INVITES per second, this problem only occurs in production
>>>>>>>>
>>>>>>>>I really don't know what else to do.
>>>>>>>>Could you help me to fix this issue or at least move on with the
>>>>>>>>investigation?
>>>>>>>>
>>>>>>>>Thanks
>>>>>>>>Diego
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>[email protected]
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>[email protected]
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>_______________________________________________
>>>>>>>>Users mailing list
>>>>>>>>[email protected]
>>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>-----Inline Attachment Follows-----
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>Users mailing list
>>>>>>>[email protected]
>>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>>
>>>>>>_______________________________________________
>>>>>>Users mailing list
>>>>>>[email protected]
>>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>>
>>>>>>
>>>>>
>>>>>-----Inline Attachment Follows-----
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>Users mailing list
>>>>>[email protected]
>>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>-----Inline Attachment Follows-----
>>>>
>>>>
>>>>_______________________________________________
>>>>Users mailing list
>>>>[email protected]
>>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>>
>>>_______________________________________________
>>>Users mailing list
>>>[email protected]
>>>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>
>_______________________________________________
>Users mailing list
>[email protected]
>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
>_______________________________________________
>Users mailing list
>[email protected]
>http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
