Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Cody Herzog]

Thanks, Juha.

You are correct. That feature request is directly relevant to this thread.

I will upvote the feature request, and add a link back to this thread.

The proposed workaround of using async workers is the one I opted to use.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Juha Heinanen]

This issue may be related to the question:

https://github.com/kamailio/kamailio/issues/1107

-- Juha

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Cody Herzog]

Thanks.

Regarding UDP, that all makes sense.

For my short-term needs, to minimize risk and infrastructure changes, it looks 
like the ASYNC module will work.

Here's the basic outline of what I'm doing:


async_workers=8

route
{
route(CHECK_IS_EXPENSIVE_OPERATION);

if ($var(isExpensiveOperation))
{
async_task_route("COMMON");
}
else
{
route(COMMON);
}
}

route[COMMON]
{
# Do all my normal routing logic.
}


I detect at the very top of the main route whether the operation will be 
expensive.
If so, I process the message with an async worker.
If not, I directly process the message on the TCP worker.

The rest of the routing logic remains unchanged.
The only difference is that I execute the expensive operations using the async 
workers.

It's not the prettiest thing in the world, but it seems like an OK short-term 
fix to relieve pressure.
I will revisit the bigger architectural ideas in a later phase.

Thanks again for all your help and good ideas.

Happy Friday.
-Cody
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Alex Balashov]

On Fri, Feb 23, 2018 at 07:17:48PM +, Cody Herzog wrote:

> That makes sense, but is unfortunately not an option for me due to
> strict security requirements.  I need to use TLS on the whole path.

Personally, I would work around that requirement, either by using a
compliant private backplane/backbone network for internal communication,
or running UDP inside encrypted tunnels. That's the most promising
avenue in my opinion.

> Another option I explored was to have the edge proxies not always use the 
> same TCP connection for sending to the registrar.
> If I could find a way to load balance across a number of TCP
> connections, that would probably work for me.  Perhaps there is a way
> the DISPATHCER module can be configured to accomplish this.  Maybe the
> dispatcher configuration can list multiple copies of the same
> destination, but each having a different send socket address, and then
> can load balance across those.

It does, but unfortunately that level of fine-grained control isn't
reasonably possible. 

The only thing I can think of would be to have the registrar close the
TCP connection after receiving the registration. I don't know of a way
to do that except by changing the lifetime to something like zero after
the fact:

https://kamailio.org/docs/modules/5.1.x/modules/tcpops.html#tcpops.f.tcp_set_connection_lifetime

That would cause new connections to land at other workers, presumably.

But it's a kludgy solution. UDP is better.

-- Alex

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Cody Herzog]

>A common design which avoids this is to use TCP at the client edge and
>UDP inside the network core. This is one of the reasons why TCP is not
>optimal for use inside the core.

That makes sense, but is unfortunately not an option for me due to strict 
security requirements.
I need to use TLS on the whole path.

Another option I explored was to have the edge proxies not always use the same 
TCP connection for sending to the registrar.
If I could find a way to load balance across a number of TCP connections, that 
would probably work for me.
Perhaps there is a way the DISPATHCER module can be configured to accomplish 
this.
Maybe the dispatcher configuration can list multiple copies of the same 
destination, but each having a different send socket address, and then can load 
balance across those.
Does that make any sense?





___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Alex Balashov]

A common design which avoids this is to use TCP at the client edge and
UDP inside the network core. This is one of the reasons why TCP is not
optimal for use inside the core.

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Cody Herzog]

Thanks very much for the quick replies, Alex and Brandon.

The main reason I'm hitting a bottleneck is because my architecture is not 
optimal.

I have a number of edge proxies which communicate with all my clients.
The clients are usually distributed pretty evenly across all the proxies.
On those proxies, client TCP connections are distributed pretty evenly across 
the TCP workers, so that's all fine.
The problem occurs when the edge proxies communicate with the central registrar.

When that happens, the SIP messages from a very large number of clients are all 
multiplexed onto a single TCP socket connection between the proxy and the 
registrar.
That narrowing results in the registrar worker processes not being utilized 
efficiently.

For example, say I have 3 edge proxies, and my registrar has 8 cores and 8 
worker processes.
I want to spread the message processing across all 8 workers, but I'm only able 
to utilize 3, because all the messages from a given edge proxy are being 
processed by a single TCP worker on the registrar.

The long-term solution is to change my architecture.
I should not be doing all the expensive work on a single central registrar.
I'm planning to move to a mesh architecture, where all servers are the same, 
and the registration processing is divided amongst all the servers.
That design makes sense, but it's more complex and will require more 
information sharing amongst the servers.

I'm looking for a short-term improvement that will give me some breathing room, 
which led me to looking at the ASYNC module.

>...one alternate suggestion that could help spread load on actual Kamailio TCP 
>workers is by firing up additional workers on alternate ports.

That makes sense. I could have my central registrar listen on several different 
ports, and could perhaps use the dispatcher module on my edge proxies to try 
and evenly divide the traffic across those ports. I will look into that.

Thanks again.
-Cody
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Brandon Armstead]

Cody,

Kamailio should receive from normal TCP /
Kernel stack handoff - you may be able to do some tuning with sysctl -
however one alternate suggestion that could help spread load on actual
Kamailio TCP workers is by firing up additional workers on alternate ports
but this still would not ensure that any given worker is executed on unique
cpu cores.  With this said however the kernel should likely do a efficient
job of this already in the handoff between TCP stack and Kamailio.

Is there a reason you feel there is an efficiency issue?  Perhaps better
knowing any current limitations could help better provide suggestions for
your specific situation.

On Fri, Feb 23, 2018 at 10:01 AM Cody Herzog 
wrote:

> Hello.
>
>
>
> I'm curious as to how SIP messages received over TLS are divided amongst
> the available TCP workers.
>
>
>
> Based on some searching, I was hoping and expecting that the messages
> would be load balanced amongst the TCP workers at the level of individual
> SIP messages.
>
>
>
> However, with my configuration, it seems that the work is load balanced at
> the TCP connection level.
>
>
>
> I will now elaborate on my basic architecture.
>
>
>
> I have several edge proxies to which the clients connect. Those proxies do
> not handle registration, or do much of anything, they just forward traffic
> to a central registrar server.
>
>
>
> On the central registrar server, it seems that a particular TCP worker is
> assigned to handle all the traffic arriving from a particular edge proxy.
> As long as the TCP connection between the servers remains open, it seems
> that the particular worker is locked into handling all traffic on that
> connection.
>
>
>
> Does that sound like expected behavior?
>
>
>
> That behavior is not ideal for me, because I would like to evenly divide
> the SIP message load across all the TCP workers on my central registrar.
>
>
>
> Perhaps I'm doing something bad in my config file that is preventing the
> load balancing from working at the SIP message level.
>
>
>
> If that behavior is expected, is there anything I can do to achieve more
> even load balancing?
>
>
>
> Perhaps the ASYNC module might work for my  needs. It seems like I could
> use async_task_route() to divide certain messages evenly amongst the async
> workers.
>
>
>
> Does that sound reasonable?
>
>
>
> In my use case, certain messages require a lot of CPU time to perform
> crypto calculations, and I want those calculations to be distributed evenly
> across all CPU cores.
>
>
>
> Thanks.
>
> -Cody
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-- 
Sent from Gmail Mobile
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] How are SIP messages divided amongst TCP workers?

[Alex Balashov]

Hi,

As with UDP workers, the kernel divides incoming TCP in a semi-random
way at a trough of TCP workers all calling accept(). So, the
distribution is indeed at the connection level rather than the message
level, and so long as the connection persists, messages go to the same
worker.

In theory, at sufficiently large values of N, this shouldn't be a
problem since an approximately equal number of TCP connections will be
allocated to each worker. Or are your expensive crypto workloads
distributed in a very lopsided way that doesn't touch all TCP clients
approximately equally in the grand scheme of things?

If you're really concerned about it, though:

On Fri, Feb 23, 2018 at 06:00:22PM +, Cody Herzog wrote:

> Perhaps the ASYNC module might work for my  needs. It seems like I
> could use async_task_route() to divide certain messages evenly amongst
> the async workers.

This sounds like a reasonable solution.

-- Alex

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] How are SIP messages divided amongst TCP workers?

[Cody Herzog]

Hello.

I'm curious as to how SIP messages received over TLS are divided amongst the 
available TCP workers.

Based on some searching, I was hoping and expecting that the messages would be 
load balanced amongst the TCP workers at the level of individual SIP messages.

However, with my configuration, it seems that the work is load balanced at the 
TCP connection level.

I will now elaborate on my basic architecture.

I have several edge proxies to which the clients connect. Those proxies do not 
handle registration, or do much of anything, they just forward traffic to a 
central registrar server.

On the central registrar server, it seems that a particular TCP worker is 
assigned to handle all the traffic arriving from a particular edge proxy. As 
long as the TCP connection between the servers remains open, it seems that the 
particular worker is locked into handling all traffic on that connection.

Does that sound like expected behavior?

That behavior is not ideal for me, because I would like to evenly divide the 
SIP message load across all the TCP workers on my central registrar.

Perhaps I'm doing something bad in my config file that is preventing the load 
balancing from working at the SIP message level.

If that behavior is expected, is there anything I can do to achieve more even 
load balancing?

Perhaps the ASYNC module might work for my  needs. It seems like I could use 
async_task_route() to divide certain messages evenly amongst the async workers.

Does that sound reasonable?

In my use case, certain messages require a lot of CPU time to perform crypto 
calculations, and I want those calculations to be distributed evenly across all 
CPU cores.

Thanks.
-Cody
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users