Re: [SR-Users] Database connection handles vs. processes

2017-07-13 Thread Ovidiu Sas 
Try to map each connection to a process and then check which process
has extra connections and what type of process that is. It may shed
some light on this.

-ovidiu

On Thu, Jul 13, 2017 at 5:16 PM, Alex Balashov
 wrote:
> Hello,
>
> Sorry for the delay in following up on this.
>
> On Tue, Jul 11, 2017 at 09:20:23AM +0200, Daniel-Constantin Mierla wrote:
>
>> There are some rpc commands that open database connection, do the
>> operation, and then close it. Normally, it still should reuse an
>> existing connection, as the "open connection" should search for existing
>> one first.
>
> I do not think I am using any such RPC commands, however. I have a
> number of RPC commands that I run for routine monitoring and stats
> collection, but they all relate to dialogs, and I use runtime
> memory-only backing for that.
>
>> Can you see if all the connections are in open state or some are pending
>> close? Is the number of connections stable or fluctuates over the time?
>
> It appears to be a stable number and they are all established.
>
>> From your summary, I noticed that the tcp workers (which handle also
>> http) had two db connections, not sure it was only because of what you
>> selected to put in the mail or that's the rule.
>
> It does appear to be the rule, from a cursory examination.
>
> So, what I guess I am trying to figure out is whether this is normal,
> and what exactly is the driver of the greater-than-expected number of
> connections.
>
> -- Alex
>
> --
> Alex Balashov | Principal | Evariste Systems LLC
>
> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users



-- 
VoIP Embedded, Inc.
http://www.voipembedded.com

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-13 Thread Ovidiu Sas 
Try to map each connection to a process and then check which process has
extra connections and what type of process that is. It may shed some light
on this.

-ovidiu

On Jul 13, 2017 5:17 PM, "Alex Balashov"  wrote:

> Hello,
>
> Sorry for the delay in following up on this.
>
> On Tue, Jul 11, 2017 at 09:20:23AM +0200, Daniel-Constantin Mierla wrote:
>
> > There are some rpc commands that open database connection, do the
> > operation, and then close it. Normally, it still should reuse an
> > existing connection, as the "open connection" should search for existing
> > one first.
>
> I do not think I am using any such RPC commands, however. I have a
> number of RPC commands that I run for routine monitoring and stats
> collection, but they all relate to dialogs, and I use runtime
> memory-only backing for that.
>
> > Can you see if all the connections are in open state or some are pending
> > close? Is the number of connections stable or fluctuates over the time?
>
> It appears to be a stable number and they are all established.
>
> > From your summary, I noticed that the tcp workers (which handle also
> > http) had two db connections, not sure it was only because of what you
> > selected to put in the mail or that's the rule.
>
> It does appear to be the rule, from a cursory examination.
>
> So, what I guess I am trying to figure out is whether this is normal,
> and what exactly is the driver of the greater-than-expected number of
> connections.
>
> -- Alex
>
> --
> Alex Balashov | Principal | Evariste Systems LLC
>
> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-13 Thread Alex Balashov 
Hello,

Sorry for the delay in following up on this.

On Tue, Jul 11, 2017 at 09:20:23AM +0200, Daniel-Constantin Mierla wrote:

> There are some rpc commands that open database connection, do the
> operation, and then close it. Normally, it still should reuse an
> existing connection, as the "open connection" should search for existing
> one first.

I do not think I am using any such RPC commands, however. I have a
number of RPC commands that I run for routine monitoring and stats
collection, but they all relate to dialogs, and I use runtime
memory-only backing for that.

> Can you see if all the connections are in open state or some are pending
> close? Is the number of connections stable or fluctuates over the time?

It appears to be a stable number and they are all established.

> From your summary, I noticed that the tcp workers (which handle also
> http) had two db connections, not sure it was only because of what you
> selected to put in the mail or that's the rule.

It does appear to be the rule, from a cursory examination. 

So, what I guess I am trying to figure out is whether this is normal,
and what exactly is the driver of the greater-than-expected number of
connections.

-- Alex

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Daniel-Constantin Mierla 

On 11.07.17 09:13, Alex Balashov wrote:
> On Tue, Jul 11, 2017 at 09:12:51AM +0200, Daniel-Constantin Mierla wrote:
>
>> Hello,
>>
>>
>> On 11.07.17 08:49, Alex Balashov wrote:
>>> On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:
>>>
 are the connections to the same database or to different databases? Even
 on the same server, the database name matters.
>>> Hi,
>>>
>>> Same database, same server, same database name.
>>>
>> are you doing xmlrpc/jsonrpc over http?
> In fact, I am. xhttp + jsonrpcs is the only reason for the TCP listener.
>
There are some rpc commands that open database connection, do the
operation, and then close it. Normally, it still should reuse an
existing connection, as the "open connection" should search for existing
one first.

Can you see if all the connections are in open state or some are pending
close? Is the number of connections stable or fluctuates over the time?

From your summary, I noticed that the tcp workers (which handle also
http) had two db connections, not sure it was only because of what you
selected to put in the mail or that's the rule.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Daniel-Constantin Mierla 
No -- if the database url is the same, sqlops should reuse existing
connections.

Cheers,
Daniel


On 11.07.17 08:57, Alex Balashov wrote:
> I do use sqlops extensively, both in the main processes and rtimer
> processes. Does that make a difference?
>
> On Tue, Jul 11, 2017 at 02:49:19AM -0400, Alex Balashov wrote:
>
>> On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:
>>
>>> are the connections to the same database or to different databases? Even
>>> on the same server, the database name matters.
>> Hi,
>>
>> Same database, same server, same database name.
>>
>> -- 
>> Alex Balashov | Principal | Evariste Systems LLC
>>
>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>>
>> ___
>> Kamailio (SER) - Users Mailing List
>> sr-users@lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Alex Balashov 
On Tue, Jul 11, 2017 at 09:12:51AM +0200, Daniel-Constantin Mierla wrote:

> Hello,
> 
> 
> On 11.07.17 08:49, Alex Balashov wrote:
> > On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:
> >
> >> are the connections to the same database or to different databases? Even
> >> on the same server, the database name matters.
> > Hi,
> >
> > Same database, same server, same database name.
> >
> are you doing xmlrpc/jsonrpc over http?

In fact, I am. xhttp + jsonrpcs is the only reason for the TCP listener.

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Daniel-Constantin Mierla 
Hello,


On 11.07.17 08:49, Alex Balashov wrote:
> On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:
>
>> are the connections to the same database or to different databases? Even
>> on the same server, the database name matters.
> Hi,
>
> Same database, same server, same database name.
>
are you doing xmlrpc/jsonrpc over http?

Cheers,
Daniel

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Alex Balashov 
I do use sqlops extensively, both in the main processes and rtimer
processes. Does that make a difference?

On Tue, Jul 11, 2017 at 02:49:19AM -0400, Alex Balashov wrote:

> On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:
> 
> > are the connections to the same database or to different databases? Even
> > on the same server, the database name matters.
> 
> Hi,
> 
> Same database, same server, same database name.
> 
> -- 
> Alex Balashov | Principal | Evariste Systems LLC
> 
> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
> 
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Alex Balashov 
On Tue, Jul 11, 2017 at 08:41:24AM +0200, Daniel-Constantin Mierla wrote:

> are the connections to the same database or to different databases? Even
> on the same server, the database name matters.

Hi,

Same database, same server, same database name.

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-11 Thread Daniel-Constantin Mierla 
Hello,

are the connections to the same database or to different databases? Even
on the same server, the database name matters.

Cheers,
Daniel


On 10.07.17 22:23, Alex Balashov wrote:
> Hi,
>
> By way of illustration, I have one server with children=8, a single UDP
> listener, and 23 processes total:
>
> [...]
>
> So, this gives rise to two questions in my mind:
>
> 1. I don't seem to understand the relationship between the number of
> database connection handles and the total number of child processes. 
>
> I had always assumed that only SIP workers and other processes
> specialised into a role with possible "database involvement" (e.g.
> rtimer, async workers, etc.) would hold database connections. 
>
> 2. What's going on in the second scenario such that some processes have
> two connections?
>
> Both servers running:
>
> --
> # kamcmd -s /tmp/kamailio_ctl core.version
> kamailio 4.4.6 (x86_64/linux) becbde
> --
>
> Insights much appreciated!
>
> -- Alex
>

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com


___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-10 Thread Alex Balashov 
On Mon, Jul 10, 2017 at 05:04:32PM -0400, E. Schmidbauer wrote:

> some modules can have child processes that each maintain a database
> connection. the number of child processes for a module can sometimes be set
> using a modparam() for that module

Well, indeed, and that is germane.

However, it doesn't explain the disproportionate number of DB
connections relative to absolute number of child processes running, in
total, and why some processes appear to generate multiple connections.

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Database connection handles vs. processes

2017-07-10 Thread E. Schmidbauer 
some modules can have child processes that each maintain a database
connection. the number of child processes for a module can sometimes be set
using a modparam() for that module

On Mon, Jul 10, 2017 at 4:23 PM, Alex Balashov 
wrote:

> Hi,
>
> By way of illustration, I have one server with children=8, a single UDP
> listener, and 23 processes total:
>
> --
> 21117   attendant
> 21118   udp receiver child=0 sock=xxx.xxx.xxx.xxx:5060
> 21119   udp receiver child=1 sock=xxx.xxx.xxx.xxx:5060
> 21120   udp receiver child=2 sock=xxx.xxx.xxx.xxx:5060
> 21121   udp receiver child=3 sock=xxx.xxx.xxx.xxx:5060
> 21122   udp receiver child=4 sock=xxx.xxx.xxx.xxx:5060
> 21123   udp receiver child=5 sock=xxx.xxx.xxx.xxx:5060
> 21124   udp receiver child=6 sock=xxx.xxx.xxx.xxx:5060
> 21125   udp receiver child=7 sock=xxx.xxx.xxx.xxx:5060
> 21126   slow timer
> 21127   timer
> 21128   MI FIFO
> 21129   ctl handler
> 21130   RTIMER USEC EXEC
> 21131   RTIMER USEC EXEC
> 21132   RTIMER USEC EXEC
> 21133   RTIMER USEC EXEC
> 21134   RTIMER USEC EXEC
> 21135   RTIMER USEC EXEC
> 21136   RTIMER USEC EXEC
> 21137   Dialog KA Timer
> 21138   Dialog Clean Timer
> 21139   tcp main process
>
> # kamcmd -s /tmp/kamailio_ctl ps | wc -l
> 23
> --
>
> This appears to result in 23 total database connections to the Postgres
> server:
>
> --
> # lsof -n | grep -i '^kam' | grep IPv4 | grep TCP | grep -i :postgres | wc
> -l
> 23
> --
>
> But I have another Kamailio server that has three listeners, two UDP and
> one TCP, and children=24, i.e.
>
> --
> children=24
> listen=udp:xxx.xxx.xxx.xxx:5060
> listen=udp:yyy.yyy.yyy.yyy:5060
> listen=tcp:zzz.zzz.zzz.zzz:5060
> --
>
> I am given to understand that the total number of worker processes there
> will be (children_setting * listeners), i.e. 72 in this case, plus some
> number of rtimer and supervisory processes, as in my case. And true
> enough:
>
> --
> # kamcmd -s /tmp/kamailio_ctl ps | wc -l
> 86
> # kamcmd -s /tmp/kamailio_ctl ps | sed 's/65.254.44.195/yyy.yyy.yyy.yyy/g'
> | sed 's/65.254.44.194/zzz.zzz.zzz.zzz/g'
> 22065   main process - attendant
> 22069   udp receiver child=0 sock=zzz.zzz.zzz.zzz:5060
> 22070   udp receiver child=1 sock=zzz.zzz.zzz.zzz:5060
> 22071   udp receiver child=2 sock=zzz.zzz.zzz.zzz:5060
> 22073   udp receiver child=3 sock=zzz.zzz.zzz.zzz:5060
> 22074   udp receiver child=4 sock=zzz.zzz.zzz.zzz:5060
> 22076   udp receiver child=5 sock=zzz.zzz.zzz.zzz:5060
> 22079   udp receiver child=6 sock=zzz.zzz.zzz.zzz:5060
> 22080   udp receiver child=7 sock=zzz.zzz.zzz.zzz:5060
> 22082   udp receiver child=8 sock=zzz.zzz.zzz.zzz:5060
> 22085   udp receiver child=9 sock=zzz.zzz.zzz.zzz:5060
> 22086   udp receiver child=10 sock=zzz.zzz.zzz.zzz:5060
> 22089   udp receiver child=11 sock=zzz.zzz.zzz.zzz:5060
> 22091   udp receiver child=12 sock=zzz.zzz.zzz.zzz:5060
> 22093   udp receiver child=13 sock=zzz.zzz.zzz.zzz:5060
> 22095   udp receiver child=14 sock=zzz.zzz.zzz.zzz:5060
> 22097   udp receiver child=15 sock=zzz.zzz.zzz.zzz:5060
> 22099   udp receiver child=16 sock=zzz.zzz.zzz.zzz:5060
> 22100   udp receiver child=17 sock=zzz.zzz.zzz.zzz:5060
> 22103   udp receiver child=18 sock=zzz.zzz.zzz.zzz:5060
> 22105   udp receiver child=19 sock=zzz.zzz.zzz.zzz:5060
> 22107   udp receiver child=20 sock=zzz.zzz.zzz.zzz:5060
> 22109   udp receiver child=21 sock=zzz.zzz.zzz.zzz:5060
> 22111   udp receiver child=22 sock=zzz.zzz.zzz.zzz:5060
> 22113   udp receiver child=23 sock=zzz.zzz.zzz.zzz:5060
> 22115   udp receiver child=0 sock=yyy.yyy.yyy.yyy:5060
> 22117   udp receiver child=1 sock=yyy.yyy.yyy.yyy:5060
> 22119   udp receiver child=2 sock=yyy.yyy.yyy.yyy:5060
> 22121   udp receiver child=3 sock=yyy.yyy.yyy.yyy:5060
> 22123   udp receiver child=4 sock=yyy.yyy.yyy.yyy:5060
> 22125   udp receiver child=5 sock=yyy.yyy.yyy.yyy:5060
> 22127   udp receiver child=6 sock=yyy.yyy.yyy.yyy:5060
> 22129   udp receiver child=7 sock=yyy.yyy.yyy.yyy:5060
> 22131   udp receiver child=8 sock=yyy.yyy.yyy.yyy:5060
> 22132   udp receiver child=9 sock=yyy.yyy.yyy.yyy:5060
> 22135   udp receiver child=10 sock=yyy.yyy.yyy.yyy:5060
> 22137   udp receiver child=11 sock=yyy.yyy.yyy.yyy:5060
> 22139   udp receiver child=12 sock=yyy.yyy.yyy.yyy:5060
> 22141   udp receiver child=13 sock=yyy.yyy.yyy.yyy:5060
> 22143   udp receiver child=14 sock=yyy.yyy.yyy.yyy:5060
> 22145   udp receiver child=15 sock=yyy.yyy.yyy.yyy:5060
> 22147   udp receiver child=16 sock=yyy.yyy.yyy.yyy:5060
> 22149   udp receiver child=17 sock=yyy.yyy.yyy.yyy:5060
> 22151   udp receiver child=18 sock=yyy.yyy.yyy.yyy:5060
> 22153   udp receiver child=19 sock=yyy.yyy.yyy.yyy:5060
> 22155   udp receiver child=20 sock=yyy.yyy.yyy.yyy:5060
> 22156   udp receiver child=21 sock=yyy.yyy.yyy.yyy:5060
> 22159   udp receiver child=22 sock=yyy.yyy.yyy.yyy:5060
> 22161   udp receiver child=23 sock=yyy.yyy.yyy.yyy:5060
> 22163   slow timer
> 22165   timer
> 22167   secondary timer
> 22168   ctl handler
> 22169   RTIMER

[SR-Users] Database connection handles vs. processes

2017-07-10 Thread Alex Balashov 
Hi,

By way of illustration, I have one server with children=8, a single UDP
listener, and 23 processes total:

--
21117   attendant
21118   udp receiver child=0 sock=xxx.xxx.xxx.xxx:5060
21119   udp receiver child=1 sock=xxx.xxx.xxx.xxx:5060
21120   udp receiver child=2 sock=xxx.xxx.xxx.xxx:5060
21121   udp receiver child=3 sock=xxx.xxx.xxx.xxx:5060
21122   udp receiver child=4 sock=xxx.xxx.xxx.xxx:5060
21123   udp receiver child=5 sock=xxx.xxx.xxx.xxx:5060
21124   udp receiver child=6 sock=xxx.xxx.xxx.xxx:5060
21125   udp receiver child=7 sock=xxx.xxx.xxx.xxx:5060
21126   slow timer
21127   timer
21128   MI FIFO
21129   ctl handler
21130   RTIMER USEC EXEC
21131   RTIMER USEC EXEC
21132   RTIMER USEC EXEC
21133   RTIMER USEC EXEC
21134   RTIMER USEC EXEC
21135   RTIMER USEC EXEC
21136   RTIMER USEC EXEC
21137   Dialog KA Timer
21138   Dialog Clean Timer
21139   tcp main process

# kamcmd -s /tmp/kamailio_ctl ps | wc -l
23
--

This appears to result in 23 total database connections to the Postgres
server:

--
# lsof -n | grep -i '^kam' | grep IPv4 | grep TCP | grep -i :postgres | wc -l
23
--

But I have another Kamailio server that has three listeners, two UDP and
one TCP, and children=24, i.e.

--
children=24
listen=udp:xxx.xxx.xxx.xxx:5060
listen=udp:yyy.yyy.yyy.yyy:5060
listen=tcp:zzz.zzz.zzz.zzz:5060
--

I am given to understand that the total number of worker processes there
will be (children_setting * listeners), i.e. 72 in this case, plus some
number of rtimer and supervisory processes, as in my case. And true
enough:

--
# kamcmd -s /tmp/kamailio_ctl ps | wc -l
86
# kamcmd -s /tmp/kamailio_ctl ps | sed 's/65.254.44.195/yyy.yyy.yyy.yyy/g' | 
sed 's/65.254.44.194/zzz.zzz.zzz.zzz/g'
22065   main process - attendant
22069   udp receiver child=0 sock=zzz.zzz.zzz.zzz:5060
22070   udp receiver child=1 sock=zzz.zzz.zzz.zzz:5060
22071   udp receiver child=2 sock=zzz.zzz.zzz.zzz:5060
22073   udp receiver child=3 sock=zzz.zzz.zzz.zzz:5060
22074   udp receiver child=4 sock=zzz.zzz.zzz.zzz:5060
22076   udp receiver child=5 sock=zzz.zzz.zzz.zzz:5060
22079   udp receiver child=6 sock=zzz.zzz.zzz.zzz:5060
22080   udp receiver child=7 sock=zzz.zzz.zzz.zzz:5060
22082   udp receiver child=8 sock=zzz.zzz.zzz.zzz:5060
22085   udp receiver child=9 sock=zzz.zzz.zzz.zzz:5060
22086   udp receiver child=10 sock=zzz.zzz.zzz.zzz:5060
22089   udp receiver child=11 sock=zzz.zzz.zzz.zzz:5060
22091   udp receiver child=12 sock=zzz.zzz.zzz.zzz:5060
22093   udp receiver child=13 sock=zzz.zzz.zzz.zzz:5060
22095   udp receiver child=14 sock=zzz.zzz.zzz.zzz:5060
22097   udp receiver child=15 sock=zzz.zzz.zzz.zzz:5060
22099   udp receiver child=16 sock=zzz.zzz.zzz.zzz:5060
22100   udp receiver child=17 sock=zzz.zzz.zzz.zzz:5060
22103   udp receiver child=18 sock=zzz.zzz.zzz.zzz:5060
22105   udp receiver child=19 sock=zzz.zzz.zzz.zzz:5060
22107   udp receiver child=20 sock=zzz.zzz.zzz.zzz:5060
22109   udp receiver child=21 sock=zzz.zzz.zzz.zzz:5060
22111   udp receiver child=22 sock=zzz.zzz.zzz.zzz:5060
22113   udp receiver child=23 sock=zzz.zzz.zzz.zzz:5060
22115   udp receiver child=0 sock=yyy.yyy.yyy.yyy:5060
22117   udp receiver child=1 sock=yyy.yyy.yyy.yyy:5060
22119   udp receiver child=2 sock=yyy.yyy.yyy.yyy:5060
22121   udp receiver child=3 sock=yyy.yyy.yyy.yyy:5060
22123   udp receiver child=4 sock=yyy.yyy.yyy.yyy:5060
22125   udp receiver child=5 sock=yyy.yyy.yyy.yyy:5060
22127   udp receiver child=6 sock=yyy.yyy.yyy.yyy:5060
22129   udp receiver child=7 sock=yyy.yyy.yyy.yyy:5060
22131   udp receiver child=8 sock=yyy.yyy.yyy.yyy:5060
22132   udp receiver child=9 sock=yyy.yyy.yyy.yyy:5060
22135   udp receiver child=10 sock=yyy.yyy.yyy.yyy:5060
22137   udp receiver child=11 sock=yyy.yyy.yyy.yyy:5060
22139   udp receiver child=12 sock=yyy.yyy.yyy.yyy:5060
22141   udp receiver child=13 sock=yyy.yyy.yyy.yyy:5060
22143   udp receiver child=14 sock=yyy.yyy.yyy.yyy:5060
22145   udp receiver child=15 sock=yyy.yyy.yyy.yyy:5060
22147   udp receiver child=16 sock=yyy.yyy.yyy.yyy:5060
22149   udp receiver child=17 sock=yyy.yyy.yyy.yyy:5060
22151   udp receiver child=18 sock=yyy.yyy.yyy.yyy:5060
22153   udp receiver child=19 sock=yyy.yyy.yyy.yyy:5060
22155   udp receiver child=20 sock=yyy.yyy.yyy.yyy:5060
22156   udp receiver child=21 sock=yyy.yyy.yyy.yyy:5060
22159   udp receiver child=22 sock=yyy.yyy.yyy.yyy:5060
22161   udp receiver child=23 sock=yyy.yyy.yyy.yyy:5060
22163   slow timer
22165   timer
22167   secondary timer
22168   ctl handler
22169   RTIMER USEC EXEC
22170   RTIMER USEC EXEC
22172   RTIMER USEC EXEC
22173   RTIMER USEC EXEC
22175   Dialog Clean Timer
22176   TIMER NH
22177   TIMER NH
22178   TIMER NH
22179   tcp receiver (generic) child=0
22180   tcp receiver (generic) child=1
22182   tcp receiver (generic) child=2
22184   tcp receiver (generic) child=3
22186   tcp receiver (generic) child=4
22188   tcp receiver (generic) child=5
22190   tcp receiver (generic) child=6
22192   tcp receiver (generic) child=7
22194   tcp