Hi,
Some time in late 2014, I reported a similar problem with async workers
causing high CPU usage in Xen and KVM environments:
https://lists.kamailio.org/pipermail/sr-users/2014-October/085432.html
and I think we had a similar discussion. These async workers were just
sitting in their normal event loop.
This was the main reason why I ended up implementing a lot of async
tasks in the more tedious way with mqueue + rtimer; rtimer processes
blocking on mqueue did not seem to have this problem.
I never really followed up to see if async workers behave better now,
but since the vast majority of my installations run in a hypervisor, I
didn't want to write a lot of code that uses the conventional
convenience wrappers and take the risk.
-- Alex
On 5/26/20 3:30 AM, Daniel-Constantin Mierla wrote:
The load average increase was at runtime, but with the async tasks not
actually doing anything.
I found a really good article explaining the load average computation --
I haven't read it thoroughly yet, but is very informative:
* http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
So it is not only measured the need for CPU, but also the
uninterruptible tasks, which in the past was the need for disk I/O, but
nowadays can be more than that.
The async workers wait on some internal sockets to read the data of the
tasks to execute. It uses recvfrom() which is I/O operation and normally
should not increase the load.
Maybe in a hypervised environment there are some signals waking the
readers of internal sockets, or the kernel there counts this operation
to be "uninterruptible task". That's why I was curios to see if any
other OS/kernel exposes the same situation. On Debian I haven't noticed
high load although I have some deployments with async workers doing same
rare operations.
Cheers,
Daniel
On 25.05.20 22:37, Sergiu Pojoga wrote:
Is Kamailio running in a hypervised environment? If so, I've seen
async workers cause high load at runtime, don't recall boot time.
On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla
<[email protected] <mailto:[email protected]>> wrote:
Hello,
the async task workers are in recvfom(), which should not increase
any load.
Do you have any chance to test on another os/version? Maybe on
centos 7 and see if it is the same case?
Cheers,
Daniel
On 25.05.20 17:56, Володимир Іванець wrote:
Hello again,
I attached a new file.
The interesting part is that Kamailio does not load the CPU at
all. /Top/ shows it at the bottom. Only the "load average" value
gets increased.
Thank you!
пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla
<[email protected] <mailto:[email protected]>> пише:
Hello,
can you install the package with kamailio debugging symbols?
Then take again the kamctl trap with two async workers, it
should contain more details about what pieces of code run.
The package should be named like kamailio-dbg...
Besides that, can you also do a 'top' and see what kamailio
processes (their PIDs) eat a lot of cpu?
Cheers,
Daniel
On 25.05.20 11:05, Володимир Іванець wrote:
Hello,
Attached are two files. One for 2 Async Task Workers and one
for 8 Workers. The second one was stuck and did not complete.
I should point out that the virtual machine has 2 CPU cores.
Load average value was stable with 2 workers and was slowly
increasing after adding more workers. * workers caused it to
increase very fast.
Thank you!
пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla
<[email protected] <mailto:[email protected]>> пише:
Hello,
if you can, it would be interesting to get the backtrace
and see what was causing the load.
Iirc, the Async Task Worker should wait on read on an
internal socket, so it should be no CPU used when
nothing is transmitted to this type of workers.
Cheers,
Daniel
On 22.05.20 19:20, Володимир Іванець wrote:
Hello Daniel,
Thank you for your response.
I run /kamctl trap/ command but the procedure got
stuck. Last line in the generated file contained
"---start 12767 -----". 12767 was an Async Task
Worker. Since I don't need them I just removed related
configuration. It must be left after the testing. This
solved the problem.
Please let me know if you are still interested in what
was going on and if I should restore the configuration
and run /kamctl trap/ again.
Thank you very much!
пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla
<[email protected] <mailto:[email protected]>> пише:
Hello,
install gdb and, when the load is high, run:
kamctl trap
It write a file with what kamailio was doing at
that moment. Send it over here on mailing list or
make it available for download somewhere. We can
look at it and guide further about what can be done.
Cheers,
Daniel
On 22.05.20 16:53, Володимир Іванець wrote:
Hello everyone!
I'm running Kamailio version 5.3.3 on a CentOS 6.
I started noticing that "load average" value
increases rapidly with the start of Kamailio:
# uptime
17:47:52 up 4 days, 17:47, 3 users, load
average: 7.02, 7.01, 6.02
It will start to decrease immediately after
Kamailio is stopped.
Does anyone know what could cause this and how to
troubleshoot it?
Thank you!
_______________________________________________
Kamailio (SER) - Users Mailing List
[email protected]
<mailto:[email protected]>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda>
--www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Funding:https://www.paypal.me/dcmierla
--
Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda>
--www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Funding:https://www.paypal.me/dcmierla
--
Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda>
--www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Funding:https://www.paypal.me/dcmierla
--
Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda>
--www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Funding:https://www.paypal.me/dcmierla
_______________________________________________
Kamailio (SER) - Users Mailing List
[email protected] <mailto:[email protected]>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
Kamailio (SER) - Users Mailing List
[email protected]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla --www.asipto.com
www.twitter.com/miconda --www.linkedin.com/in/miconda
Funding:https://www.paypal.me/dcmierla
_______________________________________________
Kamailio (SER) - Users Mailing List
[email protected]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________
Kamailio (SER) - Users Mailing List
[email protected]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
