The load average increase was at runtime, but with the async tasks not actually doing anything.
I found a really good article explaining the load average computation -- I haven't read it thoroughly yet, but is very informative: * http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html So it is not only measured the need for CPU, but also the uninterruptible tasks, which in the past was the need for disk I/O, but nowadays can be more than that. The async workers wait on some internal sockets to read the data of the tasks to execute. It uses recvfrom() which is I/O operation and normally should not increase the load. Maybe in a hypervised environment there are some signals waking the readers of internal sockets, or the kernel there counts this operation to be "uninterruptible task". That's why I was curios to see if any other OS/kernel exposes the same situation. On Debian I haven't noticed high load although I have some deployments with async workers doing same rare operations. Cheers, Daniel On 25.05.20 22:37, Sergiu Pojoga wrote: > Is Kamailio running in a hypervised environment? If so, I've seen > async workers cause high load at runtime, don't recall boot time. > > On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla > <[email protected] <mailto:[email protected]>> wrote: > > Hello, > > the async task workers are in recvfom(), which should not increase > any load. > > Do you have any chance to test on another os/version? Maybe on > centos 7 and see if it is the same case? > > Cheers, > Daniel > > On 25.05.20 17:56, Володимир Іванець wrote: >> Hello again, >> >> I attached a new file. >> >> The interesting part is that Kamailio does not load the CPU at >> all. /Top/ shows it at the bottom. Only the "load average" value >> gets increased. >> >> Thank you! >> >> пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla >> <[email protected] <mailto:[email protected]>> пише: >> >> Hello, >> >> can you install the package with kamailio debugging symbols? >> Then take again the kamctl trap with two async workers, it >> should contain more details about what pieces of code run. >> >> The package should be named like kamailio-dbg... >> >> Besides that, can you also do a 'top' and see what kamailio >> processes (their PIDs) eat a lot of cpu? >> >> Cheers, >> Daniel >> >> On 25.05.20 11:05, Володимир Іванець wrote: >>> Hello, >>> >>> Attached are two files. One for 2 Async Task Workers and one >>> for 8 Workers. The second one was stuck and did not complete. >>> >>> I should point out that the virtual machine has 2 CPU cores. >>> Load average value was stable with 2 workers and was slowly >>> increasing after adding more workers. * workers caused it to >>> increase very fast. >>> >>> Thank you! >>> >>> пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla >>> <[email protected] <mailto:[email protected]>> пише: >>> >>> Hello, >>> >>> if you can, it would be interesting to get the backtrace >>> and see what was causing the load. >>> >>> Iirc, the Async Task Worker should wait on read on an >>> internal socket, so it should be no CPU used when >>> nothing is transmitted to this type of workers. >>> >>> Cheers, >>> Daniel >>> >>> On 22.05.20 19:20, Володимир Іванець wrote: >>>> Hello Daniel, >>>> >>>> Thank you for your response. >>>> >>>> I run /kamctl trap/ command but the procedure got >>>> stuck. Last line in the generated file contained >>>> "---start 12767 -----". 12767 was an Async Task >>>> Worker. Since I don't need them I just removed related >>>> configuration. It must be left after the testing. This >>>> solved the problem. >>>> >>>> Please let me know if you are still interested in what >>>> was going on and if I should restore the configuration >>>> and run /kamctl trap/ again. >>>> >>>> Thank you very much! >>>> >>>> пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla >>>> <[email protected] <mailto:[email protected]>> пише: >>>> >>>> Hello, >>>> >>>> install gdb and, when the load is high, run: >>>> >>>> kamctl trap >>>> >>>> It write a file with what kamailio was doing at >>>> that moment. Send it over here on mailing list or >>>> make it available for download somewhere. We can >>>> look at it and guide further about what can be done. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 22.05.20 16:53, Володимир Іванець wrote: >>>>> Hello everyone! >>>>> >>>>> I'm running Kamailio version 5.3.3 on a CentOS 6. >>>>> I started noticing that "load average" value >>>>> increases rapidly with the start of Kamailio: >>>>> >>>>> # uptime >>>>> 17:47:52 up 4 days, 17:47, 3 users, load >>>>> average: 7.02, 7.01, 6.02 >>>>> >>>>> It will start to decrease immediately after >>>>> Kamailio is stopped. >>>>> >>>>> Does anyone know what could cause this and how to >>>>> troubleshoot it? >>>>> >>>>> Thank you! >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> [email protected] >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.com >>>> <http://www.asipto.com> >>>> www.twitter.com/miconda <http://www.twitter.com/miconda> >>>> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >>>> Funding: https://www.paypal.me/dcmierla >>>> >>> -- >>> Daniel-Constantin Mierla -- www.asipto.com >>> <http://www.asipto.com> >>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >>> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >>> Funding: https://www.paypal.me/dcmierla >>> >> -- >> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> >> www.twitter.com/miconda <http://www.twitter.com/miconda> -- >> www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> >> Funding: https://www.paypal.me/dcmierla >> > -- > Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> -- > www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Funding: https://www.paypal.me/dcmierla > > _______________________________________________ > Kamailio (SER) - Users Mailing List > [email protected] <mailto:[email protected]> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > _______________________________________________ > Kamailio (SER) - Users Mailing List > [email protected] > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
