[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
** Description changed: [Impact] Ruby processes can sometimes get stuck in a loop consuming 100% CPU, as described upstream and in the debian bug report. It has most commonly been seen in the puppet agent. [Test Case] It's not easy to reproduce. It has been suggested that this script eventually reproduces the problem: while nice -n19 /opt/puppetlabs/puppet/bin/ruby sched_yield_loop.rb; do :; done Where sched_yield_loop.rb comes from "https://bugs.debian.org/cgi- bin/bugreport.cgi?att=1;bug=876377;filename=sched_yield_loop.rb;msg=22" I personally haven't seen it happen with the script, but maybe it could take days. [Regression Potential] Races with threads can be hard to reproduce, and so can regressions. Patch has been applied upstream and to debian for more than a year now. [Other Info] - Not at this time. + Bionic has ruby 2.5.1 and it has this fix already, as do all later ubuntu releaes. [Original Description] Ubuntu 16.04 ruby 2.3.1-2~16.04.12 kernel 4.4.0-148-generic We've noticed an issue across multiple servers where puppet agent will seem to get stuck and consume 100% CPU for days or weeks on end until manually killed. root@ps-prod-jenkins-qa-ui02:~# ps aux|grep -i puppe[t] root 1412 0.0 0.2 143716 38680 ?Ssl Jun11 0:39 /usr/bin/ruby /usr/bin/puppet agent root 34884 74.4 0.3 286848 53724 ?Rs Jun23 1141:44 puppet agent: applying configuration root 111481 94.1 0.3 288572 54996 ?Rs Jun18 8642:32 puppet agent: applying configuration root 128479 54.8 0.3 286744 53596 ?Rs 10:30 250:17 puppet agent: applying configuration Strace shows it in a sched_yield() loop: root@ps-prod-jenkins-qa-ui02:~# strace -p 34884 -c strace: Process 34884 attached ^Cstrace: Process 34884 detached % time seconds usecs/call callserrors syscall -- --- --- - - 100.000.002130 0123189 sched_yield -- --- --- - - 100.000.002130123189 total Some googling shows this is a common issue which was supposedly fixed/backported to ruby 2.3: https://tickets.puppetlabs.com/browse/PA-1743 https://bugs.ruby-lang.org/issues/13794 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876377 The following open Ubuntu bugs look to be having the same issue and suggest that this fix made it into Debian but never made it into Ubuntu: https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1783161 https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1772998 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
In other words, we are waiting for the SRU team to go through the queue. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
Uploaded to the unapproved queue, next step is from the SRU team. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
** Description changed: [Impact] - * An explanation of the effects of the bug on users and - - * justification for backporting the fix to the stable release. - - * In addition, it is helpful, but not required, to include an -explanation of how the upload fixes this bug. + Ruby processes can sometimes get stuck in a loop consuming 100% CPU, as + described upstream and in the debian bug report. It has most commonly + been seen in the puppet agent. [Test Case] + It's not easy to reproduce. It has been suggested that this script eventually reproduces the problem: - * detailed instructions how to reproduce the bug + while nice -n19 /opt/puppetlabs/puppet/bin/ruby sched_yield_loop.rb; do + :; done - * these should allow someone who is not familiar with the affected -package to reproduce the bug and verify that the updated package fixes -the problem. + Where sched_yield_loop.rb comes from "https://bugs.debian.org/cgi- + bin/bugreport.cgi?att=1;bug=876377;filename=sched_yield_loop.rb;msg=22" + + I personally haven't seen it happen with the script, but maybe it could + take days. [Regression Potential] - - * discussion of how regressions are most likely to manifest as a result - of this change. - - * It is assumed that any SRU candidate patch is well-tested before -upload and has a low overall risk of regression, but it's important -to make the effort to think about what ''could'' happen in the -event of a regression. - - * This both shows the SRU team that the risks have been considered, -and provides guidance to testers in regression-testing the SRU. + Races with threads can be hard to reproduce, and so can regressions. + Patch has been applied upstream and to debian for more than a year now. [Other Info] - - * Anything else you think is useful to include - * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board - * and address these questions in advance + Not at this time. [Original Description] Ubuntu 16.04 ruby 2.3.1-2~16.04.12 kernel 4.4.0-148-generic We've noticed an issue across multiple servers where puppet agent will seem to get stuck and consume 100% CPU for days or weeks on end until manually killed. root@ps-prod-jenkins-qa-ui02:~# ps aux|grep -i puppe[t] root 1412 0.0 0.2 143716 38680 ?Ssl Jun11 0:39 /usr/bin/ruby /usr/bin/puppet agent root 34884 74.4 0.3 286848 53724 ?Rs Jun23 1141:44 puppet agent: applying configuration root 111481 94.1 0.3 288572 54996 ?Rs Jun18 8642:32 puppet agent: applying configuration root 128479 54.8 0.3 286744 53596 ?Rs 10:30 250:17 puppet agent: applying configuration Strace shows it in a sched_yield() loop: root@ps-prod-jenkins-qa-ui02:~# strace -p 34884 -c strace: Process 34884 attached ^Cstrace: Process 34884 detached % time seconds usecs/call callserrors syscall -- --- --- - - 100.000.002130 0123189 sched_yield -- --- --- - - 100.000.002130123189 total Some googling shows this is a common issue which was supposedly fixed/backported to ruby 2.3: https://tickets.puppetlabs.com/browse/PA-1743 https://bugs.ruby-lang.org/issues/13794 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876377 The following open Ubuntu bugs look to be having the same issue and suggest that this fix made it into Debian but never made it into Ubuntu: https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1783161 https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1772998 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
** Changed in: ruby2.3 (Ubuntu) Status: Triaged => In Progress ** Changed in: ruby2.3 (Ubuntu) Assignee: (unassigned) => Andreas Hasenack (ahasenack) ** Description changed: + [Impact] + + * An explanation of the effects of the bug on users and + + * justification for backporting the fix to the stable release. + + * In addition, it is helpful, but not required, to include an +explanation of how the upload fixes this bug. + + [Test Case] + + * detailed instructions how to reproduce the bug + + * these should allow someone who is not familiar with the affected +package to reproduce the bug and verify that the updated package fixes +the problem. + + [Regression Potential] + + * discussion of how regressions are most likely to manifest as a result + of this change. + + * It is assumed that any SRU candidate patch is well-tested before +upload and has a low overall risk of regression, but it's important +to make the effort to think about what ''could'' happen in the +event of a regression. + + * This both shows the SRU team that the risks have been considered, +and provides guidance to testers in regression-testing the SRU. + + [Other Info] + + * Anything else you think is useful to include + * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board + * and address these questions in advance + + [Original Description] + Ubuntu 16.04 - ruby 2.3.1-2~16.04.12 + ruby 2.3.1-2~16.04.12 kernel 4.4.0-148-generic We've noticed an issue across multiple servers where puppet agent will seem to get stuck and consume 100% CPU for days or weeks on end until manually killed. root@ps-prod-jenkins-qa-ui02:~# ps aux|grep -i puppe[t] root 1412 0.0 0.2 143716 38680 ?Ssl Jun11 0:39 /usr/bin/ruby /usr/bin/puppet agent root 34884 74.4 0.3 286848 53724 ?Rs Jun23 1141:44 puppet agent: applying configuration root 111481 94.1 0.3 288572 54996 ?Rs Jun18 8642:32 puppet agent: applying configuration root 128479 54.8 0.3 286744 53596 ?Rs 10:30 250:17 puppet agent: applying configuration - Strace shows it in a sched_yield() loop: root@ps-prod-jenkins-qa-ui02:~# strace -p 34884 -c strace: Process 34884 attached ^Cstrace: Process 34884 detached % time seconds usecs/call callserrors syscall -- --- --- - - 100.000.002130 0123189 sched_yield -- --- --- - - 100.000.002130123189 total - - Some googling shows this is a common issue which was supposedly fixed/backported to ruby 2.3: + Some googling shows this is a common issue which was supposedly + fixed/backported to ruby 2.3: https://tickets.puppetlabs.com/browse/PA-1743 https://bugs.ruby-lang.org/issues/13794 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876377 The following open Ubuntu bugs look to be having the same issue and suggest that this fix made it into Debian but never made it into Ubuntu: https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1783161 https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1772998 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
I was just waiting for some feedback, as this is hard to reproduce. I tried the script for a few minutes but nothing out of the ordinary happened. I see two +1s now, so I'll proceed with the SRU. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
The packages from the ppa finally built overnight, so you can also just add the ppa. Instructions are at https://launchpad.net/~ahasenack/+archive/ubuntu/ruby-cpu-busy- loop-1834072/, but it boils down to this command: sudo add-apt-repository ppa:ahasenack/ruby-cpu-busy-loop-1834072 Then: sudo apt-get update and the new ruby packages should be visible on your xenial system and ready to be upgraded via usual apt commands. -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
Thanks for your testing! -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
Thanks! The arm builders are slow, so apt won't see that ppa as populated just yet, but you can download the individual packages. Here are the amd64 ones, for example: https://launchpad.net/~ahasenack/+archive/ubuntu /ruby-cpu-busy-loop-1834072/+build/17190313 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
I built xenial packages in this ppa with the above patch, if you could give them a try: https://launchpad.net/~ahasenack/+archive/ubuntu/ruby-cpu-busy- loop-1834072/ -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
Upstream commit in the 2.3 branch: https://github.com/ruby/ruby/commit/c5a99cf3c0d493e911f499fdecdd0db68019af98 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
** Tags added: server-next ** Changed in: ruby2.3 (Ubuntu) Status: New => Triaged ** Changed in: ruby2.3 (Ubuntu) Importance: Undecided => High ** Bug watch added: Debian Bug tracker #876377 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876377 ** Also affects: ruby2.3 (Debian) via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876377 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1834072] Re: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu.
https://salsa.debian.org/ruby- team/ruby/commit/50d860d0bd7834e95214a2b1ff5b8e0ede7910a1 seems to be the fix If I build packages in a ppa, can you test for us, prior to the upload to proposed? -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to ruby2.3 in Ubuntu. https://bugs.launchpad.net/bugs/1834072 Title: Puppet agent using 100% CPU, in sched_yield() loop. Looks like an issue with ruby2.3 which has been fixed but not yet made it into Ubuntu. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ruby2.3/+bug/1834072/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
