[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Great to hear. This bug will be fixed then I guess: https://youtrack.jetbrains.com/issue/UP-6943 Thanks a lot! -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Released Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Fix Released Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
> Will my systemd scripts with TasksMax setting keep working? Yes, this only changes the builtin defaults if there is no explicit TaskMax= setting. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Released Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Fix Released Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Will my systemd scripts with TasksMax setting keep working? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Released Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Fix Released Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
This bug was fixed in the package systemd - 229-4ubuntu6 --- systemd (229-4ubuntu6) xenial-proposed; urgency=medium * Add a DHCP exit hook for pushing received NTP servers into timesyncd. (LP: #1578663) * Revert "enable TasksMax= for all services by default, and set it to 512". Introducing a default limit on number of threads broke a lot of software which regularly needs more, such as MySQL and RabbitMQ, or services that spawn off an indefinite number of subtasks that are not in a scope, like LXC or cron. 512 is way too much for most "simple" services, and it's way too little for the ones mentioned above. Effective (and much stricter) limits should instead be put into units individually. (Closes: #823530, LP: #1578080) * debian/gbp.conf: Switch to ubuntu-xenial branch. -- Martin Pitt Thu, 12 May 2016 10:39:30 +0200 ** Changed in: systemd (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Released Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Fix Released Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
** Changed in: systemd (Debian) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Committed Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Fix Released Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
I've upgraded systemd across my cluster to 229-4ubuntu6, removed all my custom tweaks to systemd settings, reloaded the daemon and both rabbitmq and mysql appear to be working fine on my openstack cluster. I'll be throwing a bit more load at it, but usually by this point mysql has fallen over, so I'd say this is a success. ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Committed Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
This bug was fixed in the package systemd - 229-6ubuntu1 --- systemd (229-6ubuntu1) yakkety; urgency=medium * Merge with Debian unstable. Remaining Ubuntu changes: - Hack to support system-image read-only /etc, and modify files in /etc/writable/ instead. systemd (229-6) unstable; urgency=medium * systemd-container: Prefer renamed "btrfs-progs" package name over "btrfs-tools". (Closes: #822629) * systemd-container: Recommend libnss-mymachines. (Closes: #822615) * Drop systemd-dbg, in favor of debhelpers' automatic -dbgsym packages. * Drop Add-targets-for-compatibility-with-Debian-insserv-sy.patch; we don't need $x-display-manager any more as most/all DMs ship native services, and $mail-transport-agent is not widely used (not even by our default MTA exim4). * Unify our two patches for Debian specific configuration files. * Drop udev-re-enable-mount-propagation-for-udevd.patch, i. e. run udevd in its own slave mount name space again. laptop-mode-tools 1.68 fixed the original bug (#762018), thus add a Breaks: to earlier versions. * Ship fbdev-blacklist.conf in /lib/modprobe.d/ instead of /etc/modprobe.d/; remove the conffile on upgrades. * Replace util-Add-hidden-suffixes-for-ucf.patch with patch that got committed upstream. * Replace Stop-syslog.socket-when-entering-emergency-mode.patch with patch that got committed upstream. * debian/udev.README.Debian: Adjust documentation of MAC based naming for USB network cards to the udev rule, where this was moved to in 229-5. * debian/extra/init-functions.d/40-systemd: Invoke status command with --no-pager, to avoid blocking scripts that call an init.d script with "status" with an unexpected pager process. (Closes: #765175, LP: #1576409) * Add debian/extra/rules/70-debian-uaccess.rules: Make FIDO U2F dongles accessible to the user session. This avoids having to install libu2f-host0 (which isn't discoverable at all) to make those devices work. (LP: #1387908) * libnss-resolve: Enable systemd-resolved.service on package installation, as this package makes little sense without resolved. * Add a DHCP exit hook for pushing received NTP servers into timesyncd. (LP: #1578663) * debian/udev.postinst: Fix migration check from the old persistent-net generator to not apply to chroots. (Closes: #813141) * Revert "enable TasksMax= for all services by default, and set it to 512". Introducing a default limit on number of threads broke a lot of software which regularly needs more, such as MySQL and RabbitMQ, or services that spawn off an indefinite number of subtasks that are not in a scope, like LXC or cron. 512 is way too much for most "simple" services, and it's way too little for the ones mentioned above. Effective (and much stricter) limits should instead be put into units individually. (Closes: #823530, LP: #1578080) * Split out udev rule to name USB network interfaces by MAC address into 73-usb-net-by-mac.rules, so that it's easier to disable. (Closes: #824025) * 73-usb-net-by-mac.rules: Disable when net.ifnames=0 is specified on the kernel command line, to be consistent with disabling the *.link files. * 73-special-net-names.rule: Name the IBM integrated management module virtual USB network card "ibmimm". Thanks Marco d'Itri! -- Martin Pitt Thu, 12 May 2016 10:30:59 +0200 ** Changed in: systemd (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Committed Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Hello Brad, or anyone else affected, Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu6 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: systemd (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Committed Status in systemd source package in Xenial: Fix Committed Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
** Description changed: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: - 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 + 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: - $ sudo systemctl edit mysql + $ sudo systemctl edit mysql and set: - TasksMax=infinity + TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. + + SRU INFORMATION + --- + Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. + Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) + Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. + Test case: + - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. + - Check its TaskMax: systemctl show -p TasksMax cron.service + - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Committed Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. SRU INFORMATION --- Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads. Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it) Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction. Test case: - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit. - Check its TaskMax: systemctl show -p TasksMax cron.service - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Debian unstable/yakkety fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 Xenial backport: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-xenial&id=643089b1d ** Changed in: systemd (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: Fix Committed Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
** Changed in: percona-cluster (Juju Charms Collection) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: In Progress Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: Won't Fix Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
+1 on your proposed approach Martin -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: In Progress Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: New Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
** Changed in: systemd (Debian) Status: Unknown => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: In Progress Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: New Status in systemd package in Debian: Confirmed Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Upstream report: https://github.com/systemd/systemd/issues/3211 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: In Progress Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: New Status in systemd package in Debian: Unknown Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
This was discussed upstream last November: https://lists.freedesktop.org/archives/systemd- devel/2015-November/035006.html And then enabled by default in 228 in https://github.com/systemd/systemd/commit/9ded9cd14. So in retrospect, having a default limit there was not such a good idea after all: 512 is way too much for most "simple" services, and it's way too little for others such as the ones mentioned above. There is also no particular rationale about "512", so even if we'd bump it to 1024 we'd just make the limit even less useful while still breaking software. So I think we should disable the default limit at least for Xenial in an SRU, but probably also in devel. It is both much safer and also much more effective in terms of guarding against berserk programs/bugs/unintended fork bombs etc. to set limits in units individually. Once someone looks at one, this is then a great time to also flip on the other resource and privilege limitations that systemd offers, such as CapabilityBoundingSet=, SecureBits=, PrivateDevices=, PrivateNetwork=, ProtectSystem=, ProtectHome=, etc. ** Changed in: percona-xtradb-cluster-5.6 (Ubuntu) Status: New => Won't Fix ** Changed in: systemd (Ubuntu) Importance: Undecided => High ** Changed in: systemd (Ubuntu) Status: New => In Progress ** Also affects: systemd (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial) Importance: Undecided Status: New ** Bug watch added: Debian Bug tracker #823530 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530 ** Also affects: systemd (Debian) via http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530 Importance: Unknown Status: Unknown ** No longer affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial) ** Changed in: systemd (Ubuntu Xenial) Status: New => In Progress ** Changed in: systemd (Ubuntu Xenial) Importance: Undecided => High ** Changed in: systemd (Ubuntu Xenial) Assignee: (unassigned) => Martin Pitt (pitti) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: Won't Fix Status in systemd package in Ubuntu: In Progress Status in systemd source package in Xenial: In Progress Status in percona-cluster package in Juju Charms Collection: New Status in systemd package in Debian: Unknown Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Raising systemd task and assigning to Martin... ** Also affects: systemd (Ubuntu) Importance: Undecided Status: New ** Changed in: systemd (Ubuntu) Assignee: (unassigned) => Martin Pitt (pitti) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1578080 Title: percona cluster hits resource limits in HA Openstack cloud with xenial Status in percona-xtradb-cluster-5.6 package in Ubuntu: New Status in systemd package in Ubuntu: New Status in percona-cluster package in Juju Charms Collection: New Bug description: I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible. When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring: 2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed When I look at systemd-cgtop, I can see it maxing out at 512 connections. To get it going again I do a: $ sudo systemctl edit mysql and set: TasksMax=infinity Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running. I've noticed that dropping worker-multiplier setting on nova-cloud- controller, neutron-api etc all help to reduce the load, but I still need to bump it up. Please let me know if you need any more information. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp