[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Waldemar Hamm]

Great to hear. This bug will be fixed then I guess:
https://youtrack.jetbrains.com/issue/UP-6943

Thanks a lot!

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

> Will my systemd scripts with TasksMax setting keep working?

Yes, this only changes the builtin defaults if there is no explicit
TaskMax= setting.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Waldemar Hamm]

Will my systemd scripts with TasksMax setting keep working?

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Launchpad Bug Tracker]

This bug was fixed in the package systemd - 229-4ubuntu6

---
systemd (229-4ubuntu6) xenial-proposed; urgency=medium

  * Add a DHCP exit hook for pushing received NTP servers into timesyncd.
(LP: #1578663)
  * Revert "enable TasksMax= for all services by default, and set it to 512".
Introducing a default limit on number of threads broke a lot of software
which regularly needs more, such as MySQL and RabbitMQ, or services that
spawn off an indefinite number of subtasks that are not in a scope, like
LXC or cron. 512 is way too much for most "simple" services, and it's way
too little for the ones mentioned above. Effective (and much stricter)
limits should instead be put into units individually.
(Closes: #823530, LP: #1578080)
  * debian/gbp.conf: Switch to ubuntu-xenial branch.

 -- Martin Pitt   Thu, 12 May 2016 10:39:30
+0200

** Changed in: systemd (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Bug Watch Updater]

** Changed in: systemd (Debian)
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Brad Marshall]

I've upgraded systemd across my cluster to 229-4ubuntu6, removed all my
custom tweaks to systemd settings, reloaded the daemon and both rabbitmq
and mysql appear to be working fine on my openstack cluster.  I'll be
throwing a bit more load at it, but usually by this point mysql has
fallen over, so I'd say this is a success.


** Tags removed: verification-needed
** Tags added: verification-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Launchpad Bug Tracker]

This bug was fixed in the package systemd - 229-6ubuntu1

---
systemd (229-6ubuntu1) yakkety; urgency=medium

  * Merge with Debian unstable. Remaining Ubuntu changes:
- Hack to support system-image read-only /etc, and modify files in
  /etc/writable/ instead.

systemd (229-6) unstable; urgency=medium

  * systemd-container: Prefer renamed "btrfs-progs" package name over
"btrfs-tools". (Closes: #822629)
  * systemd-container: Recommend libnss-mymachines. (Closes: #822615)
  * Drop systemd-dbg, in favor of debhelpers' automatic -dbgsym packages.
  * Drop Add-targets-for-compatibility-with-Debian-insserv-sy.patch; we don't
need $x-display-manager any more as most/all DMs ship native services, and
$mail-transport-agent is not widely used (not even by our default MTA
exim4).
  * Unify our two patches for Debian specific configuration files.
  * Drop udev-re-enable-mount-propagation-for-udevd.patch, i. e. run udevd in
its own slave mount name space again. laptop-mode-tools 1.68 fixed the
original bug (#762018), thus add a Breaks: to earlier versions.
  * Ship fbdev-blacklist.conf in /lib/modprobe.d/ instead of /etc/modprobe.d/;
remove the conffile on upgrades.
  * Replace util-Add-hidden-suffixes-for-ucf.patch with patch that got
committed upstream.
  * Replace Stop-syslog.socket-when-entering-emergency-mode.patch with patch
that got committed upstream.
  * debian/udev.README.Debian: Adjust documentation of MAC based naming for
USB network cards to the udev rule, where this was moved to in 229-5.
  * debian/extra/init-functions.d/40-systemd: Invoke status command with
--no-pager, to avoid blocking scripts that call an init.d script with
"status" with an unexpected pager process. (Closes: #765175, LP: #1576409)
  * Add debian/extra/rules/70-debian-uaccess.rules: Make FIDO U2F dongles
accessible to the user session. This avoids having to install libu2f-host0
(which isn't discoverable at all) to make those devices work.
(LP: #1387908)
  * libnss-resolve: Enable systemd-resolved.service on package installation,
as this package makes little sense without resolved.
  * Add a DHCP exit hook for pushing received NTP servers into timesyncd.
(LP: #1578663)
  * debian/udev.postinst: Fix migration check from the old persistent-net
generator to not apply to chroots. (Closes: #813141)
  * Revert "enable TasksMax= for all services by default, and set it to 512".
Introducing a default limit on number of threads broke a lot of software
which regularly needs more, such as MySQL and RabbitMQ, or services that
spawn off an indefinite number of subtasks that are not in a scope, like
LXC or cron. 512 is way too much for most "simple" services, and it's way
too little for the ones mentioned above. Effective (and much stricter)
limits should instead be put into units individually.
(Closes: #823530, LP: #1578080)
  * Split out udev rule to name USB network interfaces by MAC address into
73-usb-net-by-mac.rules, so that it's easier to disable. (Closes: #824025)
  * 73-usb-net-by-mac.rules: Disable when net.ifnames=0 is specified on the
kernel command line, to be consistent with disabling the *.link files.
  * 73-special-net-names.rule: Name the IBM integrated management module
virtual USB network card "ibmimm". Thanks Marco d'Itri!

 -- Martin Pitt   Thu, 12 May 2016 10:30:59
+0200

** Changed in: systemd (Ubuntu)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Adam Conrad]

Hello Brad, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu6
in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: systemd (Ubuntu Xenial)
   Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

** Description changed:

  I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial
  using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack
  being HA as possible.
  
  When deployed, after running for a while mysql (which is a 3 node
  cluster) starts refusing connections, and erroring:
  
-   2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50
+   2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50
  'Can't create thread to handle new connection' error(s) suppressed
  
  When I look at systemd-cgtop, I can see it maxing out at 512
  connections.
  
  To get it going again I do a:
  
-   $ sudo systemctl edit mysql
+   $ sudo systemctl edit mysql
  
  and set:
  
-   TasksMax=infinity
+   TasksMax=infinity
  
  Sometimes I even need to edit /etc/systemd/system.conf and bump
  DefaultTasksMax to 1024 or higher, depending on long its been left
  running.
  
  I've noticed that dropping worker-multiplier setting on nova-cloud-
  controller, neutron-api etc all help to reduce the load, but I still
  need to bump it up.
  
  Please let me know if you need any more information.
+ 
+ SRU INFORMATION
+ ---
+ Impact: Introducing a default #thread limit of 512 broke an unknown set of 
services which regularly run many threads.
+ Fix: 
http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 
(essentially, revert the upstream commit that enabled it)
+ Regression potential: Very low -- this just restores the pre-228 behaviour 
and does not impose any new restriction.
+ Test case:
+  - Pick some unit like cron.service or mysql.service that does not specify an 
explicit TaskMax= limit.
+  - Check its TaskMax: systemctl show -p TasksMax cron.service
+  - In current xenial this is "512", after the update it should be a very big 
number (maxint minus 1, which means "infinity")

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

Debian unstable/yakkety fix: 
http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0
Xenial backport: 
http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-xenial=643089b1d


** Changed in: systemd (Ubuntu)
   Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[James Page]

** Changed in: percona-cluster (Juju Charms Collection)
   Status: New => Won't Fix

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[James Page]

+1 on your proposed approach Martin

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Bug Watch Updater]

** Changed in: systemd (Debian)
   Status: Unknown => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

Upstream report: https://github.com/systemd/systemd/issues/3211

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

This was discussed upstream last November:
https://lists.freedesktop.org/archives/systemd-
devel/2015-November/035006.html

And then enabled by default in 228 in
https://github.com/systemd/systemd/commit/9ded9cd14.

So in retrospect, having a default limit there was not such a good idea
after all: 512 is way too much for most "simple" services, and it's way
too little for others such as the ones mentioned above. There is also no
particular rationale about "512", so even if we'd bump it to 1024 we'd
just make the limit even less useful while still breaking software.

So I think we should disable the default limit at least for Xenial in an
SRU, but probably also in devel. It is both much safer and also much
more effective in terms of guarding against berserk
programs/bugs/unintended fork bombs etc. to set limits in units
individually. Once someone looks at one, this is then a great time to
also flip on the other resource and privilege limitations that systemd
offers, such as CapabilityBoundingSet=, SecureBits=, PrivateDevices=,
PrivateNetwork=, ProtectSystem=, ProtectHome=, etc.


** Changed in: percona-xtradb-cluster-5.6 (Ubuntu)
   Status: New => Won't Fix

** Changed in: systemd (Ubuntu)
   Importance: Undecided => High

** Changed in: systemd (Ubuntu)
   Status: New => In Progress

** Also affects: systemd (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Also affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Bug watch added: Debian Bug tracker #823530
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530

** Also affects: systemd (Debian) via
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530
   Importance: Unknown
   Status: Unknown

** No longer affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial)

** Changed in: systemd (Ubuntu Xenial)
   Status: New => In Progress

** Changed in: systemd (Ubuntu Xenial)
   Importance: Undecided => High

** Changed in: systemd (Ubuntu Xenial)
 Assignee: (unassigned) => Martin Pitt (pitti)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[James Page]

Raising systemd task and assigning to Martin...


** Also affects: systemd (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: systemd (Ubuntu)
 Assignee: (unassigned) => Martin Pitt (pitti)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[James Page]

Re "Failed to allocate directory watch: Too many open files" message - I
think that you might be on the right track, but it might not
specifically be to the tasks limit systemd is imposing.

I have seen this message during testing on an alternative architecture,
but assumed it was not related...

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Brad Marshall]

FWIW I'm also seeing limits hit in other areas, rabbitmq-server seems to
be a common one.  I'm also randomly seeing:

  Failed to allocate directory watch: Too many open files

on the command line.  I suspect its related to the percona and rabbitmq
services hitting limits, but I'm not exactly sure.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[James Page]

Raising a package bug for this one; there is also an ongoing
conversation about whether this limit should be dropped at the systemd
level as an SRU, but that's still not decided AFAIK.

** Also affects: percona-xtradb-cluster-5.6 (Ubuntu)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs