Public bug reported: [Impact]
systemd-journald allows you to configure a per-service journal rate limit in /etc/systemd/journald.conf via the RateLimitBurst parameter. systemd-journald has code that effectively increases the rate limit when there is a lot of disk space available. However, all versions of systemd before v232 had a bug which would shrink the rate limit when there is between 1 and 16 MB available on disk. If you designed a service to log at a rate R and configured RateLimitBurst to a little above R, this can lead to loss of logs when free disk is between 1 and 16 MB, as your service will be surprisingly rate limited at lower than your configured rate. This bug was fixed upstream in https://github.com/systemd/systemd/pull/4218 It is a straightforward one-line change that makes the code match the comments under it. [Test Case] Run a systemd service that prints lots of logs (eg `yes`). Fill your disk to have only 1MB full. Use journalctl to see how many log lines are between "Suppressed" lines. Note that it is 1/4 of what you'd expect. (Admittedly this test case is a little hard to achieve since journald itself is writing to disk. I did run into this in production.) [Regression Potential] This does mean that journald can write slightly more to disk than it did before when free disk is between 1 and 16MB, but given that the full burst rate is available below 1MB it seems unlikely that any systems are depending on this change in order to not break. The fix has been in systemd since v232 (shipped in Zesty). I would like to see it in Xenial. [Other Info] I am seeing this on: ubuntu@ip-10-0-2-135[i-0b196ce4b8dc3fc55] 1 ~/systemd-229$ lsb_release -rd Description: Ubuntu 16.04.3 LTS Release: 16.04 ubuntu@ip-10-0-2-135[i-0b196ce4b8dc3fc55] 0 ~/systemd-229$ apt-cache policy systemd systemd: Installed: 229-4ubuntu19 Candidate: 229-4ubuntu21 Version table: 229-4ubuntu21 500 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages *** 229-4ubuntu19 100 100 /var/lib/dpkg/status 229-4ubuntu10 500 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 229-4ubuntu4 500 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages This bug seems to date back to the original implementation of rate limiting (https://github.com/systemd/systemd/commit/6e409ce10d). ** Affects: systemd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1732803 Title: systemd-journald RateLimitBurst is sometimes divided by 4 Status in systemd package in Ubuntu: New Bug description: [Impact] systemd-journald allows you to configure a per-service journal rate limit in /etc/systemd/journald.conf via the RateLimitBurst parameter. systemd-journald has code that effectively increases the rate limit when there is a lot of disk space available. However, all versions of systemd before v232 had a bug which would shrink the rate limit when there is between 1 and 16 MB available on disk. If you designed a service to log at a rate R and configured RateLimitBurst to a little above R, this can lead to loss of logs when free disk is between 1 and 16 MB, as your service will be surprisingly rate limited at lower than your configured rate. This bug was fixed upstream in https://github.com/systemd/systemd/pull/4218 It is a straightforward one-line change that makes the code match the comments under it. [Test Case] Run a systemd service that prints lots of logs (eg `yes`). Fill your disk to have only 1MB full. Use journalctl to see how many log lines are between "Suppressed" lines. Note that it is 1/4 of what you'd expect. (Admittedly this test case is a little hard to achieve since journald itself is writing to disk. I did run into this in production.) [Regression Potential] This does mean that journald can write slightly more to disk than it did before when free disk is between 1 and 16MB, but given that the full burst rate is available below 1MB it seems unlikely that any systems are depending on this change in order to not break. The fix has been in systemd since v232 (shipped in Zesty). I would like to see it in Xenial. [Other Info] I am seeing this on: ubuntu@ip-10-0-2-135[i-0b196ce4b8dc3fc55] 1 ~/systemd-229$ lsb_release -rd Description: Ubuntu 16.04.3 LTS Release: 16.04 ubuntu@ip-10-0-2-135[i-0b196ce4b8dc3fc55] 0 ~/systemd-229$ apt-cache policy systemd systemd: Installed: 229-4ubuntu19 Candidate: 229-4ubuntu21 Version table: 229-4ubuntu21 500 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages *** 229-4ubuntu19 100 100 /var/lib/dpkg/status 229-4ubuntu10 500 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 229-4ubuntu4 500 500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages This bug seems to date back to the original implementation of rate limiting (https://github.com/systemd/systemd/commit/6e409ce10d). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1732803/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
