[Bug 1831021] Re: Extremely high memory consumption under heavy workload
This bug was fixed in the package cups - 2.2.11-0ubuntu1 --- cups (2.2.11-0ubuntu1) eoan; urgency=medium * New upstream release * Removed 10 patches taken from upstream. * Refreshed 13 patches with quilt. * Removed patch to set "LogDebugHistory 9" as it causes high system load when there are very many jobs (LP: #1831021). * Set "DirtyCleanInterval 0" in CUPS configuration to avoid losses of jobs when cupsd gets killed (LP: #1830022). -- Till Kamppeter Mon, 05 Aug 2019 11:40:10 +0200 ** Changed in: cups (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
** Tags removed: rls-ee-incoming ** Tags added: rls-ee-notfixing -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
Thank you for the explanation Till. I believe the lack of previous complaints is because this happens in a pretty unique conditions (very high MaxJobs and large batches of jobs). The test case however was based on a real-life scenario reported to me. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
I have raised the default LogDebugHistory some years ago as most printing problems happen during the execution of a job and this way one gets a logging with debug verbosity only for jobs and standard verbosity for all the rest. This way one often did not need to ask the bug- reporting users to reproduce their problems after having set logging to debug mode but instead, one could simply start to work with the error_log auto-captured by Apport. Up to now no one complained that it somehow overloaded system resources. Please add a line "LogDebugHistory 200" to your /etc/cups/cupsd.conf and restart CUPS. I will remove this change from the next release of the CUPS package. OdyX, could you remove the LogDebugHistory=9 default change from the CUPS package for Debian (remove log-debug-history-nearly- unlimited.patch)? Thanks. ** Changed in: cups (Ubuntu) Status: Confirmed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
In a recent test with MaxJobs of 4 I was easily able to reach over 65 GB of memory consumption within 60 mins, while with LogDebugHistory set to 0 the consumption was below 400 MB. My recommendation is to lower the default LogDebugHistory value back to what is upstream: 200. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
** Tags added: rls-ee-incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: cups (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
** Description changed: Under heavy workload conditions cups can reach irrationally high memory consumption very quickly (tens of GBs). Test case: 1. Set MaxJobs to 4 in cupsd.conf. 2. sudo apt install cups-pdf 3. Fill the queue with jobs: while [ 1 ]; lp -d PDF /usr/share/cups/data/default.pdf; done 4. Cancel all jobs cancel -a PDF 5. Restart cups. 6. Start filling the queue again (as in step 3). Expected result: Jobs are processed and memory consumption is proportional to the number of jobs. Actual result: - After step 5 or at latest step 6 memory consumption starts to increase exponentially - from ~150-200 MB to 8+GB. Without foreseeing this it's very easy to get cups killed by OOM killer. + After step 5 or at latest step 6 memory consumption starts to increase exponentially - from ~150-200 MB to 8+GB. Without foreseeing this it's very easy to get cupsd killed by OOM killer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1831021] Re: Extremely high memory consumption under heavy workload
After some analysis I believe the very high default value of LogDebugHistory (9) is to blame for this (attaching massif report from the test case). With a high number of jobs it's very easy to get hunderds-thousands of messages in LogDebugHistory which in turn will lead to enormous memory consumption, e.g. 4 jobs * 2000 lines * (16 bytes for cupsd_joblog_t structure + 50-60 chars per message on average) - 5.66 GB. After setting LogDebugHistory to 0 the memory consumption never exceeded 300 MB in my tests under the same heavy load. That's a very significant difference. The possible workarounds: 1. Decrease LogDebugHistory to a low value (even to 0, upstream it's set to 200 by default). 2. Change LogLevel to debug (it will slow down cupsd operation significantly, but at the price of lower memory consumption as the LogDebugHistory is unused). 3. Make sure jobs are cleaned periodically by setting e.g. PreserveJobHistory 300. I am wondering what was the rationale behind increasing LogDebugHistory to such a high value. ** Attachment added: "massif.out.xz" https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+attachment/5267574/+files/massif.out.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1831021 Title: Extremely high memory consumption under heavy workload To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1831021/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
