Can you guys try foreman-tracer utility (SystemTap based, will only work on CentOS 7 or higher or ST-enabled kernels) on your production instance? No changes required in Foreman, setup is quite easy:
https://github.com/lzap/foreman-tracer Interesting statistics would be foreman-tracer rails objects-total and foreman-tracer rails objects It's like "top" utility experience, pastebin the bottlenecks please. Thanks On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]> wrote: > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1 and > experience the same performance degradation on the foreman web app, ruby > processes taking a lot of memory, no new plugins were introduced. > Same as what was written above, I suddenly see ruby instances using of up to > 4GB RAM/instance as of before i never saw this happens. > > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote: >> >> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <[email protected]> wrote: >>> >>> Hi all, >>> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are finding >>> that Foreman's ENC/facts/reports components are requiring a lot more >>> resources than the old version. >>> >>> We have around 1,700 hosts in Foreman, with 600 of them checking in with >>> external puppet masters every 30 minutes. Foreman runs under passenger and >>> has a mysql database. >>> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling the web UI, >>> database and ENC/facts/reports. On 1.12.3, we have expanded to 4 x 8 vCPU, >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8 vCPU, 32GB >>> RAM Foreman server handling the web UI and database. We are currently >>> restarting apache on the ENC/fact/report servers every 20 minutes to stop >>> them running out of memory (if we tune passenger more aggressively, the >>> request queue fills up). >>> >>> I noticed a couple of other posts in the last few days regarding >>> performance. Is anyone running a similarly-sized installation on version >>> 1.12/1.13? If so, what performance are you getting? Any tips for tuning >>> passenger or other components? >> >> >> Are you using a new plugins? or just plain foreman? any information you >> could share that would provide some light on where things are slow? (e.g. >> slow operations, etc). >> >> thanks, >> Ohad >>> >>> >>> Regards >>> Guy. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
