This is the problem, it's seems like running a query without any field is causing this problem, it became more aggressive now probably because i am running it under the "Classes" dashboard which now have much much more objects than before.
Running a query without a specific field: "puppet", memory usage from 1G to 5G, a ruby instance takes 100% CPU for about a minute. Running a query with a specific field: "name = puppet", memory usage remains quite the same, small spike in CPU, returns immediately (less than a second). On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote: > > I have the "theforeman/puppet" module, when i search under the "Classes" > dashboard the word "puppet" it's seems to cause a very high CPU and memory > starts to increase up to 6GB from almost 1GB. > > On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote: >> >> *rails objects-total* >> >> *OBJECT COUNTERB 826Nic::Managed >> 104Foreman::Parameters::Caster 38IPAddr >> 24ConfigReportStatusCalculator >> 24ActiveSupport::HashWithIndifferentAccess 18SafeRender >> 10Range 9ActionView::OutputBuffer >> 8Classification::GlobalParam 6ActionDispatch::Request >> 6Foreman::Parameters::Validator 6OpenSSL::ASN1::ASN1Data >> 6OpenSSL::ASN1::Sequence 6Authorizer >> 5CertificateExtract 3OpenSSL::ASN1::ObjectId >> 3OpenSSL::ASN1::OctetString 3OpenSSL::X509::Certificate >> 3Classification::ClassParam 2ActiveModel::Name >> 2PuppetFactParser 1HostStatus::Global 1PuppetFactImporter >> 1* >> *rails objects* >> FILENAME >> LINE OBJECT COUNT >> /usr/share/foreman/app/services/safe_render.rb >> 39 ERB 826 >> /usr/share/foreman/app/models/concerns/interface_cloning.rb >> 10 Nic::Managed 96 >> /usr/share/foreman/app/services/classification/base.rb >> 151 Foreman::Parameters::Caster 38 >> /usr/share/foreman/app/models/config_report.rb >> 88 ConfigReportStatusCalculator 18 >> /usr/share/foreman/app/models/host/base.rb >> 390 ActiveSupport::HashWithIndifferentAccess 11 >> /usr/share/foreman/app/models/concerns/encryptable.rb >> 50 Range 8 >> /usr/share/foreman/app/models/host/base.rb >> 78 Nic::Managed 8 >> /usr/share/foreman/lib/net/validations.rb >> 31 IPAddr 8 >> /usr/share/foreman/lib/net/validations.rb >> 18 IPAddr 8 >> /usr/share/foreman/app/services/classification/base.rb >> 8 SafeRender 8 >> /usr/share/foreman/lib/net/validations.rb >> 102 IPAddr 8 >> /usr/share/foreman/app/models/lookup_value.rb >> 51 Foreman::Parameters::Validator 6 >> /usr/share/foreman/lib/middleware/tagged_logging.rb >> 10 ActionDispatch::Request 6 >> /usr/share/foreman/app/models/host/managed.rb >> 934 Classification::GlobalParam 6 >> /usr/share/foreman/app/models/host_status/configuration_status.rb >> 112 ConfigReportStatusCalculator 6 >> /usr/share/foreman/app/services/certificate_extract.rb >> 17 OpenSSL::ASN1::ASN1Data 6 >> /usr/share/foreman/app/helpers/application_helper.rb >> 104 Authorizer 5 >> /usr/share/foreman/app/services/authorizer_cache.rb >> 3 ActiveSupport::HashWithIndifferentAccess 5 >> /usr/share/foreman/app/services/certificate_extract.rb >> 5 OpenSSL::X509::Certificate 3 >> /usr/share/foreman/app/services/certificate_extract.rb >> 16 OpenSSL::ASN1::OctetString 3 >> /usr/share/foreman/app/services/certificate_extract.rb >> 16 OpenSSL::ASN1::Sequence 3 >> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb >> >> 59 CertificateExtract 3 >> /usr/share/foreman/app/services/certificate_extract.rb >> 16 OpenSSL::ASN1::ObjectId 3 >> /usr/share/foreman/app/services/certificate_extract.rb >> 17 OpenSSL::ASN1::Sequence 3 >> /usr/share/foreman/app/models/host/managed.rb >> 420 SafeRender 2 >> >> *rails calls* >> FILENAME LINE >> METHOD CALLS >> /usr/share/foreman/app/models/setting.rb 218 >> cache 2120 >> /usr/share/foreman/app/models/setting.rb 72 >> [] 2086 >> /usr/share/foreman/app/models/concerns/exportable.rb 55 >> exportable_attributes 2054 >> /usr/share/foreman/app/services/safe_render.rb 12 >> parse 1138 >> /usr/share/foreman/app/services/safe_render.rb 31 >> parse_string 826 >> /usr/share/foreman/app/models/concerns/exportable.rb 33 >> export_attr 810 >> /usr/share/foreman/app/models/fact_value.rb 7 >> name 509 >> /usr/share/foreman/app/models/user.rb 137 >> admin? 180 >> /usr/share/foreman/app/models/taxonomy.rb 100 >> expand 180 >> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92 >> to_s 112 >> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9 >> setup_object_clone 104 >> /usr/share/foreman/app/models/nic/managed.rb 45 >> hostname 96 >> /usr/share/foreman/app/models/concerns/taxonomix.rb 34 >> with_taxonomy_scope 84 >> /usr/share/foreman/app/models/concerns/taxonomix.rb 115 >> scope_by_taxable_ids 84 >> /usr/share/foreman/app/models/concerns/taxonomix.rb 75 >> taxable_ids 84 >> /usr/share/foreman/app/models/host/base.rb 441 >> get_interface_by_flag 74 >> /usr/share/foreman/app/models/concerns/exportable.rb 35 >> call 66 >> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12 >> dns_ready? 64 >> /usr/share/foreman/app/models/concerns/taxonomix.rb 131 >> set_current_taxonomy 62 >> /usr/share/foreman/app/models/concerns/exportable.rb 14 >> to_export 62 >> /usr/share/foreman/app/services/structured_fact_importer.rb 9 >> normalize_recurse 60 >> /usr/share/foreman/app/models/host/base.rb 237 >> primary_interface 54 >> /usr/share/foreman/app/services/classification/base.rb 167 >> update_generic_matcher 54 >> /usr/share/foreman/app/services/classification/base.rb 72 >> value_of_key 54 >> /usr/share/foreman/app/services/classification/base.rb 73 >> yield 44 >> >> >> >> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal wrote: >>> >>> Two or three minutes is just fine, counters do reset after 5 minutes >>> so not longer. >>> >>> You can also add "rails calls" output as well, that might be interesting >>> too. >>> >>> LZ >>> >>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]> wrote: >>> > No problem, for how long you think i should have it running to give >>> valuable >>> > data to work with? >>> > I am running only on objects-total and objects (as you mentioned) >>> > >>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal wrote: >>> >> >>> >> Can you guys try foreman-tracer utility (SystemTap based, will only >>> >> work on CentOS 7 or higher or ST-enabled kernels) on your production >>> >> instance? No changes required in Foreman, setup is quite easy: >>> >> >>> >> https://github.com/lzap/foreman-tracer >>> >> >>> >> Interesting statistics would be >>> >> >>> >> foreman-tracer rails objects-total >>> >> >>> >> and >>> >> >>> >> foreman-tracer rails objects >>> >> >>> >> It's like "top" utility experience, pastebin the bottlenecks please. >>> >> >>> >> Thanks >>> >> >>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]> >>> wrote: >>> >> > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1 and >>> >> > experience the same performance degradation on the foreman web app, >>> ruby >>> >> > processes taking a lot of memory, no new plugins were introduced. >>> >> > Same as what was written above, I suddenly see ruby instances using >>> of >>> >> > up to >>> >> > 4GB RAM/instance as of before i never saw this happens. >>> >> > >>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote: >>> >> >> >>> >> >> >>> >> >> >>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <[email protected]> >>> wrote: >>> >> >>> >>> >> >>> Hi all, >>> >> >>> >>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are >>> finding >>> >> >>> that Foreman's ENC/facts/reports components are requiring a lot >>> more >>> >> >>> resources than the old version. >>> >> >>> >>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them checking >>> in >>> >> >>> with >>> >> >>> external puppet masters every 30 minutes. Foreman runs under >>> passenger >>> >> >>> and >>> >> >>> has a mysql database. >>> >> >>> >>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling the >>> web >>> >> >>> UI, >>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded to 4 >>> x 8 >>> >> >>> vCPU, >>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8 >>> vCPU, >>> >> >>> 32GB >>> >> >>> RAM Foreman server handling the web UI and database. We are >>> currently >>> >> >>> restarting apache on the ENC/fact/report servers every 20 minutes >>> to >>> >> >>> stop >>> >> >>> them running out of memory (if we tune passenger more >>> aggressively, >>> >> >>> the >>> >> >>> request queue fills up). >>> >> >>> >>> >> >>> I noticed a couple of other posts in the last few days regarding >>> >> >>> performance. Is anyone running a similarly-sized installation on >>> >> >>> version >>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips for >>> >> >>> tuning >>> >> >>> passenger or other components? >>> >> >> >>> >> >> >>> >> >> Are you using a new plugins? or just plain foreman? any >>> information you >>> >> >> could share that would provide some light on where things are >>> slow? >>> >> >> (e.g. >>> >> >> slow operations, etc). >>> >> >> >>> >> >> thanks, >>> >> >> Ohad >>> >> >>> >>> >> >>> >>> >> >>> Regards >>> >> >>> Guy. >>> >> >>> >>> >> >>> -- >>> >> >>> You received this message because you are subscribed to the >>> Google >>> >> >>> Groups >>> >> >>> "Foreman users" group. >>> >> >>> To unsubscribe from this group and stop receiving emails from it, >>> send >>> >> >>> an >>> >> >>> email to [email protected]. >>> >> >>> To post to this group, send email to [email protected]. >>> >> >>> Visit this group at https://groups.google.com/group/foreman-users. >>> >>> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >>> >> >> >>> >> > -- >>> >> > You received this message because you are subscribed to the Google >>> >> > Groups >>> >> > "Foreman users" group. >>> >> > To unsubscribe from this group and stop receiving emails from it, >>> send >>> >> > an >>> >> > email to [email protected]. >>> >> > To post to this group, send email to [email protected]. >>> >> > Visit this group at https://groups.google.com/group/foreman-users. >>> >> > For more options, visit https://groups.google.com/d/optout. >>> >> >>> >> >>> >> >>> >> -- >>> >> Later, >>> >> Lukas @lzap Zapletal >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "Foreman users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an >>> > email to [email protected]. >>> > To post to this group, send email to [email protected]. >>> > Visit this group at https://groups.google.com/group/foreman-users. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> Later, >>> Lukas @lzap Zapletal >>> >> -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
