Were you running the "foreman-trace rails calls" utility when you searched the problematic "puppet" query? Because the output looks like sane output, I can see one hundred of NIC models allocated, but this should not take 6 Gbs at all.
I've slightly enhanced the foreman-tracer, can you also try to run this while doing the problematic query: foreman-tracer rails objects-total all foreman-tracer rails objects all foreman-tracer rails calls all We must be allocating some number of objects somewhere, this should reveal them. LZ On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]> wrote: > This is the problem, it's seems like running a query without any field is > causing this problem, it became more aggressive now probably because i am > running it under the "Classes" dashboard which now have much much more > objects than before. > > Running a query without a specific field: "puppet", memory usage from 1G to > 5G, a ruby instance takes 100% CPU for about a minute. > Running a query with a specific field: "name = puppet", memory usage remains > quite the same, small spike in CPU, returns immediately (less than a > second). > > > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote: >> >> I have the "theforeman/puppet" module, when i search under the "Classes" >> dashboard the word "puppet" it's seems to cause a very high CPU and memory >> starts to increase up to 6GB from almost 1GB. >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote: >>> >>> rails objects-total >>> OBJECT COUNT >>> ERB 826 >>> Nic::Managed 104 >>> Foreman::Parameters::Caster 38 >>> IPAddr 24 >>> ConfigReportStatusCalculator 24 >>> ActiveSupport::HashWithIndifferentAccess 18 >>> SafeRender 10 >>> Range 9 >>> ActionView::OutputBuffer 8 >>> Classification::GlobalParam 6 >>> ActionDispatch::Request 6 >>> Foreman::Parameters::Validator 6 >>> OpenSSL::ASN1::ASN1Data 6 >>> OpenSSL::ASN1::Sequence 6 >>> Authorizer 5 >>> CertificateExtract 3 >>> OpenSSL::ASN1::ObjectId 3 >>> OpenSSL::ASN1::OctetString 3 >>> OpenSSL::X509::Certificate 3 >>> Classification::ClassParam 2 >>> ActiveModel::Name 2 >>> PuppetFactParser 1 >>> HostStatus::Global 1 >>> PuppetFactImporter 1 >>> >>> rails objects >>> FILENAME >>> LINE OBJECT COUNT >>> /usr/share/foreman/app/services/safe_render.rb >>> 39 ERB 826 >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb >>> 10 Nic::Managed 96 >>> /usr/share/foreman/app/services/classification/base.rb >>> 151 Foreman::Parameters::Caster 38 >>> /usr/share/foreman/app/models/config_report.rb >>> 88 ConfigReportStatusCalculator 18 >>> /usr/share/foreman/app/models/host/base.rb >>> 390 ActiveSupport::HashWithIndifferentAccess 11 >>> /usr/share/foreman/app/models/concerns/encryptable.rb >>> 50 Range 8 >>> /usr/share/foreman/app/models/host/base.rb >>> 78 Nic::Managed 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 31 IPAddr 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 18 IPAddr 8 >>> /usr/share/foreman/app/services/classification/base.rb >>> 8 SafeRender 8 >>> /usr/share/foreman/lib/net/validations.rb >>> 102 IPAddr 8 >>> /usr/share/foreman/app/models/lookup_value.rb >>> 51 Foreman::Parameters::Validator 6 >>> /usr/share/foreman/lib/middleware/tagged_logging.rb >>> 10 ActionDispatch::Request 6 >>> /usr/share/foreman/app/models/host/managed.rb >>> 934 Classification::GlobalParam 6 >>> /usr/share/foreman/app/models/host_status/configuration_status.rb >>> 112 ConfigReportStatusCalculator 6 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 17 OpenSSL::ASN1::ASN1Data 6 >>> /usr/share/foreman/app/helpers/application_helper.rb >>> 104 Authorizer 5 >>> /usr/share/foreman/app/services/authorizer_cache.rb >>> 3 ActiveSupport::HashWithIndifferentAccess 5 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 5 OpenSSL::X509::Certificate 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::OctetString 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::Sequence 3 >>> >>> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb >>> 59 CertificateExtract 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 16 OpenSSL::ASN1::ObjectId 3 >>> /usr/share/foreman/app/services/certificate_extract.rb >>> 17 OpenSSL::ASN1::Sequence 3 >>> /usr/share/foreman/app/models/host/managed.rb >>> 420 SafeRender 2 >>> >>> rails calls >>> FILENAME LINE >>> METHOD CALLS >>> /usr/share/foreman/app/models/setting.rb 218 >>> cache 2120 >>> /usr/share/foreman/app/models/setting.rb 72 >>> [] 2086 >>> /usr/share/foreman/app/models/concerns/exportable.rb 55 >>> exportable_attributes 2054 >>> /usr/share/foreman/app/services/safe_render.rb 12 >>> parse 1138 >>> /usr/share/foreman/app/services/safe_render.rb 31 >>> parse_string 826 >>> /usr/share/foreman/app/models/concerns/exportable.rb 33 >>> export_attr 810 >>> /usr/share/foreman/app/models/fact_value.rb 7 >>> name 509 >>> /usr/share/foreman/app/models/user.rb 137 >>> admin? 180 >>> /usr/share/foreman/app/models/taxonomy.rb 100 >>> expand 180 >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92 >>> to_s 112 >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9 >>> setup_object_clone 104 >>> /usr/share/foreman/app/models/nic/managed.rb 45 >>> hostname 96 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34 >>> with_taxonomy_scope 84 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115 >>> scope_by_taxable_ids 84 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75 >>> taxable_ids 84 >>> /usr/share/foreman/app/models/host/base.rb 441 >>> get_interface_by_flag 74 >>> /usr/share/foreman/app/models/concerns/exportable.rb 35 >>> call 66 >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12 >>> dns_ready? 64 >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131 >>> set_current_taxonomy 62 >>> /usr/share/foreman/app/models/concerns/exportable.rb 14 >>> to_export 62 >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9 >>> normalize_recurse 60 >>> /usr/share/foreman/app/models/host/base.rb 237 >>> primary_interface 54 >>> /usr/share/foreman/app/services/classification/base.rb 167 >>> update_generic_matcher 54 >>> /usr/share/foreman/app/services/classification/base.rb 72 >>> value_of_key 54 >>> /usr/share/foreman/app/services/classification/base.rb 73 >>> yield 44 >>> >>> >>> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal wrote: >>>> >>>> Two or three minutes is just fine, counters do reset after 5 minutes >>>> so not longer. >>>> >>>> You can also add "rails calls" output as well, that might be interesting >>>> too. >>>> >>>> LZ >>>> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]> wrote: >>>> > No problem, for how long you think i should have it running to give >>>> > valuable >>>> > data to work with? >>>> > I am running only on objects-total and objects (as you mentioned) >>>> > >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal wrote: >>>> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will only >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your production >>>> >> instance? No changes required in Foreman, setup is quite easy: >>>> >> >>>> >> https://github.com/lzap/foreman-tracer >>>> >> >>>> >> Interesting statistics would be >>>> >> >>>> >> foreman-tracer rails objects-total >>>> >> >>>> >> and >>>> >> >>>> >> foreman-tracer rails objects >>>> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks please. >>>> >> >>>> >> Thanks >>>> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]> >>>> >> wrote: >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1 and >>>> >> > experience the same performance degradation on the foreman web app, >>>> >> > ruby >>>> >> > processes taking a lot of memory, no new plugins were introduced. >>>> >> > Same as what was written above, I suddenly see ruby instances using >>>> >> > of >>>> >> > up to >>>> >> > 4GB RAM/instance as of before i never saw this happens. >>>> >> > >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote: >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <[email protected]> >>>> >> >> wrote: >>>> >> >>> >>>> >> >>> Hi all, >>>> >> >>> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are >>>> >> >>> finding >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a lot >>>> >> >>> more >>>> >> >>> resources than the old version. >>>> >> >>> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them checking >>>> >> >>> in >>>> >> >>> with >>>> >> >>> external puppet masters every 30 minutes. Foreman runs under >>>> >> >>> passenger >>>> >> >>> and >>>> >> >>> has a mysql database. >>>> >> >>> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling the >>>> >> >>> web >>>> >> >>> UI, >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded to 4 >>>> >> >>> x 8 >>>> >> >>> vCPU, >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8 >>>> >> >>> vCPU, >>>> >> >>> 32GB >>>> >> >>> RAM Foreman server handling the web UI and database. We are >>>> >> >>> currently >>>> >> >>> restarting apache on the ENC/fact/report servers every 20 minutes >>>> >> >>> to >>>> >> >>> stop >>>> >> >>> them running out of memory (if we tune passenger more >>>> >> >>> aggressively, >>>> >> >>> the >>>> >> >>> request queue fills up). >>>> >> >>> >>>> >> >>> I noticed a couple of other posts in the last few days regarding >>>> >> >>> performance. Is anyone running a similarly-sized installation on >>>> >> >>> version >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips for >>>> >> >>> tuning >>>> >> >>> passenger or other components? >>>> >> >> >>>> >> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any >>>> >> >> information you >>>> >> >> could share that would provide some light on where things are >>>> >> >> slow? >>>> >> >> (e.g. >>>> >> >> slow operations, etc). >>>> >> >> >>>> >> >> thanks, >>>> >> >> Ohad >>>> >> >>> >>>> >> >>> >>>> >> >>> Regards >>>> >> >>> Guy. >>>> >> >>> >>>> >> >>> -- >>>> >> >>> You received this message because you are subscribed to the >>>> >> >>> Google >>>> >> >>> Groups >>>> >> >>> "Foreman users" group. >>>> >> >>> To unsubscribe from this group and stop receiving emails from it, >>>> >> >>> send >>>> >> >>> an >>>> >> >>> email to [email protected]. >>>> >> >>> To post to this group, send email to [email protected]. >>>> >> >>> Visit this group at >>>> >> >>> https://groups.google.com/group/foreman-users. >>>> >> >>> For more options, visit https://groups.google.com/d/optout. >>>> >> >> >>>> >> >> >>>> >> > -- >>>> >> > You received this message because you are subscribed to the Google >>>> >> > Groups >>>> >> > "Foreman users" group. >>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>> >> > send >>>> >> > an >>>> >> > email to [email protected]. >>>> >> > To post to this group, send email to [email protected]. >>>> >> > Visit this group at https://groups.google.com/group/foreman-users. >>>> >> > For more options, visit https://groups.google.com/d/optout. >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Later, >>>> >> Lukas @lzap Zapletal >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups >>>> > "Foreman users" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> > an >>>> > email to [email protected]. >>>> > To post to this group, send email to [email protected]. >>>> > Visit this group at https://groups.google.com/group/foreman-users. >>>> > For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> -- >>>> Later, >>>> Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
