Oh, now I realized you used Guy's thread for a different problem, two issues, one thread issue :-)
This one I think is actively worked on, I believe Ori was debugging this. Anyone knows if we are tracking this scoped_search issue already? On Thu, Nov 3, 2016 at 2:28 PM, Erez Zarum <[email protected]> wrote: > Hey, > It doesn't really let me know, but i'm not sure it's the same issue, this > only happens when searching the dashboard without using any specific fields > ("puppet" vs "name = puppet"), i believe this has nothing to do with any > facts API calls (internally). > 2016-11-03 13:22:38 b2812ac1 [app] [I] Started GET > "/puppetclasses?utf8=%E2%9C%93&search=puppet" for 10.20.30.11 at 2016-11-03 > 13:22:38 +0000 > 2016-11-03 13:22:38 b2812ac1 [app] [I] Processing by > PuppetclassesController#index as HTML > 2016-11-03 13:22:38 b2812ac1 [app] [I] Parameters: {"utf8"=>"✓", > "search"=>"puppet"} > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > puppetclasses/index.html.erb within layouts/application (1176.4ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered common/_searchbar.html.erb > (14.7ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > layouts/_application_content.html.erb (16.0ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > home/_user_dropdown.html.erb (3.9ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Read fragment > views/tabs_and_title_records-4 (0.1ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > home/_organization_dropdown.html.erb (16.1ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > home/_location_dropdown.html.erb (21.1ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered > home/_org_switcher.html.erb (38.6ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb > (4.1ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb > (3.0ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb > (2.2ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb > (2.0ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb > (2.8ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Write fragment > views/tabs_and_title_records-4 (0.5ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_topbar.html.erb > (131.1ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered layouts/base.html.erb > (132.8ms) > 2016-11-03 13:23:40 b2812ac1 [app] [I] Completed 200 OK in 62045ms (Views: > 1218.4ms | ActiveRecord: 4650.5ms) > > Even if this is expected, there should be some kind of protection again > those kind of queries or set a default field when no field is set. > > > > On Thursday, November 3, 2016 at 2:45:43 PM UTC+2, Lukas Zapletal wrote: >> >> Thanks, >> >> all I can say from these outputs is we have some eager loading of data >> into hash most likely. Can you tell from the production.log file which >> HTTP calls are slowest? Is it ENC, report or facts? I bet it's facts, >> but please confirm. >> >> There were some performance issues and improvements in both 1.12 and >> 1.13, particularly: >> >> http://projects.theforeman.org/issues/16749 - I think this might be >> relevant >> http://projects.theforeman.org/issues/9016 >> http://projects.theforeman.org/issues/16576 >> >> Do you have some hosts with high number of NICs (ipaddress facts) >> reported? >> >> LZ >> >> On Thu, Nov 3, 2016 at 11:26 AM, Erez Zarum <[email protected]> wrote: >> > It's not the terminal size issue, the format is set by the script, for >> > this >> > tasks i increased it (Line 92,94,95,190,192) >> > >> > rails calls all >> > >> > FILENAME LINE METHOD CALLS >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> > 260 hash 51976 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> > 160 nil? 48142 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> > 171 association_instance_get 48142 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> > 157 association 48142 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb >> > 63 loaded! 47962 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 116 to_s 37385 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 115 _reflect_on_association 37385 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 889 name 37385 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb >> > 86 _reflections 37385 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 889 options 37314 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 889 active_record 37292 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 686 through_reflection 37292 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 442 belongs_to? 37207 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb >> > 67 stale_state 37204 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb >> > 6 through_reflection 37204 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> > 592 collection? 37157 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb >> > 72 primary_key 37059 >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb >> > 18 primary_key 36975 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> > 31 column_alias 36975 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> > 32 hash 36975 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> > 254 nil? 36974 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb >> > 39 [] 35363 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb >> > 9 [] 35363 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb >> > 35 value 35340 >> > >> > >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb >> > 30 fetch_value 35317 >> > >> > rails objects-total all >> > OBJECT COUNT >> > Hash 8495 >> > Arel::Nodes::SqlLiteral 7548 >> > Arel::Attributes::Attribute 7401 >> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 6981 >> > Array 6478 >> > Taxonomy::ActiveRecord_Relation 4720 >> > String 4409 >> > Arel::Nodes::As 3091 >> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091 >> > ActiveRecord::Relation::Merger 2563 >> > Arel::Table 1669 >> > Time 1508 >> > Arel::Nodes::Equality 1428 >> > Hostgroup::ActiveRecord_Relation 1283 >> > ActiveRecord::Attribute::FromDatabase 1146 >> > Arel::Nodes::Casted 1136 >> > Arel::Nodes::SelectStatement 1068 >> > Arel::Nodes::JoinSource 1068 >> > Arel::SelectManager 1068 >> > Arel::Nodes::SelectCore 1068 >> > TaxableTaxonomy::ActiveRecord_Relation 1065 >> > Arel::Nodes::And 1032 >> > VariableLookupKey::ActiveRecord_Relation 984 >> > Set 960 >> > ActiveRecord::Result 916 >> > >> > I couldn't understand what you mean about the interfaces, though i have >> > only >> > one host with about 7 interfaces, i saw that post but it's seems not >> > relevant to me. >> > >> > On Wednesday, November 2, 2016 at 5:40:54 PM UTC+2, Lukas Zapletal >> > wrote: >> >> >> >> That's expected, the tracer is topped to prevent eating lot of memory, >> >> so it rather crash than consume more. I have slightly increased the >> >> limit, can you update the script and run once again, now for the last >> >> time: >> >> >> >> rails objects-total app 100 >> >> >> >> and >> >> >> >> rails calls all 100 >> >> >> >> Please set minimum font size for your terminal to fit 100 rows. I am >> >> just curious if we can spot the Model that is causing this. >> >> >> >> But I suspect this is the similar issue that someone else just had >> >> (see replies from today in the list). Hosts were created with too many >> >> NICs. The suggestion was: "See the 'ignore interfaces with matching >> >> identifier' option under Settings -> Provisioning." Then you need to >> >> delete all the NICs from the database to get things rolling again. >> >> >> >> LZ >> >> >> >> On Wed, Nov 2, 2016 at 2:50 PM, Erez Zarum <[email protected]> wrote: >> >> > Hey, >> >> > I think you nailed it :) >> >> > >> >> > rails calls all probe failed at some point though. >> >> > >> >> > rails calls all >> >> > >> >> > FILENAME LINE METHOD CALLS >> >> > >> >> > >> >> > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> >> > 260 hash 86006 >> >> > >> >> > >> >> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> >> > 171 association_instance_get 76602 >> >> > >> >> > >> >> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> >> > 157 association 76602 >> >> > >> >> > >> >> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb >> >> > 160 nil? 76602 >> >> > >> >> > >> >> > are/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb >> >> > 63 loaded! 76422 >> >> > >> >> > >> >> > e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb >> >> > 86 _reflections 60166 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 115 _reflect_on_association 60165 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 116 to_s 60164 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 889 name 60162 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 889 options 60086 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 686 through_reflection 60065 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 889 active_record 60065 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 442 belongs_to? 59980 >> >> > >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb >> >> > 67 stale_state 59979 >> >> > >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb >> >> > 6 through_reflection 59979 >> >> > >> >> > >> >> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb >> >> > 592 collection? 59894 >> >> > >> >> > >> >> > ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb >> >> > 72 primary_key 59802 >> >> > >> >> > >> >> > activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb >> >> > 18 primary_key 59712 >> >> > >> >> > >> >> > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> >> > 31 column_alias 59711 >> >> > >> >> > >> >> > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> >> > 254 nil? 59710 >> >> > >> >> > >> >> > gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> >> > 32 hash 59710 >> >> > >> >> > >> >> > /share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb >> >> > 39 [] 58136 >> >> > >> >> > >> >> > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb >> >> > 9 [] 58136 >> >> > >> >> > >> >> > r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb >> >> > 35 value 58078 >> >> > >> >> > >> >> > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb >> >> > 30 fetch_value 58042 >> >> > WARNING: Number of errors: 0, skipped probes: 101 >> >> > WARNING: /usr/bin/staprun exited with status: 1 >> >> > Pass 5: run failed. [man error::pass5] >> >> > >> >> > rails objects all >> >> > >> >> > FILENAME >> >> > LINE OBJECT COUNT >> >> > >> >> > cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb >> >> > 72 rd::ConnectionAdapters::PostgreSQL::Name 7037 >> >> > >> >> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb >> >> > 100 Arel::Attributes::Attribute 4317 >> >> > >> >> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb >> >> > 27 Arel::Nodes::SqlLiteral 4174 >> >> > >> >> > t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb >> >> > 110 String 3668 >> >> > >> >> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb >> >> > 9 Arel::Attributes::Attribute 3095 >> >> > >> >> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb >> >> > 4 Arel::Nodes::As 3091 >> >> > >> >> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb >> >> > 4 Arel::Nodes::SqlLiteral 3091 >> >> > >> >> > activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb >> >> > 126 iations::JoinDependency::Aliases::Column 3091 >> >> > >> >> > usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb >> >> > 35 Array 2753 >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb >> >> > 10 Taxonomy::ActiveRecord_Relation 2655 >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb >> >> > 44 ActiveRecord::Relation::Merger 2589 >> >> > >> >> > hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb >> >> > 84 Hash 1825 >> >> > >> >> > s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb >> >> > 63 Arel::Table 1666 >> >> > >> >> > clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb >> >> > 16 Arel::Nodes::Equality 1436 >> >> > >> >> > ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb >> >> > 106 Taxonomy::ActiveRecord_Relation 1180 >> >> > >> >> > sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb >> >> > 5 ActiveRecord::Attribute::FromDatabase 1180 >> >> > >> >> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb >> >> > 88 Arel::Nodes::Casted 1152 >> >> > >> >> > o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb >> >> > 11 Arel::Nodes::SelectStatement 1088 >> >> > >> >> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb >> >> > 9 Arel::Nodes::JoinSource 1088 >> >> > >> >> > root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb >> >> > 7 Arel::Nodes::SelectCore 1088 >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb >> >> > 864 Arel::SelectManager 1088 >> >> > >> >> > /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb >> >> > 81 Hash 992 >> >> > >> >> > hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb >> >> > 26 Taxonomy::ActiveRecord_Relation 885 >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb >> >> > 944 Arel::Nodes::And 880 >> >> > >> >> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb >> >> > 10 Hostgroup::ActiveRecord_Relation 869 >> >> > >> >> > rails objects-total all >> >> > OBJECT COUNT >> >> > Hash 11318 >> >> > Arel::Nodes::SqlLiteral 7566 >> >> > Arel::Attributes::Attribute 7453 >> >> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037 >> >> > Array 6752 >> >> > Taxonomy::ActiveRecord_Relation 4720 >> >> > String 4651 >> >> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091 >> >> > Arel::Nodes::As 3091 >> >> > ActiveRecord::Relation::Merger 2589 >> >> > Arel::Table 1670 >> >> > Time 1597 >> >> > Arel::Nodes::Equality 1436 >> >> > Hostgroup::ActiveRecord_Relation 1283 >> >> > ActiveRecord::Attribute::FromDatabase 1180 >> >> > Arel::Nodes::Casted 1152 >> >> > Arel::SelectManager 1088 >> >> > Arel::Nodes::JoinSource 1088 >> >> > Arel::Nodes::SelectCore 1088 >> >> > Arel::Nodes::SelectStatement 1088 >> >> > TaxableTaxonomy::ActiveRecord_Relation 1065 >> >> > Arel::Nodes::And 1052 >> >> > Set 992 >> >> > VariableLookupKey::ActiveRecord_Relation 984 >> >> > ActiveRecord::Result 949 >> >> > >> >> > >> >> > >> >> > On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal >> >> > wrote: >> >> >> >> >> >> Were you running the "foreman-trace rails calls" utility when you >> >> >> searched the problematic "puppet" query? Because the output looks >> >> >> like >> >> >> sane output, I can see one hundred of NIC models allocated, but this >> >> >> should not take 6 Gbs at all. >> >> >> >> >> >> I've slightly enhanced the foreman-tracer, can you also try to run >> >> >> this while doing the problematic query: >> >> >> >> >> >> foreman-tracer rails objects-total all >> >> >> >> >> >> foreman-tracer rails objects all >> >> >> >> >> >> foreman-tracer rails calls all >> >> >> >> >> >> We must be allocating some number of objects somewhere, this should >> >> >> reveal >> >> >> them. >> >> >> >> >> >> LZ >> >> >> >> >> >> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]> >> >> >> wrote: >> >> >> > This is the problem, it's seems like running a query without any >> >> >> > field >> >> >> > is >> >> >> > causing this problem, it became more aggressive now probably >> >> >> > because >> >> >> > i >> >> >> > am >> >> >> > running it under the "Classes" dashboard which now have much much >> >> >> > more >> >> >> > objects than before. >> >> >> > >> >> >> > Running a query without a specific field: "puppet", memory usage >> >> >> > from >> >> >> > 1G >> >> >> > to >> >> >> > 5G, a ruby instance takes 100% CPU for about a minute. >> >> >> > Running a query with a specific field: "name = puppet", memory >> >> >> > usage >> >> >> > remains >> >> >> > quite the same, small spike in CPU, returns immediately (less than >> >> >> > a >> >> >> > second). >> >> >> > >> >> >> > >> >> >> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum >> >> >> > wrote: >> >> >> >> >> >> >> >> I have the "theforeman/puppet" module, when i search under the >> >> >> >> "Classes" >> >> >> >> dashboard the word "puppet" it's seems to cause a very high CPU >> >> >> >> and >> >> >> >> memory >> >> >> >> starts to increase up to 6GB from almost 1GB. >> >> >> >> >> >> >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum >> >> >> >> wrote: >> >> >> >>> >> >> >> >>> rails objects-total >> >> >> >>> OBJECT COUNT >> >> >> >>> ERB 826 >> >> >> >>> Nic::Managed 104 >> >> >> >>> Foreman::Parameters::Caster 38 >> >> >> >>> IPAddr 24 >> >> >> >>> ConfigReportStatusCalculator 24 >> >> >> >>> ActiveSupport::HashWithIndifferentAccess 18 >> >> >> >>> SafeRender 10 >> >> >> >>> Range 9 >> >> >> >>> ActionView::OutputBuffer 8 >> >> >> >>> Classification::GlobalParam 6 >> >> >> >>> ActionDispatch::Request 6 >> >> >> >>> Foreman::Parameters::Validator 6 >> >> >> >>> OpenSSL::ASN1::ASN1Data 6 >> >> >> >>> OpenSSL::ASN1::Sequence 6 >> >> >> >>> Authorizer 5 >> >> >> >>> CertificateExtract 3 >> >> >> >>> OpenSSL::ASN1::ObjectId 3 >> >> >> >>> OpenSSL::ASN1::OctetString 3 >> >> >> >>> OpenSSL::X509::Certificate 3 >> >> >> >>> Classification::ClassParam 2 >> >> >> >>> ActiveModel::Name 2 >> >> >> >>> PuppetFactParser 1 >> >> >> >>> HostStatus::Global 1 >> >> >> >>> PuppetFactImporter 1 >> >> >> >>> >> >> >> >>> rails objects >> >> >> >>> FILENAME >> >> >> >>> LINE OBJECT COUNT >> >> >> >>> /usr/share/foreman/app/services/safe_render.rb >> >> >> >>> 39 ERB 826 >> >> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb >> >> >> >>> 10 Nic::Managed 96 >> >> >> >>> /usr/share/foreman/app/services/classification/base.rb >> >> >> >>> 151 Foreman::Parameters::Caster 38 >> >> >> >>> /usr/share/foreman/app/models/config_report.rb >> >> >> >>> 88 ConfigReportStatusCalculator 18 >> >> >> >>> /usr/share/foreman/app/models/host/base.rb >> >> >> >>> 390 ActiveSupport::HashWithIndifferentAccess 11 >> >> >> >>> /usr/share/foreman/app/models/concerns/encryptable.rb >> >> >> >>> 50 Range 8 >> >> >> >>> /usr/share/foreman/app/models/host/base.rb >> >> >> >>> 78 Nic::Managed 8 >> >> >> >>> /usr/share/foreman/lib/net/validations.rb >> >> >> >>> 31 IPAddr 8 >> >> >> >>> /usr/share/foreman/lib/net/validations.rb >> >> >> >>> 18 IPAddr 8 >> >> >> >>> /usr/share/foreman/app/services/classification/base.rb >> >> >> >>> 8 SafeRender 8 >> >> >> >>> /usr/share/foreman/lib/net/validations.rb >> >> >> >>> 102 IPAddr 8 >> >> >> >>> /usr/share/foreman/app/models/lookup_value.rb >> >> >> >>> 51 Foreman::Parameters::Validator 6 >> >> >> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb >> >> >> >>> 10 ActionDispatch::Request 6 >> >> >> >>> /usr/share/foreman/app/models/host/managed.rb >> >> >> >>> 934 Classification::GlobalParam 6 >> >> >> >>> >> >> >> >>> /usr/share/foreman/app/models/host_status/configuration_status.rb >> >> >> >>> 112 ConfigReportStatusCalculator 6 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 17 OpenSSL::ASN1::ASN1Data 6 >> >> >> >>> /usr/share/foreman/app/helpers/application_helper.rb >> >> >> >>> 104 Authorizer 5 >> >> >> >>> /usr/share/foreman/app/services/authorizer_cache.rb >> >> >> >>> 3 ActiveSupport::HashWithIndifferentAccess 5 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 5 OpenSSL::X509::Certificate 3 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 16 OpenSSL::ASN1::OctetString 3 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 16 OpenSSL::ASN1::Sequence 3 >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb >> >> >> >>> 59 CertificateExtract 3 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 16 OpenSSL::ASN1::ObjectId 3 >> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb >> >> >> >>> 17 OpenSSL::ASN1::Sequence 3 >> >> >> >>> /usr/share/foreman/app/models/host/managed.rb >> >> >> >>> 420 SafeRender 2 >> >> >> >>> >> >> >> >>> rails calls >> >> >> >>> FILENAME LINE >> >> >> >>> METHOD CALLS >> >> >> >>> /usr/share/foreman/app/models/setting.rb 218 >> >> >> >>> cache 2120 >> >> >> >>> /usr/share/foreman/app/models/setting.rb 72 >> >> >> >>> [] 2086 >> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55 >> >> >> >>> exportable_attributes 2054 >> >> >> >>> /usr/share/foreman/app/services/safe_render.rb 12 >> >> >> >>> parse 1138 >> >> >> >>> /usr/share/foreman/app/services/safe_render.rb 31 >> >> >> >>> parse_string 826 >> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33 >> >> >> >>> export_attr 810 >> >> >> >>> /usr/share/foreman/app/models/fact_value.rb 7 >> >> >> >>> name 509 >> >> >> >>> /usr/share/foreman/app/models/user.rb 137 >> >> >> >>> admin? 180 >> >> >> >>> /usr/share/foreman/app/models/taxonomy.rb 100 >> >> >> >>> expand 180 >> >> >> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92 >> >> >> >>> to_s 112 >> >> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9 >> >> >> >>> setup_object_clone 104 >> >> >> >>> /usr/share/foreman/app/models/nic/managed.rb 45 >> >> >> >>> hostname 96 >> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34 >> >> >> >>> with_taxonomy_scope 84 >> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115 >> >> >> >>> scope_by_taxable_ids 84 >> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75 >> >> >> >>> taxable_ids 84 >> >> >> >>> /usr/share/foreman/app/models/host/base.rb 441 >> >> >> >>> get_interface_by_flag 74 >> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35 >> >> >> >>> call 66 >> >> >> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12 >> >> >> >>> dns_ready? 64 >> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131 >> >> >> >>> set_current_taxonomy 62 >> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14 >> >> >> >>> to_export 62 >> >> >> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9 >> >> >> >>> normalize_recurse 60 >> >> >> >>> /usr/share/foreman/app/models/host/base.rb 237 >> >> >> >>> primary_interface 54 >> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 167 >> >> >> >>> update_generic_matcher 54 >> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 72 >> >> >> >>> value_of_key 54 >> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 73 >> >> >> >>> yield 44 >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> Two or three minutes is just fine, counters do reset after 5 >> >> >> >>>> minutes >> >> >> >>>> so not longer. >> >> >> >>>> >> >> >> >>>> You can also add "rails calls" output as well, that might be >> >> >> >>>> interesting >> >> >> >>>> too. >> >> >> >>>> >> >> >> >>>> LZ >> >> >> >>>> >> >> >> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]> >> >> >> >>>> wrote: >> >> >> >>>> > No problem, for how long you think i should have it running >> >> >> >>>> > to >> >> >> >>>> > give >> >> >> >>>> > valuable >> >> >> >>>> > data to work with? >> >> >> >>>> > I am running only on objects-total and objects (as you >> >> >> >>>> > mentioned) >> >> >> >>>> > >> >> >> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas >> >> >> >>>> > Zapletal >> >> >> >>>> > wrote: >> >> >> >>>> >> >> >> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, >> >> >> >>>> >> will >> >> >> >>>> >> only >> >> >> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your >> >> >> >>>> >> production >> >> >> >>>> >> instance? No changes required in Foreman, setup is quite >> >> >> >>>> >> easy: >> >> >> >>>> >> >> >> >> >>>> >> https://github.com/lzap/foreman-tracer >> >> >> >>>> >> >> >> >> >>>> >> Interesting statistics would be >> >> >> >>>> >> >> >> >> >>>> >> foreman-tracer rails objects-total >> >> >> >>>> >> >> >> >> >>>> >> and >> >> >> >>>> >> >> >> >> >>>> >> foreman-tracer rails objects >> >> >> >>>> >> >> >> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks >> >> >> >>>> >> please. >> >> >> >>>> >> >> >> >> >>>> >> Thanks >> >> >> >>>> >> >> >> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum >> >> >> >>>> >> <[email protected]> >> >> >> >>>> >> wrote: >> >> >> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from >> >> >> >>>> >> > 1.12.1 >> >> >> >>>> >> > and >> >> >> >>>> >> > experience the same performance degradation on the foreman >> >> >> >>>> >> > web >> >> >> >>>> >> > app, >> >> >> >>>> >> > ruby >> >> >> >>>> >> > processes taking a lot of memory, no new plugins were >> >> >> >>>> >> > introduced. >> >> >> >>>> >> > Same as what was written above, I suddenly see ruby >> >> >> >>>> >> > instances >> >> >> >>>> >> > using >> >> >> >>>> >> > of >> >> >> >>>> >> > up to >> >> >> >>>> >> > 4GB RAM/instance as of before i never saw this happens. >> >> >> >>>> >> > >> >> >> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad >> >> >> >>>> >> > wrote: >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh >> >> >> >>>> >> >> <[email protected]> >> >> >> >>>> >> >> wrote: >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> Hi all, >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 >> >> >> >>>> >> >>> and >> >> >> >>>> >> >>> are >> >> >> >>>> >> >>> finding >> >> >> >>>> >> >>> that Foreman's ENC/facts/reports components are >> >> >> >>>> >> >>> requiring a >> >> >> >>>> >> >>> lot >> >> >> >>>> >> >>> more >> >> >> >>>> >> >>> resources than the old version. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them >> >> >> >>>> >> >>> checking >> >> >> >>>> >> >>> in >> >> >> >>>> >> >>> with >> >> >> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs >> >> >> >>>> >> >>> under >> >> >> >>>> >> >>> passenger >> >> >> >>>> >> >>> and >> >> >> >>>> >> >>> has a mysql database. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server >> >> >> >>>> >> >>> handling >> >> >> >>>> >> >>> the >> >> >> >>>> >> >>> web >> >> >> >>>> >> >>> UI, >> >> >> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have >> >> >> >>>> >> >>> expanded >> >> >> >>>> >> >>> to >> >> >> >>>> >> >>> 4 >> >> >> >>>> >> >>> x 8 >> >> >> >>>> >> >>> vCPU, >> >> >> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and >> >> >> >>>> >> >>> 1 >> >> >> >>>> >> >>> x 8 >> >> >> >>>> >> >>> vCPU, >> >> >> >>>> >> >>> 32GB >> >> >> >>>> >> >>> RAM Foreman server handling the web UI and database. We >> >> >> >>>> >> >>> are >> >> >> >>>> >> >>> currently >> >> >> >>>> >> >>> restarting apache on the ENC/fact/report servers every >> >> >> >>>> >> >>> 20 >> >> >> >>>> >> >>> minutes >> >> >> >>>> >> >>> to >> >> >> >>>> >> >>> stop >> >> >> >>>> >> >>> them running out of memory (if we tune passenger more >> >> >> >>>> >> >>> aggressively, >> >> >> >>>> >> >>> the >> >> >> >>>> >> >>> request queue fills up). >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> I noticed a couple of other posts in the last few days >> >> >> >>>> >> >>> regarding >> >> >> >>>> >> >>> performance. Is anyone running a similarly-sized >> >> >> >>>> >> >>> installation >> >> >> >>>> >> >>> on >> >> >> >>>> >> >>> version >> >> >> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any >> >> >> >>>> >> >>> tips >> >> >> >>>> >> >>> for >> >> >> >>>> >> >>> tuning >> >> >> >>>> >> >>> passenger or other components? >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any >> >> >> >>>> >> >> information you >> >> >> >>>> >> >> could share that would provide some light on where things >> >> >> >>>> >> >> are >> >> >> >>>> >> >> slow? >> >> >> >>>> >> >> (e.g. >> >> >> >>>> >> >> slow operations, etc). >> >> >> >>>> >> >> >> >> >> >>>> >> >> thanks, >> >> >> >>>> >> >> Ohad >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> Regards >> >> >> >>>> >> >>> Guy. >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> -- >> >> >> >>>> >> >>> You received this message because you are subscribed to >> >> >> >>>> >> >>> the >> >> >> >>>> >> >>> Google >> >> >> >>>> >> >>> Groups >> >> >> >>>> >> >>> "Foreman users" group. >> >> >> >>>> >> >>> To unsubscribe from this group and stop receiving emails >> >> >> >>>> >> >>> from >> >> >> >>>> >> >>> it, >> >> >> >>>> >> >>> send >> >> >> >>>> >> >>> an >> >> >> >>>> >> >>> email to [email protected]. >> >> >> >>>> >> >>> To post to this group, send email to >> >> >> >>>> >> >>> [email protected]. >> >> >> >>>> >> >>> Visit this group at >> >> >> >>>> >> >>> https://groups.google.com/group/foreman-users. >> >> >> >>>> >> >>> For more options, visit >> >> >> >>>> >> >>> https://groups.google.com/d/optout. >> >> >> >>>> >> >> >> >> >> >>>> >> >> >> >> >> >>>> >> > -- >> >> >> >>>> >> > You received this message because you are subscribed to >> >> >> >>>> >> > the >> >> >> >>>> >> > Google >> >> >> >>>> >> > Groups >> >> >> >>>> >> > "Foreman users" group. >> >> >> >>>> >> > To unsubscribe from this group and stop receiving emails >> >> >> >>>> >> > from >> >> >> >>>> >> > it, >> >> >> >>>> >> > send >> >> >> >>>> >> > an >> >> >> >>>> >> > email to [email protected]. >> >> >> >>>> >> > To post to this group, send email to >> >> >> >>>> >> > [email protected]. >> >> >> >>>> >> > Visit this group at >> >> >> >>>> >> > https://groups.google.com/group/foreman-users. >> >> >> >>>> >> > For more options, visit >> >> >> >>>> >> > https://groups.google.com/d/optout. >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> -- >> >> >> >>>> >> Later, >> >> >> >>>> >> Lukas @lzap Zapletal >> >> >> >>>> > >> >> >> >>>> > -- >> >> >> >>>> > You received this message because you are subscribed to the >> >> >> >>>> > Google >> >> >> >>>> > Groups >> >> >> >>>> > "Foreman users" group. >> >> >> >>>> > To unsubscribe from this group and stop receiving emails from >> >> >> >>>> > it, >> >> >> >>>> > send >> >> >> >>>> > an >> >> >> >>>> > email to [email protected]. >> >> >> >>>> > To post to this group, send email to >> >> >> >>>> > [email protected]. >> >> >> >>>> > Visit this group at >> >> >> >>>> > https://groups.google.com/group/foreman-users. >> >> >> >>>> > For more options, visit https://groups.google.com/d/optout. >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> -- >> >> >> >>>> Later, >> >> >> >>>> Lukas @lzap Zapletal >> >> >> > >> >> >> > -- >> >> >> > You received this message because you are subscribed to the Google >> >> >> > Groups >> >> >> > "Foreman users" group. >> >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> >> > send >> >> >> > an >> >> >> > email to [email protected]. >> >> >> > To post to this group, send email to [email protected]. >> >> >> > Visit this group at https://groups.google.com/group/foreman-users. >> >> >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Later, >> >> >> Lukas @lzap Zapletal >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Foreman users" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> > send >> >> > an >> >> > email to [email protected]. >> >> > To post to this group, send email to [email protected]. >> >> > Visit this group at https://groups.google.com/group/foreman-users. >> >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> >> >> -- >> >> Later, >> >> Lukas @lzap Zapletal >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Foreman users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to [email protected]. >> > To post to this group, send email to [email protected]. >> > Visit this group at https://groups.google.com/group/foreman-users. >> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Later, >> Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
