Hey,
It doesn't really let me know, but i'm not sure it's the same issue, this
only happens when searching the dashboard without using any specific fields
("puppet" vs "name = puppet"), i believe this has nothing to do with any
facts API calls (internally).
2016-11-03 13:22:38 b2812ac1 [app] [I] Started GET
"/puppetclasses?utf8=%E2%9C%93&search=puppet" for 10.20.30.11 at 2016-11-03
13:22:38 +0000
2016-11-03 13:22:38 b2812ac1 [app] [I] Processing by
PuppetclassesController#index as HTML
2016-11-03 13:22:38 b2812ac1 [app] [I] Parameters: {"utf8"=>"✓",
"search"=>"puppet"}
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
puppetclasses/index.html.erb within layouts/application (1176.4ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
common/_searchbar.html.erb (14.7ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
layouts/_application_content.html.erb (16.0ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
home/_user_dropdown.html.erb (3.9ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Read fragment
views/tabs_and_title_records-4 (0.1ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
home/_organization_dropdown.html.erb (16.1ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
home/_location_dropdown.html.erb (21.1ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered
home/_org_switcher.html.erb (38.6ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb
(4.1ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb
(3.0ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb
(2.2ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb
(2.0ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_submenu.html.erb
(2.8ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Write fragment
views/tabs_and_title_records-4 (0.5ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered home/_topbar.html.erb
(131.1ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Rendered layouts/base.html.erb
(132.8ms)
2016-11-03 13:23:40 b2812ac1 [app] [I] Completed 200 OK in 62045ms (Views:
1218.4ms | ActiveRecord: 4650.5ms)
Even if this is expected, there should be some kind of protection again
those kind of queries or set a default field when no field is set.
On Thursday, November 3, 2016 at 2:45:43 PM UTC+2, Lukas Zapletal wrote:
>
> Thanks,
>
> all I can say from these outputs is we have some eager loading of data
> into hash most likely. Can you tell from the production.log file which
> HTTP calls are slowest? Is it ENC, report or facts? I bet it's facts,
> but please confirm.
>
> There were some performance issues and improvements in both 1.12 and
> 1.13, particularly:
>
> http://projects.theforeman.org/issues/16749 - I think this might be
> relevant
> http://projects.theforeman.org/issues/9016
> http://projects.theforeman.org/issues/16576
>
> Do you have some hosts with high number of NICs (ipaddress facts)
> reported?
>
> LZ
>
> On Thu, Nov 3, 2016 at 11:26 AM, Erez Zarum <[email protected]
> <javascript:>> wrote:
> > It's not the terminal size issue, the format is set by the script, for
> this
> > tasks i increased it (Line 92,94,95,190,192)
> >
> > rails calls all
> >
> > FILENAME LINE METHOD CALLS
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 260 hash 51976
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 160 nil? 48142
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 171 association_instance_get 48142
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 157 association 48142
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb
>
>
> > 63 loaded! 47962
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 116 to_s 37385
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 115 _reflect_on_association 37385
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 name 37385
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb
>
>
> > 86 _reflections 37385
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 options 37314
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 active_record 37292
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 686 through_reflection 37292
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 442 belongs_to? 37207
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> > 67 stale_state 37204
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> > 6 through_reflection 37204
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 592 collection? 37157
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb
>
>
> > 72 primary_key 37059
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb
>
>
> > 18 primary_key 36975
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 31 column_alias 36975
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 32 hash 36975
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 254 nil? 36974
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
>
>
> > 39 [] 35363
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> > 9 [] 35363
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
>
>
> > 35 value 35340
> >
> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> > 30 fetch_value 35317
> >
> > rails objects-total all
> > OBJECT COUNT
> > Hash 8495
> > Arel::Nodes::SqlLiteral 7548
> > Arel::Attributes::Attribute 7401
> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 6981
> > Array 6478
> > Taxonomy::ActiveRecord_Relation 4720
> > String 4409
> > Arel::Nodes::As 3091
> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
> > ActiveRecord::Relation::Merger 2563
> > Arel::Table 1669
> > Time 1508
> > Arel::Nodes::Equality 1428
> > Hostgroup::ActiveRecord_Relation 1283
> > ActiveRecord::Attribute::FromDatabase 1146
> > Arel::Nodes::Casted 1136
> > Arel::Nodes::SelectStatement 1068
> > Arel::Nodes::JoinSource 1068
> > Arel::SelectManager 1068
> > Arel::Nodes::SelectCore 1068
> > TaxableTaxonomy::ActiveRecord_Relation 1065
> > Arel::Nodes::And 1032
> > VariableLookupKey::ActiveRecord_Relation 984
> > Set 960
> > ActiveRecord::Result 916
> >
> > I couldn't understand what you mean about the interfaces, though i have
> only
> > one host with about 7 interfaces, i saw that post but it's seems not
> > relevant to me.
> >
> > On Wednesday, November 2, 2016 at 5:40:54 PM UTC+2, Lukas Zapletal
> wrote:
> >>
> >> That's expected, the tracer is topped to prevent eating lot of memory,
> >> so it rather crash than consume more. I have slightly increased the
> >> limit, can you update the script and run once again, now for the last
> >> time:
> >>
> >> rails objects-total app 100
> >>
> >> and
> >>
> >> rails calls all 100
> >>
> >> Please set minimum font size for your terminal to fit 100 rows. I am
> >> just curious if we can spot the Model that is causing this.
> >>
> >> But I suspect this is the similar issue that someone else just had
> >> (see replies from today in the list). Hosts were created with too many
> >> NICs. The suggestion was: "See the 'ignore interfaces with matching
> >> identifier' option under Settings -> Provisioning." Then you need to
> >> delete all the NICs from the database to get things rolling again.
> >>
> >> LZ
> >>
> >> On Wed, Nov 2, 2016 at 2:50 PM, Erez Zarum <[email protected]> wrote:
> >> > Hey,
> >> > I think you nailed it :)
> >> >
> >> > rails calls all probe failed at some point though.
> >> >
> >> > rails calls all
> >> >
> >> > FILENAME LINE METHOD CALLS
> >> >
> >> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> >> > 260 hash 86006
> >> >
> >> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> >> > 171 association_instance_get 76602
> >> >
> >> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> >> > 157 association 76602
> >> >
> >> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> >> > 160 nil? 76602
> >> >
> >> >
> are/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb
>
>
> >> > 63 loaded! 76422
> >> >
> >> >
> e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb
>
>
> >> > 86 _reflections 60166
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 115 _reflect_on_association 60165
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 116 to_s 60164
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 889 name 60162
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 889 options 60086
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 686 through_reflection 60065
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 889 active_record 60065
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 442 belongs_to? 59980
> >> >
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> >> > 67 stale_state 59979
> >> >
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> >> > 6 through_reflection 59979
> >> >
> >> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> >> > 592 collection? 59894
> >> >
> >> >
> ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb
>
>
> >> > 72 primary_key 59802
> >> >
> >> >
> activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb
>
>
> >> > 18 primary_key 59712
> >> >
> >> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> >> > 31 column_alias 59711
> >> >
> >> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> >> > 254 nil? 59710
> >> >
> >> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> >> > 32 hash 59710
> >> >
> >> >
> /share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
>
>
> >> > 39 [] 58136
> >> >
> >> >
> root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> >> > 9 [] 58136
> >> >
> >> >
> r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
>
>
> >> > 35 value 58078
> >> >
> >> >
> root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> >> > 30 fetch_value 58042
> >> > WARNING: Number of errors: 0, skipped probes: 101
> >> > WARNING: /usr/bin/staprun exited with status: 1
> >> > Pass 5: run failed. [man error::pass5]
> >> >
> >> > rails objects all
> >> >
> FILENAME
> >> > LINE OBJECT COUNT
> >> >
> cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb
> >> > 72 rd::ConnectionAdapters::PostgreSQL::Name 7037
> >> >
> pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb
> >> > 100 Arel::Attributes::Attribute 4317
> >> >
> /opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb
> >> > 27 Arel::Nodes::SqlLiteral 4174
> >> >
> t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb
> >> > 110 String 3668
> >> >
> or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb
> >> > 9 Arel::Attributes::Attribute 3095
> >> >
> or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> >> > 4 Arel::Nodes::As 3091
> >> >
> or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> >> > 4 Arel::Nodes::SqlLiteral 3091
> >> >
> activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
> >> > 126 iations::JoinDependency::Aliases::Column 3091
> >> >
> usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb
> >> > 35 Array 2753
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> >> > 10 Taxonomy::ActiveRecord_Relation 2655
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> >> > 44 ActiveRecord::Relation::Merger 2589
> >> >
> hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb
> >> > 84 Hash 1825
> >> >
> s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb
> >> > 63 Arel::Table 1666
> >> >
> clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb
> >> > 16 Arel::Nodes::Equality 1436
> >> >
> ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb
> >> > 106 Taxonomy::ActiveRecord_Relation 1180
> >> >
> sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
> >> > 5 ActiveRecord::Attribute::FromDatabase 1180
> >> >
> pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb
> >> > 88 Arel::Nodes::Casted 1152
> >> >
> o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb
> >> > 11 Arel::Nodes::SelectStatement 1088
> >> >
> or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb
> >> > 9 Arel::Nodes::JoinSource 1088
> >> >
> root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb
> >> > 7 Arel::Nodes::SelectCore 1088
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> >> > 864 Arel::SelectManager 1088
> >> >
> /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb
> >> > 81 Hash 992
> >> >
> hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb
> >> > 26 Taxonomy::ActiveRecord_Relation 885
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> >> > 944 Arel::Nodes::And 880
> >> >
> /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> >> > 10 Hostgroup::ActiveRecord_Relation 869
> >> >
> >> > rails objects-total all
> >> > OBJECT COUNT
> >> > Hash 11318
> >> > Arel::Nodes::SqlLiteral 7566
> >> > Arel::Attributes::Attribute 7453
> >> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037
> >> > Array 6752
> >> > Taxonomy::ActiveRecord_Relation 4720
> >> > String 4651
> >> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
> >> > Arel::Nodes::As 3091
> >> > ActiveRecord::Relation::Merger 2589
> >> > Arel::Table 1670
> >> > Time 1597
> >> > Arel::Nodes::Equality 1436
> >> > Hostgroup::ActiveRecord_Relation 1283
> >> > ActiveRecord::Attribute::FromDatabase 1180
> >> > Arel::Nodes::Casted 1152
> >> > Arel::SelectManager 1088
> >> > Arel::Nodes::JoinSource 1088
> >> > Arel::Nodes::SelectCore 1088
> >> > Arel::Nodes::SelectStatement 1088
> >> > TaxableTaxonomy::ActiveRecord_Relation 1065
> >> > Arel::Nodes::And 1052
> >> > Set 992
> >> > VariableLookupKey::ActiveRecord_Relation 984
> >> > ActiveRecord::Result 949
> >> >
> >> >
> >> >
> >> > On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal
> >> > wrote:
> >> >>
> >> >> Were you running the "foreman-trace rails calls" utility when you
> >> >> searched the problematic "puppet" query? Because the output looks
> like
> >> >> sane output, I can see one hundred of NIC models allocated, but this
> >> >> should not take 6 Gbs at all.
> >> >>
> >> >> I've slightly enhanced the foreman-tracer, can you also try to run
> >> >> this while doing the problematic query:
> >> >>
> >> >> foreman-tracer rails objects-total all
> >> >>
> >> >> foreman-tracer rails objects all
> >> >>
> >> >> foreman-tracer rails calls all
> >> >>
> >> >> We must be allocating some number of objects somewhere, this should
> >> >> reveal
> >> >> them.
> >> >>
> >> >> LZ
> >> >>
> >> >> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]>
> wrote:
> >> >> > This is the problem, it's seems like running a query without any
> >> >> > field
> >> >> > is
> >> >> > causing this problem, it became more aggressive now probably
> because
> >> >> > i
> >> >> > am
> >> >> > running it under the "Classes" dashboard which now have much much
> >> >> > more
> >> >> > objects than before.
> >> >> >
> >> >> > Running a query without a specific field: "puppet", memory usage
> from
> >> >> > 1G
> >> >> > to
> >> >> > 5G, a ruby instance takes 100% CPU for about a minute.
> >> >> > Running a query with a specific field: "name = puppet", memory
> usage
> >> >> > remains
> >> >> > quite the same, small spike in CPU, returns immediately (less than
> a
> >> >> > second).
> >> >> >
> >> >> >
> >> >> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum
> wrote:
> >> >> >>
> >> >> >> I have the "theforeman/puppet" module, when i search under the
> >> >> >> "Classes"
> >> >> >> dashboard the word "puppet" it's seems to cause a very high CPU
> and
> >> >> >> memory
> >> >> >> starts to increase up to 6GB from almost 1GB.
> >> >> >>
> >> >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum
> wrote:
> >> >> >>>
> >> >> >>> rails objects-total
> >> >> >>> OBJECT COUNT
> >> >> >>> ERB 826
> >> >> >>> Nic::Managed 104
> >> >> >>> Foreman::Parameters::Caster 38
> >> >> >>> IPAddr 24
> >> >> >>> ConfigReportStatusCalculator 24
> >> >> >>> ActiveSupport::HashWithIndifferentAccess 18
> >> >> >>> SafeRender 10
> >> >> >>> Range 9
> >> >> >>> ActionView::OutputBuffer 8
> >> >> >>> Classification::GlobalParam 6
> >> >> >>> ActionDispatch::Request 6
> >> >> >>> Foreman::Parameters::Validator 6
> >> >> >>> OpenSSL::ASN1::ASN1Data 6
> >> >> >>> OpenSSL::ASN1::Sequence 6
> >> >> >>> Authorizer 5
> >> >> >>> CertificateExtract 3
> >> >> >>> OpenSSL::ASN1::ObjectId 3
> >> >> >>> OpenSSL::ASN1::OctetString 3
> >> >> >>> OpenSSL::X509::Certificate 3
> >> >> >>> Classification::ClassParam 2
> >> >> >>> ActiveModel::Name 2
> >> >> >>> PuppetFactParser 1
> >> >> >>> HostStatus::Global 1
> >> >> >>> PuppetFactImporter 1
> >> >> >>>
> >> >> >>> rails objects
> >> >> >>> FILENAME
> >> >> >>> LINE OBJECT COUNT
> >> >> >>> /usr/share/foreman/app/services/safe_render.rb
> >> >> >>> 39 ERB 826
> >> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb
> >> >> >>> 10 Nic::Managed 96
> >> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >> >>> 151 Foreman::Parameters::Caster 38
> >> >> >>> /usr/share/foreman/app/models/config_report.rb
> >> >> >>> 88 ConfigReportStatusCalculator 18
> >> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >> >>> 390 ActiveSupport::HashWithIndifferentAccess 11
> >> >> >>> /usr/share/foreman/app/models/concerns/encryptable.rb
> >> >> >>> 50 Range 8
> >> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >> >>> 78 Nic::Managed 8
> >> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >> >>> 31 IPAddr 8
> >> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >> >>> 18 IPAddr 8
> >> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >> >>> 8 SafeRender 8
> >> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >> >>> 102 IPAddr 8
> >> >> >>> /usr/share/foreman/app/models/lookup_value.rb
> >> >> >>> 51 Foreman::Parameters::Validator 6
> >> >> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb
> >> >> >>> 10 ActionDispatch::Request 6
> >> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >> >>> 934 Classification::GlobalParam 6
> >> >> >>>
> /usr/share/foreman/app/models/host_status/configuration_status.rb
> >> >> >>> 112 ConfigReportStatusCalculator 6
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 17 OpenSSL::ASN1::ASN1Data 6
> >> >> >>> /usr/share/foreman/app/helpers/application_helper.rb
> >> >> >>> 104 Authorizer 5
> >> >> >>> /usr/share/foreman/app/services/authorizer_cache.rb
> >> >> >>> 3 ActiveSupport::HashWithIndifferentAccess 5
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 5 OpenSSL::X509::Certificate 3
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 16 OpenSSL::ASN1::OctetString 3
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 16 OpenSSL::ASN1::Sequence 3
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb
>
>
> >> >> >>> 59 CertificateExtract 3
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 16 OpenSSL::ASN1::ObjectId 3
> >> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >> >>> 17 OpenSSL::ASN1::Sequence 3
> >> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >> >>> 420 SafeRender 2
> >> >> >>>
> >> >> >>> rails calls
> >> >> >>> FILENAME LINE
> >> >> >>> METHOD CALLS
> >> >> >>> /usr/share/foreman/app/models/setting.rb 218
> >> >> >>> cache 2120
> >> >> >>> /usr/share/foreman/app/models/setting.rb 72
> >> >> >>> [] 2086
> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55
> >> >> >>> exportable_attributes 2054
> >> >> >>> /usr/share/foreman/app/services/safe_render.rb 12
> >> >> >>> parse 1138
> >> >> >>> /usr/share/foreman/app/services/safe_render.rb 31
> >> >> >>> parse_string 826
> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33
> >> >> >>> export_attr 810
> >> >> >>> /usr/share/foreman/app/models/fact_value.rb 7
> >> >> >>> name 509
> >> >> >>> /usr/share/foreman/app/models/user.rb 137
> >> >> >>> admin? 180
> >> >> >>> /usr/share/foreman/app/models/taxonomy.rb 100
> >> >> >>> expand 180
> >> >> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92
> >> >> >>> to_s 112
> >> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9
> >> >> >>> setup_object_clone 104
> >> >> >>> /usr/share/foreman/app/models/nic/managed.rb 45
> >> >> >>> hostname 96
> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34
> >> >> >>> with_taxonomy_scope 84
> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115
> >> >> >>> scope_by_taxable_ids 84
> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75
> >> >> >>> taxable_ids 84
> >> >> >>> /usr/share/foreman/app/models/host/base.rb 441
> >> >> >>> get_interface_by_flag 74
> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35
> >> >> >>> call 66
> >> >> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12
> >> >> >>> dns_ready? 64
> >> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131
> >> >> >>> set_current_taxonomy 62
> >> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14
> >> >> >>> to_export 62
> >> >> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9
> >> >> >>> normalize_recurse 60
> >> >> >>> /usr/share/foreman/app/models/host/base.rb 237
> >> >> >>> primary_interface 54
> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 167
> >> >> >>> update_generic_matcher 54
> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 72
> >> >> >>> value_of_key 54
> >> >> >>> /usr/share/foreman/app/services/classification/base.rb 73
> >> >> >>> yield 44
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal
> >> >> >>> wrote:
> >> >> >>>>
> >> >> >>>> Two or three minutes is just fine, counters do reset after 5
> >> >> >>>> minutes
> >> >> >>>> so not longer.
> >> >> >>>>
> >> >> >>>> You can also add "rails calls" output as well, that might be
> >> >> >>>> interesting
> >> >> >>>> too.
> >> >> >>>>
> >> >> >>>> LZ
> >> >> >>>>
> >> >> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]>
>
> >> >> >>>> wrote:
> >> >> >>>> > No problem, for how long you think i should have it running
> to
> >> >> >>>> > give
> >> >> >>>> > valuable
> >> >> >>>> > data to work with?
> >> >> >>>> > I am running only on objects-total and objects (as you
> >> >> >>>> > mentioned)
> >> >> >>>> >
> >> >> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas
> Zapletal
> >> >> >>>> > wrote:
> >> >> >>>> >>
> >> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based,
> will
> >> >> >>>> >> only
> >> >> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your
> >> >> >>>> >> production
> >> >> >>>> >> instance? No changes required in Foreman, setup is quite
> easy:
> >> >> >>>> >>
> >> >> >>>> >> https://github.com/lzap/foreman-tracer
> >> >> >>>> >>
> >> >> >>>> >> Interesting statistics would be
> >> >> >>>> >>
> >> >> >>>> >> foreman-tracer rails objects-total
> >> >> >>>> >>
> >> >> >>>> >> and
> >> >> >>>> >>
> >> >> >>>> >> foreman-tracer rails objects
> >> >> >>>> >>
> >> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks
> >> >> >>>> >> please.
> >> >> >>>> >>
> >> >> >>>> >> Thanks
> >> >> >>>> >>
> >> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum
> >> >> >>>> >> <[email protected]>
> >> >> >>>> >> wrote:
> >> >> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from
> >> >> >>>> >> > 1.12.1
> >> >> >>>> >> > and
> >> >> >>>> >> > experience the same performance degradation on the foreman
> >> >> >>>> >> > web
> >> >> >>>> >> > app,
> >> >> >>>> >> > ruby
> >> >> >>>> >> > processes taking a lot of memory, no new plugins were
> >> >> >>>> >> > introduced.
> >> >> >>>> >> > Same as what was written above, I suddenly see ruby
> instances
> >> >> >>>> >> > using
> >> >> >>>> >> > of
> >> >> >>>> >> > up to
> >> >> >>>> >> > 4GB RAM/instance as of before i never saw this happens.
> >> >> >>>> >> >
> >> >> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad
> wrote:
> >> >> >>>> >> >>
> >> >> >>>> >> >>
> >> >> >>>> >> >>
> >> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh
> >> >> >>>> >> >> <[email protected]>
> >> >> >>>> >> >> wrote:
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> Hi all,
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3
> and
> >> >> >>>> >> >>> are
> >> >> >>>> >> >>> finding
> >> >> >>>> >> >>> that Foreman's ENC/facts/reports components are
> requiring a
> >> >> >>>> >> >>> lot
> >> >> >>>> >> >>> more
> >> >> >>>> >> >>> resources than the old version.
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them
> >> >> >>>> >> >>> checking
> >> >> >>>> >> >>> in
> >> >> >>>> >> >>> with
> >> >> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs
> >> >> >>>> >> >>> under
> >> >> >>>> >> >>> passenger
> >> >> >>>> >> >>> and
> >> >> >>>> >> >>> has a mysql database.
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server
> >> >> >>>> >> >>> handling
> >> >> >>>> >> >>> the
> >> >> >>>> >> >>> web
> >> >> >>>> >> >>> UI,
> >> >> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have
> expanded
> >> >> >>>> >> >>> to
> >> >> >>>> >> >>> 4
> >> >> >>>> >> >>> x 8
> >> >> >>>> >> >>> vCPU,
> >> >> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and
> 1
> >> >> >>>> >> >>> x 8
> >> >> >>>> >> >>> vCPU,
> >> >> >>>> >> >>> 32GB
> >> >> >>>> >> >>> RAM Foreman server handling the web UI and database. We
> are
> >> >> >>>> >> >>> currently
> >> >> >>>> >> >>> restarting apache on the ENC/fact/report servers every
> 20
> >> >> >>>> >> >>> minutes
> >> >> >>>> >> >>> to
> >> >> >>>> >> >>> stop
> >> >> >>>> >> >>> them running out of memory (if we tune passenger more
> >> >> >>>> >> >>> aggressively,
> >> >> >>>> >> >>> the
> >> >> >>>> >> >>> request queue fills up).
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> I noticed a couple of other posts in the last few days
> >> >> >>>> >> >>> regarding
> >> >> >>>> >> >>> performance. Is anyone running a similarly-sized
> >> >> >>>> >> >>> installation
> >> >> >>>> >> >>> on
> >> >> >>>> >> >>> version
> >> >> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any
> >> >> >>>> >> >>> tips
> >> >> >>>> >> >>> for
> >> >> >>>> >> >>> tuning
> >> >> >>>> >> >>> passenger or other components?
> >> >> >>>> >> >>
> >> >> >>>> >> >>
> >> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any
> >> >> >>>> >> >> information you
> >> >> >>>> >> >> could share that would provide some light on where things
> >> >> >>>> >> >> are
> >> >> >>>> >> >> slow?
> >> >> >>>> >> >> (e.g.
> >> >> >>>> >> >> slow operations, etc).
> >> >> >>>> >> >>
> >> >> >>>> >> >> thanks,
> >> >> >>>> >> >> Ohad
> >> >> >>>> >> >>>
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> Regards
> >> >> >>>> >> >>> Guy.
> >> >> >>>> >> >>>
> >> >> >>>> >> >>> --
> >> >> >>>> >> >>> You received this message because you are subscribed to
> the
> >> >> >>>> >> >>> Google
> >> >> >>>> >> >>> Groups
> >> >> >>>> >> >>> "Foreman users" group.
> >> >> >>>> >> >>> To unsubscribe from this group and stop receiving emails
> >> >> >>>> >> >>> from
> >> >> >>>> >> >>> it,
> >> >> >>>> >> >>> send
> >> >> >>>> >> >>> an
> >> >> >>>> >> >>> email to [email protected].
> >> >> >>>> >> >>> To post to this group, send email to
> >> >> >>>> >> >>> [email protected].
> >> >> >>>> >> >>> Visit this group at
> >> >> >>>> >> >>> https://groups.google.com/group/foreman-users.
> >> >> >>>> >> >>> For more options, visit
> https://groups.google.com/d/optout.
> >> >> >>>> >> >>
> >> >> >>>> >> >>
> >> >> >>>> >> > --
> >> >> >>>> >> > You received this message because you are subscribed to
> the
> >> >> >>>> >> > Google
> >> >> >>>> >> > Groups
> >> >> >>>> >> > "Foreman users" group.
> >> >> >>>> >> > To unsubscribe from this group and stop receiving emails
> from
> >> >> >>>> >> > it,
> >> >> >>>> >> > send
> >> >> >>>> >> > an
> >> >> >>>> >> > email to [email protected].
> >> >> >>>> >> > To post to this group, send email to
> >> >> >>>> >> > [email protected].
> >> >> >>>> >> > Visit this group at
> >> >> >>>> >> > https://groups.google.com/group/foreman-users.
> >> >> >>>> >> > For more options, visit https://groups.google.com/d/optout.
>
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >>
> >> >> >>>> >> --
> >> >> >>>> >> Later,
> >> >> >>>> >> Lukas @lzap Zapletal
> >> >> >>>> >
> >> >> >>>> > --
> >> >> >>>> > You received this message because you are subscribed to the
> >> >> >>>> > Google
> >> >> >>>> > Groups
> >> >> >>>> > "Foreman users" group.
> >> >> >>>> > To unsubscribe from this group and stop receiving emails from
> >> >> >>>> > it,
> >> >> >>>> > send
> >> >> >>>> > an
> >> >> >>>> > email to [email protected].
> >> >> >>>> > To post to this group, send email to
> [email protected].
> >> >> >>>> > Visit this group at
> >> >> >>>> > https://groups.google.com/group/foreman-users.
> >> >> >>>> > For more options, visit https://groups.google.com/d/optout.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Later,
> >> >> >>>> Lukas @lzap Zapletal
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "Foreman users" group.
> >> >> > To unsubscribe from this group and stop receiving emails from it,
> >> >> > send
> >> >> > an
> >> >> > email to [email protected].
> >> >> > To post to this group, send email to [email protected].
> >> >> > Visit this group at https://groups.google.com/group/foreman-users.
>
> >> >> > For more options, visit https://groups.google.com/d/optout.
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Later,
> >> >> Lukas @lzap Zapletal
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Foreman users" group.
> >> > To unsubscribe from this group and stop receiving emails from it,
> send
> >> > an
> >> > email to [email protected].
> >> > To post to this group, send email to [email protected].
> >> > Visit this group at https://groups.google.com/group/foreman-users.
> >> > For more options, visit https://groups.google.com/d/optout.
> >>
> >>
> >>
> >> --
> >> Later,
> >> Lukas @lzap Zapletal
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to [email protected] <javascript:>.
> > To post to this group, send email to [email protected]
> <javascript:>.
> > Visit this group at https://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
