Hey,
I think you nailed it :)
rails calls all probe failed at some point though.
*rails calls all*
FILENAME LINE METHOD CALLS
gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
260 hash 86006
/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
171 association_instance_get 76602
/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
157 association 76602
/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
160 nil? 76602
are/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb
63 loaded! 76422
e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb
86 _reflections 60166
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
115 _reflect_on_association 60165
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
116 to_s 60164
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 name 60162
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 options 60086
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
686 through_reflection 60065
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 active_record 60065
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
442 belongs_to? 59980
/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
67 stale_state 59979
/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
6 through_reflection 59979
42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
592 collection? 59894
ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb
72 primary_key 59802
activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb
18 primary_key 59712
gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
31 column_alias 59711
gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
254 nil? 59710
gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
32 hash 59710
/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
39 [] 58136
root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
9 [] 58136
r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
35 value 58078
root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
30 fetch_value 58042
WARNING: Number of errors: 0, skipped probes: 101
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run failed. [man error::pass5]
*rails objects all*
FILENAME
LINE OBJECT COUNT
cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb
72 rd::ConnectionAdapters::PostgreSQL::Name 7037
pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb
100 Arel::Attributes::Attribute 4317
/opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb
27 Arel::Nodes::SqlLiteral 4174
t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb
110 String 3668
or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb
9 Arel::Attributes::Attribute 3095
or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
4 Arel::Nodes::As 3091
or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
4 Arel::Nodes::SqlLiteral 3091
activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
126 iations::JoinDependency::Aliases::Column 3091
usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb
35 Array 2753
/gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
10 Taxonomy::ActiveRecord_Relation 2655
/gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
44 ActiveRecord::Relation::Merger 2589
hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb
84 Hash 1825
s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb
63 Arel::Table 1666
clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb
16 Arel::Nodes::Equality 1436
ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb
106 Taxonomy::ActiveRecord_Relation 1180
sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
5 ActiveRecord::Attribute::FromDatabase 1180
pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb
88 Arel::Nodes::Casted 1152
o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb
11 Arel::Nodes::SelectStatement 1088
or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb
9 Arel::Nodes::JoinSource 1088
root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb
7 Arel::Nodes::SelectCore 1088
/gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
864 Arel::SelectManager 1088
/opt/rh/rh-ruby22/root/usr/share/ruby/set.rb
81 Hash 992
hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb
26 Taxonomy::ActiveRecord_Relation 885
/gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
944 Arel::Nodes::And 880
/gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
10 Hostgroup::ActiveRecord_Relation 869
*rails objects-total all*
OBJECT COUNT
Hash 11318
Arel::Nodes::SqlLiteral 7566
Arel::Attributes::Attribute 7453
ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037
Array 6752
Taxonomy::ActiveRecord_Relation 4720
String 4651
ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
Arel::Nodes::As 3091
ActiveRecord::Relation::Merger 2589
Arel::Table 1670
Time 1597
Arel::Nodes::Equality 1436
Hostgroup::ActiveRecord_Relation 1283
ActiveRecord::Attribute::FromDatabase 1180
Arel::Nodes::Casted 1152
Arel::SelectManager 1088
Arel::Nodes::JoinSource 1088
Arel::Nodes::SelectCore 1088
Arel::Nodes::SelectStatement 1088
TaxableTaxonomy::ActiveRecord_Relation 1065
Arel::Nodes::And 1052
Set 992
VariableLookupKey::ActiveRecord_Relation 984
ActiveRecord::Result 949
On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal wrote:
>
> Were you running the "foreman-trace rails calls" utility when you
> searched the problematic "puppet" query? Because the output looks like
> sane output, I can see one hundred of NIC models allocated, but this
> should not take 6 Gbs at all.
>
> I've slightly enhanced the foreman-tracer, can you also try to run
> this while doing the problematic query:
>
> foreman-tracer rails objects-total all
>
> foreman-tracer rails objects all
>
> foreman-tracer rails calls all
>
> We must be allocating some number of objects somewhere, this should reveal
> them.
>
> LZ
>
> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]
> <javascript:>> wrote:
> > This is the problem, it's seems like running a query without any field
> is
> > causing this problem, it became more aggressive now probably because i
> am
> > running it under the "Classes" dashboard which now have much much more
> > objects than before.
> >
> > Running a query without a specific field: "puppet", memory usage from 1G
> to
> > 5G, a ruby instance takes 100% CPU for about a minute.
> > Running a query with a specific field: "name = puppet", memory usage
> remains
> > quite the same, small spike in CPU, returns immediately (less than a
> > second).
> >
> >
> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote:
> >>
> >> I have the "theforeman/puppet" module, when i search under the
> "Classes"
> >> dashboard the word "puppet" it's seems to cause a very high CPU and
> memory
> >> starts to increase up to 6GB from almost 1GB.
> >>
> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote:
> >>>
> >>> rails objects-total
> >>> OBJECT COUNT
> >>> ERB 826
> >>> Nic::Managed 104
> >>> Foreman::Parameters::Caster 38
> >>> IPAddr 24
> >>> ConfigReportStatusCalculator 24
> >>> ActiveSupport::HashWithIndifferentAccess 18
> >>> SafeRender 10
> >>> Range 9
> >>> ActionView::OutputBuffer 8
> >>> Classification::GlobalParam 6
> >>> ActionDispatch::Request 6
> >>> Foreman::Parameters::Validator 6
> >>> OpenSSL::ASN1::ASN1Data 6
> >>> OpenSSL::ASN1::Sequence 6
> >>> Authorizer 5
> >>> CertificateExtract 3
> >>> OpenSSL::ASN1::ObjectId 3
> >>> OpenSSL::ASN1::OctetString 3
> >>> OpenSSL::X509::Certificate 3
> >>> Classification::ClassParam 2
> >>> ActiveModel::Name 2
> >>> PuppetFactParser 1
> >>> HostStatus::Global 1
> >>> PuppetFactImporter 1
> >>>
> >>> rails objects
> >>> FILENAME
> >>> LINE OBJECT COUNT
> >>> /usr/share/foreman/app/services/safe_render.rb
> >>> 39 ERB 826
> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb
> >>> 10 Nic::Managed 96
> >>> /usr/share/foreman/app/services/classification/base.rb
> >>> 151 Foreman::Parameters::Caster 38
> >>> /usr/share/foreman/app/models/config_report.rb
> >>> 88 ConfigReportStatusCalculator 18
> >>> /usr/share/foreman/app/models/host/base.rb
> >>> 390 ActiveSupport::HashWithIndifferentAccess 11
> >>> /usr/share/foreman/app/models/concerns/encryptable.rb
> >>> 50 Range 8
> >>> /usr/share/foreman/app/models/host/base.rb
> >>> 78 Nic::Managed 8
> >>> /usr/share/foreman/lib/net/validations.rb
> >>> 31 IPAddr 8
> >>> /usr/share/foreman/lib/net/validations.rb
> >>> 18 IPAddr 8
> >>> /usr/share/foreman/app/services/classification/base.rb
> >>> 8 SafeRender 8
> >>> /usr/share/foreman/lib/net/validations.rb
> >>> 102 IPAddr 8
> >>> /usr/share/foreman/app/models/lookup_value.rb
> >>> 51 Foreman::Parameters::Validator 6
> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb
> >>> 10 ActionDispatch::Request 6
> >>> /usr/share/foreman/app/models/host/managed.rb
> >>> 934 Classification::GlobalParam 6
> >>> /usr/share/foreman/app/models/host_status/configuration_status.rb
> >>> 112 ConfigReportStatusCalculator 6
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 17 OpenSSL::ASN1::ASN1Data 6
> >>> /usr/share/foreman/app/helpers/application_helper.rb
> >>> 104 Authorizer 5
> >>> /usr/share/foreman/app/services/authorizer_cache.rb
> >>> 3 ActiveSupport::HashWithIndifferentAccess 5
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 5 OpenSSL::X509::Certificate 3
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 16 OpenSSL::ASN1::OctetString 3
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 16 OpenSSL::ASN1::Sequence 3
> >>>
> >>>
> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb
>
>
> >>> 59 CertificateExtract 3
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 16 OpenSSL::ASN1::ObjectId 3
> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >>> 17 OpenSSL::ASN1::Sequence 3
> >>> /usr/share/foreman/app/models/host/managed.rb
> >>> 420 SafeRender 2
> >>>
> >>> rails calls
> >>> FILENAME LINE
> >>> METHOD CALLS
> >>> /usr/share/foreman/app/models/setting.rb 218
> >>> cache 2120
> >>> /usr/share/foreman/app/models/setting.rb 72
> >>> [] 2086
> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55
> >>> exportable_attributes 2054
> >>> /usr/share/foreman/app/services/safe_render.rb 12
> >>> parse 1138
> >>> /usr/share/foreman/app/services/safe_render.rb 31
> >>> parse_string 826
> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33
> >>> export_attr 810
> >>> /usr/share/foreman/app/models/fact_value.rb 7
> >>> name 509
> >>> /usr/share/foreman/app/models/user.rb 137
> >>> admin? 180
> >>> /usr/share/foreman/app/models/taxonomy.rb 100
> >>> expand 180
> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92
> >>> to_s 112
> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9
> >>> setup_object_clone 104
> >>> /usr/share/foreman/app/models/nic/managed.rb 45
> >>> hostname 96
> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34
> >>> with_taxonomy_scope 84
> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115
> >>> scope_by_taxable_ids 84
> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75
> >>> taxable_ids 84
> >>> /usr/share/foreman/app/models/host/base.rb 441
> >>> get_interface_by_flag 74
> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35
> >>> call 66
> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12
> >>> dns_ready? 64
> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131
> >>> set_current_taxonomy 62
> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14
> >>> to_export 62
> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9
> >>> normalize_recurse 60
> >>> /usr/share/foreman/app/models/host/base.rb 237
> >>> primary_interface 54
> >>> /usr/share/foreman/app/services/classification/base.rb 167
> >>> update_generic_matcher 54
> >>> /usr/share/foreman/app/services/classification/base.rb 72
> >>> value_of_key 54
> >>> /usr/share/foreman/app/services/classification/base.rb 73
> >>> yield 44
> >>>
> >>>
> >>>
> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal
> wrote:
> >>>>
> >>>> Two or three minutes is just fine, counters do reset after 5 minutes
> >>>> so not longer.
> >>>>
> >>>> You can also add "rails calls" output as well, that might be
> interesting
> >>>> too.
> >>>>
> >>>> LZ
> >>>>
> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]>
> wrote:
> >>>> > No problem, for how long you think i should have it running to give
> >>>> > valuable
> >>>> > data to work with?
> >>>> > I am running only on objects-total and objects (as you mentioned)
> >>>> >
> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal
> wrote:
> >>>> >>
> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will
> only
> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your
> production
> >>>> >> instance? No changes required in Foreman, setup is quite easy:
> >>>> >>
> >>>> >> https://github.com/lzap/foreman-tracer
> >>>> >>
> >>>> >> Interesting statistics would be
> >>>> >>
> >>>> >> foreman-tracer rails objects-total
> >>>> >>
> >>>> >> and
> >>>> >>
> >>>> >> foreman-tracer rails objects
> >>>> >>
> >>>> >> It's like "top" utility experience, pastebin the bottlenecks
> please.
> >>>> >>
> >>>> >> Thanks
> >>>> >>
> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]>
> >>>> >> wrote:
> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from 1.12.1
> and
> >>>> >> > experience the same performance degradation on the foreman web
> app,
> >>>> >> > ruby
> >>>> >> > processes taking a lot of memory, no new plugins were
> introduced.
> >>>> >> > Same as what was written above, I suddenly see ruby instances
> using
> >>>> >> > of
> >>>> >> > up to
> >>>> >> > 4GB RAM/instance as of before i never saw this happens.
> >>>> >> >
> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote:
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <[email protected]>
>
> >>>> >> >> wrote:
> >>>> >> >>>
> >>>> >> >>> Hi all,
> >>>> >> >>>
> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and are
> >>>> >> >>> finding
> >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a
> lot
> >>>> >> >>> more
> >>>> >> >>> resources than the old version.
> >>>> >> >>>
> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them
> checking
> >>>> >> >>> in
> >>>> >> >>> with
> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs under
> >>>> >> >>> passenger
> >>>> >> >>> and
> >>>> >> >>> has a mysql database.
> >>>> >> >>>
> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling
> the
> >>>> >> >>> web
> >>>> >> >>> UI,
> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded to
> 4
> >>>> >> >>> x 8
> >>>> >> >>> vCPU,
> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1 x 8
> >>>> >> >>> vCPU,
> >>>> >> >>> 32GB
> >>>> >> >>> RAM Foreman server handling the web UI and database. We are
> >>>> >> >>> currently
> >>>> >> >>> restarting apache on the ENC/fact/report servers every 20
> minutes
> >>>> >> >>> to
> >>>> >> >>> stop
> >>>> >> >>> them running out of memory (if we tune passenger more
> >>>> >> >>> aggressively,
> >>>> >> >>> the
> >>>> >> >>> request queue fills up).
> >>>> >> >>>
> >>>> >> >>> I noticed a couple of other posts in the last few days
> regarding
> >>>> >> >>> performance. Is anyone running a similarly-sized installation
> on
> >>>> >> >>> version
> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips
> for
> >>>> >> >>> tuning
> >>>> >> >>> passenger or other components?
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> Are you using a new plugins? or just plain foreman? any
> >>>> >> >> information you
> >>>> >> >> could share that would provide some light on where things are
> >>>> >> >> slow?
> >>>> >> >> (e.g.
> >>>> >> >> slow operations, etc).
> >>>> >> >>
> >>>> >> >> thanks,
> >>>> >> >> Ohad
> >>>> >> >>>
> >>>> >> >>>
> >>>> >> >>> Regards
> >>>> >> >>> Guy.
> >>>> >> >>>
> >>>> >> >>> --
> >>>> >> >>> You received this message because you are subscribed to the
> >>>> >> >>> Google
> >>>> >> >>> Groups
> >>>> >> >>> "Foreman users" group.
> >>>> >> >>> To unsubscribe from this group and stop receiving emails from
> it,
> >>>> >> >>> send
> >>>> >> >>> an
> >>>> >> >>> email to [email protected].
> >>>> >> >>> To post to this group, send email to
> [email protected].
> >>>> >> >>> Visit this group at
> >>>> >> >>> https://groups.google.com/group/foreman-users.
> >>>> >> >>> For more options, visit https://groups.google.com/d/optout.
> >>>> >> >>
> >>>> >> >>
> >>>> >> > --
> >>>> >> > You received this message because you are subscribed to the
> Google
> >>>> >> > Groups
> >>>> >> > "Foreman users" group.
> >>>> >> > To unsubscribe from this group and stop receiving emails from
> it,
> >>>> >> > send
> >>>> >> > an
> >>>> >> > email to [email protected].
> >>>> >> > To post to this group, send email to [email protected].
>
> >>>> >> > Visit this group at
> https://groups.google.com/group/foreman-users.
> >>>> >> > For more options, visit https://groups.google.com/d/optout.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Later,
> >>>> >> Lukas @lzap Zapletal
> >>>> >
> >>>> > --
> >>>> > You received this message because you are subscribed to the Google
> >>>> > Groups
> >>>> > "Foreman users" group.
> >>>> > To unsubscribe from this group and stop receiving emails from it,
> send
> >>>> > an
> >>>> > email to [email protected].
> >>>> > To post to this group, send email to [email protected].
> >>>> > Visit this group at https://groups.google.com/group/foreman-users.
> >>>> > For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Later,
> >>>> Lukas @lzap Zapletal
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to [email protected] <javascript:>.
> > To post to this group, send email to [email protected]
> <javascript:>.
> > Visit this group at https://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
