It's not the terminal size issue, the format is set by the script, for this
tasks i increased it (Line 92,94,95,190,192)
*rails calls all*
FILENAME LINE
METHOD CALLS
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
260 hash 51976
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
160 nil? 48142
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
171 association_instance_get 48142
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
157 association 48142
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb
63 loaded! 47962
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
116 to_s 37385
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
115 _reflect_on_association 37385
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 name 37385
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb
86 _reflections 37385
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 options 37314
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
889 active_record 37292
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
686 through_reflection 37292
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
442 belongs_to? 37207
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
67 stale_state 37204
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
6 through_reflection 37204
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
592 collection? 37157
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb
72 primary_key 37059
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb
18 primary_key 36975
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
31 column_alias 36975
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
32 hash 36975
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
254 nil? 36974
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
39 [] 35363
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
9 [] 35363
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
35 value 35340
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
30 fetch_value 35317
*rails objects-total all*
OBJECT COUNT
Hash 8495
Arel::Nodes::SqlLiteral 7548
Arel::Attributes::Attribute 7401
ActiveRecord::ConnectionAdapters::PostgreSQL::Name 6981
Array 6478
Taxonomy::ActiveRecord_Relation 4720
String 4409
Arel::Nodes::As 3091
ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
ActiveRecord::Relation::Merger 2563
Arel::Table 1669
Time 1508
Arel::Nodes::Equality 1428
Hostgroup::ActiveRecord_Relation 1283
ActiveRecord::Attribute::FromDatabase 1146
Arel::Nodes::Casted 1136
Arel::Nodes::SelectStatement 1068
Arel::Nodes::JoinSource 1068
Arel::SelectManager 1068
Arel::Nodes::SelectCore 1068
TaxableTaxonomy::ActiveRecord_Relation 1065
Arel::Nodes::And 1032
VariableLookupKey::ActiveRecord_Relation 984
Set 960
ActiveRecord::Result 916
I couldn't understand what you mean about the interfaces, though i have
only one host with about 7 interfaces, i saw that post but it's seems not
relevant to me.
On Wednesday, November 2, 2016 at 5:40:54 PM UTC+2, Lukas Zapletal wrote:
>
> That's expected, the tracer is topped to prevent eating lot of memory,
> so it rather crash than consume more. I have slightly increased the
> limit, can you update the script and run once again, now for the last
> time:
>
> rails objects-total app 100
>
> and
>
> rails calls all 100
>
> Please set minimum font size for your terminal to fit 100 rows. I am
> just curious if we can spot the Model that is causing this.
>
> But I suspect this is the similar issue that someone else just had
> (see replies from today in the list). Hosts were created with too many
> NICs. The suggestion was: "See the 'ignore interfaces with matching
> identifier' option under Settings -> Provisioning." Then you need to
> delete all the NICs from the database to get things rolling again.
>
> LZ
>
> On Wed, Nov 2, 2016 at 2:50 PM, Erez Zarum <[email protected]
> <javascript:>> wrote:
> > Hey,
> > I think you nailed it :)
> >
> > rails calls all probe failed at some point though.
> >
> > rails calls all
> >
> > FILENAME LINE METHOD CALLS
> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 260 hash 86006
> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 171 association_instance_get 76602
> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 157 association 76602
> >
> /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
>
>
> > 160 nil? 76602
> >
> are/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/association.rb
>
>
> > 63 loaded! 76422
> >
> e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_ext/class/attribute.rb
>
>
> > 86 _reflections 60166
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 115 _reflect_on_association 60165
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 116 to_s 60164
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 name 60162
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 options 60086
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 686 through_reflection 60065
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 889 active_record 60065
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 442 belongs_to? 59980
> >
> /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> > 67 stale_state 59979
> >
> /gems/activerecord-4.2.5.1/lib/active_record/associations/through_association.rb
>
>
> > 6 through_reflection 59979
> >
> 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/reflection.rb
>
>
> > 592 collection? 59894
> >
> ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/primary_key.rb
>
>
> > 72 primary_key 59802
> >
> activerecord-4.2.5.1/lib/active_record/associations/join_dependency/join_part.rb
>
>
> > 18 primary_key 59712
> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 31 column_alias 59711
> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 254 nil? 59710
> >
> gems/gems/activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
>
>
> > 32 hash 59710
> >
> /share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
>
>
> > 39 [] 58136
> >
> root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> > 9 [] 58136
> >
> r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
>
>
> > 35 value 58078
> >
> root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
>
>
> > 30 fetch_value 58042
> > WARNING: Number of errors: 0, skipped probes: 101
> > WARNING: /usr/bin/staprun exited with status: 1
> > Pass 5: run failed. [man error::pass5]
> >
> > rails objects all
> > FILENAME
> > LINE OBJECT COUNT
> > cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb
> > 72 rd::ConnectionAdapters::PostgreSQL::Name 7037
> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb
> > 100 Arel::Attributes::Attribute 4317
> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb
> > 27 Arel::Nodes::SqlLiteral 4174
> > t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb
> > 110 String 3668
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb
> > 9 Arel::Attributes::Attribute 3095
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> > 4 Arel::Nodes::As 3091
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> > 4 Arel::Nodes::SqlLiteral 3091
> > activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
> > 126 iations::JoinDependency::Aliases::Column 3091
> > usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb
> > 35 Array 2753
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 10 Taxonomy::ActiveRecord_Relation 2655
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 44 ActiveRecord::Relation::Merger 2589
> > hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb
> > 84 Hash 1825
> > s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb
> > 63 Arel::Table 1666
> > clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb
> > 16 Arel::Nodes::Equality 1436
> > ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb
> > 106 Taxonomy::ActiveRecord_Relation 1180
> > sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
> > 5 ActiveRecord::Attribute::FromDatabase 1180
> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb
> > 88 Arel::Nodes::Casted 1152
> > o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb
> > 11 Arel::Nodes::SelectStatement 1088
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb
> > 9 Arel::Nodes::JoinSource 1088
> > root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb
> > 7 Arel::Nodes::SelectCore 1088
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> > 864 Arel::SelectManager 1088
> > /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb
> > 81 Hash 992
> > hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb
> > 26 Taxonomy::ActiveRecord_Relation 885
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> > 944 Arel::Nodes::And 880
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 10 Hostgroup::ActiveRecord_Relation 869
> >
> > rails objects-total all
> > OBJECT COUNT
> > Hash 11318
> > Arel::Nodes::SqlLiteral 7566
> > Arel::Attributes::Attribute 7453
> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037
> > Array 6752
> > Taxonomy::ActiveRecord_Relation 4720
> > String 4651
> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
> > Arel::Nodes::As 3091
> > ActiveRecord::Relation::Merger 2589
> > Arel::Table 1670
> > Time 1597
> > Arel::Nodes::Equality 1436
> > Hostgroup::ActiveRecord_Relation 1283
> > ActiveRecord::Attribute::FromDatabase 1180
> > Arel::Nodes::Casted 1152
> > Arel::SelectManager 1088
> > Arel::Nodes::JoinSource 1088
> > Arel::Nodes::SelectCore 1088
> > Arel::Nodes::SelectStatement 1088
> > TaxableTaxonomy::ActiveRecord_Relation 1065
> > Arel::Nodes::And 1052
> > Set 992
> > VariableLookupKey::ActiveRecord_Relation 984
> > ActiveRecord::Result 949
> >
> >
> >
> > On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal
> wrote:
> >>
> >> Were you running the "foreman-trace rails calls" utility when you
> >> searched the problematic "puppet" query? Because the output looks like
> >> sane output, I can see one hundred of NIC models allocated, but this
> >> should not take 6 Gbs at all.
> >>
> >> I've slightly enhanced the foreman-tracer, can you also try to run
> >> this while doing the problematic query:
> >>
> >> foreman-tracer rails objects-total all
> >>
> >> foreman-tracer rails objects all
> >>
> >> foreman-tracer rails calls all
> >>
> >> We must be allocating some number of objects somewhere, this should
> reveal
> >> them.
> >>
> >> LZ
> >>
> >> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]> wrote:
> >> > This is the problem, it's seems like running a query without any
> field
> >> > is
> >> > causing this problem, it became more aggressive now probably because
> i
> >> > am
> >> > running it under the "Classes" dashboard which now have much much
> more
> >> > objects than before.
> >> >
> >> > Running a query without a specific field: "puppet", memory usage from
> 1G
> >> > to
> >> > 5G, a ruby instance takes 100% CPU for about a minute.
> >> > Running a query with a specific field: "name = puppet", memory usage
> >> > remains
> >> > quite the same, small spike in CPU, returns immediately (less than a
> >> > second).
> >> >
> >> >
> >> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote:
> >> >>
> >> >> I have the "theforeman/puppet" module, when i search under the
> >> >> "Classes"
> >> >> dashboard the word "puppet" it's seems to cause a very high CPU and
> >> >> memory
> >> >> starts to increase up to 6GB from almost 1GB.
> >> >>
> >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote:
> >> >>>
> >> >>> rails objects-total
> >> >>> OBJECT COUNT
> >> >>> ERB 826
> >> >>> Nic::Managed 104
> >> >>> Foreman::Parameters::Caster 38
> >> >>> IPAddr 24
> >> >>> ConfigReportStatusCalculator 24
> >> >>> ActiveSupport::HashWithIndifferentAccess 18
> >> >>> SafeRender 10
> >> >>> Range 9
> >> >>> ActionView::OutputBuffer 8
> >> >>> Classification::GlobalParam 6
> >> >>> ActionDispatch::Request 6
> >> >>> Foreman::Parameters::Validator 6
> >> >>> OpenSSL::ASN1::ASN1Data 6
> >> >>> OpenSSL::ASN1::Sequence 6
> >> >>> Authorizer 5
> >> >>> CertificateExtract 3
> >> >>> OpenSSL::ASN1::ObjectId 3
> >> >>> OpenSSL::ASN1::OctetString 3
> >> >>> OpenSSL::X509::Certificate 3
> >> >>> Classification::ClassParam 2
> >> >>> ActiveModel::Name 2
> >> >>> PuppetFactParser 1
> >> >>> HostStatus::Global 1
> >> >>> PuppetFactImporter 1
> >> >>>
> >> >>> rails objects
> >> >>> FILENAME
> >> >>> LINE OBJECT COUNT
> >> >>> /usr/share/foreman/app/services/safe_render.rb
> >> >>> 39 ERB 826
> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb
> >> >>> 10 Nic::Managed 96
> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >>> 151 Foreman::Parameters::Caster 38
> >> >>> /usr/share/foreman/app/models/config_report.rb
> >> >>> 88 ConfigReportStatusCalculator 18
> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >>> 390 ActiveSupport::HashWithIndifferentAccess 11
> >> >>> /usr/share/foreman/app/models/concerns/encryptable.rb
> >> >>> 50 Range 8
> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >>> 78 Nic::Managed 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 31 IPAddr 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 18 IPAddr 8
> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >>> 8 SafeRender 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 102 IPAddr 8
> >> >>> /usr/share/foreman/app/models/lookup_value.rb
> >> >>> 51 Foreman::Parameters::Validator 6
> >> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb
> >> >>> 10 ActionDispatch::Request 6
> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >>> 934 Classification::GlobalParam 6
> >> >>> /usr/share/foreman/app/models/host_status/configuration_status.rb
> >> >>> 112 ConfigReportStatusCalculator 6
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 17 OpenSSL::ASN1::ASN1Data 6
> >> >>> /usr/share/foreman/app/helpers/application_helper.rb
> >> >>> 104 Authorizer 5
> >> >>> /usr/share/foreman/app/services/authorizer_cache.rb
> >> >>> 3 ActiveSupport::HashWithIndifferentAccess 5
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 5 OpenSSL::X509::Certificate 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::OctetString 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::Sequence 3
> >> >>>
> >> >>>
> >> >>>
> /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb
>
>
> >> >>> 59 CertificateExtract 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::ObjectId 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 17 OpenSSL::ASN1::Sequence 3
> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >>> 420 SafeRender 2
> >> >>>
> >> >>> rails calls
> >> >>> FILENAME LINE
> >> >>> METHOD CALLS
> >> >>> /usr/share/foreman/app/models/setting.rb 218
> >> >>> cache 2120
> >> >>> /usr/share/foreman/app/models/setting.rb 72
> >> >>> [] 2086
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55
> >> >>> exportable_attributes 2054
> >> >>> /usr/share/foreman/app/services/safe_render.rb 12
> >> >>> parse 1138
> >> >>> /usr/share/foreman/app/services/safe_render.rb 31
> >> >>> parse_string 826
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33
> >> >>> export_attr 810
> >> >>> /usr/share/foreman/app/models/fact_value.rb 7
> >> >>> name 509
> >> >>> /usr/share/foreman/app/models/user.rb 137
> >> >>> admin? 180
> >> >>> /usr/share/foreman/app/models/taxonomy.rb 100
> >> >>> expand 180
> >> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92
> >> >>> to_s 112
> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9
> >> >>> setup_object_clone 104
> >> >>> /usr/share/foreman/app/models/nic/managed.rb 45
> >> >>> hostname 96
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34
> >> >>> with_taxonomy_scope 84
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115
> >> >>> scope_by_taxable_ids 84
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75
> >> >>> taxable_ids 84
> >> >>> /usr/share/foreman/app/models/host/base.rb 441
> >> >>> get_interface_by_flag 74
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35
> >> >>> call 66
> >> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12
> >> >>> dns_ready? 64
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131
> >> >>> set_current_taxonomy 62
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14
> >> >>> to_export 62
> >> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9
> >> >>> normalize_recurse 60
> >> >>> /usr/share/foreman/app/models/host/base.rb 237
> >> >>> primary_interface 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 167
> >> >>> update_generic_matcher 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 72
> >> >>> value_of_key 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 73
> >> >>> yield 44
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal
> >> >>> wrote:
> >> >>>>
> >> >>>> Two or three minutes is just fine, counters do reset after 5
> minutes
> >> >>>> so not longer.
> >> >>>>
> >> >>>> You can also add "rails calls" output as well, that might be
> >> >>>> interesting
> >> >>>> too.
> >> >>>>
> >> >>>> LZ
> >> >>>>
> >> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]>
> >> >>>> wrote:
> >> >>>> > No problem, for how long you think i should have it running to
> give
> >> >>>> > valuable
> >> >>>> > data to work with?
> >> >>>> > I am running only on objects-total and objects (as you
> mentioned)
> >> >>>> >
> >> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal
> >> >>>> > wrote:
> >> >>>> >>
> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will
> >> >>>> >> only
> >> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your
> >> >>>> >> production
> >> >>>> >> instance? No changes required in Foreman, setup is quite easy:
> >> >>>> >>
> >> >>>> >> https://github.com/lzap/foreman-tracer
> >> >>>> >>
> >> >>>> >> Interesting statistics would be
> >> >>>> >>
> >> >>>> >> foreman-tracer rails objects-total
> >> >>>> >>
> >> >>>> >> and
> >> >>>> >>
> >> >>>> >> foreman-tracer rails objects
> >> >>>> >>
> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks
> >> >>>> >> please.
> >> >>>> >>
> >> >>>> >> Thanks
> >> >>>> >>
> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]>
>
> >> >>>> >> wrote:
> >> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from
> 1.12.1
> >> >>>> >> > and
> >> >>>> >> > experience the same performance degradation on the foreman
> web
> >> >>>> >> > app,
> >> >>>> >> > ruby
> >> >>>> >> > processes taking a lot of memory, no new plugins were
> >> >>>> >> > introduced.
> >> >>>> >> > Same as what was written above, I suddenly see ruby instances
> >> >>>> >> > using
> >> >>>> >> > of
> >> >>>> >> > up to
> >> >>>> >> > 4GB RAM/instance as of before i never saw this happens.
> >> >>>> >> >
> >> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote:
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <
> [email protected]>
> >> >>>> >> >> wrote:
> >> >>>> >> >>>
> >> >>>> >> >>> Hi all,
> >> >>>> >> >>>
> >> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and
> are
> >> >>>> >> >>> finding
> >> >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a
> >> >>>> >> >>> lot
> >> >>>> >> >>> more
> >> >>>> >> >>> resources than the old version.
> >> >>>> >> >>>
> >> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them
> >> >>>> >> >>> checking
> >> >>>> >> >>> in
> >> >>>> >> >>> with
> >> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs
> under
> >> >>>> >> >>> passenger
> >> >>>> >> >>> and
> >> >>>> >> >>> has a mysql database.
> >> >>>> >> >>>
> >> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server
> handling
> >> >>>> >> >>> the
> >> >>>> >> >>> web
> >> >>>> >> >>> UI,
> >> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded
> to
> >> >>>> >> >>> 4
> >> >>>> >> >>> x 8
> >> >>>> >> >>> vCPU,
> >> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1
> x 8
> >> >>>> >> >>> vCPU,
> >> >>>> >> >>> 32GB
> >> >>>> >> >>> RAM Foreman server handling the web UI and database. We are
> >> >>>> >> >>> currently
> >> >>>> >> >>> restarting apache on the ENC/fact/report servers every 20
> >> >>>> >> >>> minutes
> >> >>>> >> >>> to
> >> >>>> >> >>> stop
> >> >>>> >> >>> them running out of memory (if we tune passenger more
> >> >>>> >> >>> aggressively,
> >> >>>> >> >>> the
> >> >>>> >> >>> request queue fills up).
> >> >>>> >> >>>
> >> >>>> >> >>> I noticed a couple of other posts in the last few days
> >> >>>> >> >>> regarding
> >> >>>> >> >>> performance. Is anyone running a similarly-sized
> installation
> >> >>>> >> >>> on
> >> >>>> >> >>> version
> >> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any
> tips
> >> >>>> >> >>> for
> >> >>>> >> >>> tuning
> >> >>>> >> >>> passenger or other components?
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any
> >> >>>> >> >> information you
> >> >>>> >> >> could share that would provide some light on where things
> are
> >> >>>> >> >> slow?
> >> >>>> >> >> (e.g.
> >> >>>> >> >> slow operations, etc).
> >> >>>> >> >>
> >> >>>> >> >> thanks,
> >> >>>> >> >> Ohad
> >> >>>> >> >>>
> >> >>>> >> >>>
> >> >>>> >> >>> Regards
> >> >>>> >> >>> Guy.
> >> >>>> >> >>>
> >> >>>> >> >>> --
> >> >>>> >> >>> You received this message because you are subscribed to the
> >> >>>> >> >>> Google
> >> >>>> >> >>> Groups
> >> >>>> >> >>> "Foreman users" group.
> >> >>>> >> >>> To unsubscribe from this group and stop receiving emails
> from
> >> >>>> >> >>> it,
> >> >>>> >> >>> send
> >> >>>> >> >>> an
> >> >>>> >> >>> email to [email protected].
> >> >>>> >> >>> To post to this group, send email to
> >> >>>> >> >>> [email protected].
> >> >>>> >> >>> Visit this group at
> >> >>>> >> >>> https://groups.google.com/group/foreman-users.
> >> >>>> >> >>> For more options, visit https://groups.google.com/d/optout.
>
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> > --
> >> >>>> >> > You received this message because you are subscribed to the
> >> >>>> >> > Google
> >> >>>> >> > Groups
> >> >>>> >> > "Foreman users" group.
> >> >>>> >> > To unsubscribe from this group and stop receiving emails from
> >> >>>> >> > it,
> >> >>>> >> > send
> >> >>>> >> > an
> >> >>>> >> > email to [email protected].
> >> >>>> >> > To post to this group, send email to
> [email protected].
> >> >>>> >> > Visit this group at
> >> >>>> >> > https://groups.google.com/group/foreman-users.
> >> >>>> >> > For more options, visit https://groups.google.com/d/optout.
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> --
> >> >>>> >> Later,
> >> >>>> >> Lukas @lzap Zapletal
> >> >>>> >
> >> >>>> > --
> >> >>>> > You received this message because you are subscribed to the
> Google
> >> >>>> > Groups
> >> >>>> > "Foreman users" group.
> >> >>>> > To unsubscribe from this group and stop receiving emails from
> it,
> >> >>>> > send
> >> >>>> > an
> >> >>>> > email to [email protected].
> >> >>>> > To post to this group, send email to [email protected].
>
> >> >>>> > Visit this group at
> https://groups.google.com/group/foreman-users.
> >> >>>> > For more options, visit https://groups.google.com/d/optout.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Later,
> >> >>>> Lukas @lzap Zapletal
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Foreman users" group.
> >> > To unsubscribe from this group and stop receiving emails from it,
> send
> >> > an
> >> > email to [email protected].
> >> > To post to this group, send email to [email protected].
> >> > Visit this group at https://groups.google.com/group/foreman-users.
> >> > For more options, visit https://groups.google.com/d/optout.
> >>
> >>
> >>
> >> --
> >> Later,
> >> Lukas @lzap Zapletal
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to [email protected] <javascript:>.
> > To post to this group, send email to [email protected]
> <javascript:>.
> > Visit this group at https://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
