Sorry for the delay in replying. This is our Foreman server setup:
The four ENC/fact/report servers are CentOS 7.2 running
kernel 3.10.0-327.el7.x86_64.
The following ruby*, tfm* and foreman* packages are installed:
foreman-1.12.3-1.el7.noarch
foreman-cli-1.12.3-1.el7.noarch
foreman-compute-1.12.3-1.el7.noarch
foreman-console-1.12.3-1.el7.noarch
foreman-debug-1.12.3-1.el7.noarch
foreman-ec2-1.12.3-1.el7.noarch
foreman-installer-1.12.3-1.el7.noarch
foreman-libvirt-1.12.3-1.el7.noarch
foreman-mysql2-1.12.3-1.el7.noarch
foreman-openstack-1.12.3-1.el7.noarch
foreman-ovirt-1.12.3-1.el7.noarch
foreman-proxy-1.12.3-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
foreman-selinux-1.12.3-1.el7.noarch
foreman-sqlite-1.12.3-1.el7.noarch
foreman-vmware-1.12.3-1.el7.noarch
libselinux-ruby-2.2.2-6.el7.x86_64
rh-ruby22-ruby-2.2.2-12.el7.x86_64
rh-ruby22-rubygem-bigdecimal-1.2.6-12.el7.x86_64
rh-ruby22-rubygem-bundler-1.7.8-2.sc1.el7.noarch
rh-ruby22-rubygem-io-console-0.4.3-12.el7.x86_64
rh-ruby22-rubygem-json-1.8.1-12.el7.x86_64
rh-ruby22-rubygem-minitest-5.4.3-12.el7.noarch
rh-ruby22-rubygem-net-http-persistent-2.9.4-4.el7.noarch
rh-ruby22-rubygem-psych-2.0.8-12.el7.x86_64
rh-ruby22-rubygem-rake-10.4.2-12.el7.noarch
rh-ruby22-rubygem-rdoc-4.2.0-12.el7.noarch
rh-ruby22-rubygems-2.4.5-12.el7.x86_64
rh-ruby22-rubygem-thor-0.19.1-1.el7.noarch
rh-ruby22-ruby-irb-2.2.2-12.el7.noarch
rh-ruby22-ruby-libs-2.2.2-12.el7.x86_64
rh-ruby22-runtime-2.0-6.sc1.el7.x86_64
ruby-2.0.0.598-25.el7_1.x86_64
ruby-augeas-0.5.0-1.el7.x86_64
rubygem-abrt-0.0.6-2.el7.noarch
rubygem-ansi-1.4.3-2.el7.noarch
rubygem-bigdecimal-1.2.0-25.el7_1.x86_64
rubygem-bundler-1.7.8-3.el7.noarch
rubygem-bundler_ext-0.3.0-9.el7.noarch
rubygem-clamp-1.0.0-5.el7.noarch
rubygem-daemon_controller-1.1.2-2.el7.noarch
rubygem-ffi-1.9.10-2.el7.x86_64
rubygem-gssapi-1.1.2-3.el7.noarch
rubygem-hashie-2.0.5-4.el7.noarch
rubygem-highline-1.6.21-5.el7.noarch
rubygem-io-console-0.4.2-25.el7_1.x86_64
rubygem-json-1.7.7-25.el7_1.x86_64
rubygem-kafo-0.9.2-1.el7.noarch
rubygem-kafo_parsers-0.1.2-1.el7.noarch
rubygem-kafo_wizards-0.0.1-2.el7.noarch
rubygem-little-plugger-1.1.3-21.el7.noarch
rubygem-logging-1.8.2-4.el7.noarch
rubygem-multi_json-1.10.1-3.el7.noarch
rubygem-net-http-persistent-2.8-5.el7.noarch
rubygem-passenger-4.0.18-9.11.el7.x86_64
rubygem-passenger-native-4.0.18-9.11.el7.x86_64
rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
rubygem-powerbar-1.0.17-1.el7.noarch
rubygem-psych-2.0.0-25.el7_1.x86_64
rubygem-rack-1.6.4-2.el7.noarch
rubygem-rack-protection-1.5.3-3.el7.noarch
rubygem-rake-0.9.6-25.el7_1.noarch
rubygem-rdoc-4.0.0-25.el7_1.noarch
rubygem-rkerberos-0.1.3-5.el7.x86_64
rubygem-rubyipmi-0.10.0-2.el7.noarch
rubygems-2.0.14-25.el7_1.noarch
rubygem-sinatra-1.3.5-1.el7.noarch
rubygem-thor-0.19.1-1.el7.noarch
rubygem-tilt-1.3.7-2.el7.noarch
ruby-irb-2.0.0.598-25.el7_1.noarch
ruby-libs-2.0.0.598-25.el7_1.x86_64
ruby-shadow-2.2.0-2.el7.x86_64
sclo-ror42-rubygem-actionmailer-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-actionpack-4.2.5.1-2.el7.noarch
sclo-ror42-rubygem-actionview-4.2.5.1-3.el7.noarch
sclo-ror42-rubygem-activejob-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-activemodel-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-activerecord-4.2.5.1-4.el7.noarch
sclo-ror42-rubygem-activesupport-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-arel-6.0.3-2.el7.noarch
sclo-ror42-rubygem-builder-3.2.2-2.el7.noarch
sclo-ror42-rubygem-coffee-rails-4.1.0-3.el7.noarch
sclo-ror42-rubygem-coffee-script-2.4.1-1.el7.noarch
sclo-ror42-rubygem-coffee-script-source-1.10.0-1.el7.noarch
sclo-ror42-rubygem-concurrent-ruby-1.0.0-2.el7.noarch
sclo-ror42-rubygem-dalli-2.7.4-3.el7.noarch
sclo-ror42-rubygem-erubis-2.7.0-9.el7.noarch
sclo-ror42-rubygem-execjs-2.2.0-2.el7.noarch
sclo-ror42-rubygem-globalid-0.3.3-3.el7.noarch
sclo-ror42-rubygem-i18n-0.7.0-3.el7.noarch
sclo-ror42-rubygem-loofah-2.0.2-3.el7.noarch
sclo-ror42-rubygem-mail-2.6.1-1.el7.noarch
sclo-ror42-rubygem-mime-types-1.25.1-2.el7.noarch
sclo-ror42-rubygem-multi_json-1.10.1-3.el7.noarch
sclo-ror42-rubygem-nokogiri-1.6.7-0.4.rc3.el7.x86_64
sclo-ror42-rubygem-polyglot-0.3.4-2.el7.noarch
sclo-ror42-rubygem-rack-1.6.2-3.el7.noarch
sclo-ror42-rubygem-rack-test-0.6.3-3.el7.noarch
sclo-ror42-rubygem-rails-4.2.5.1-1.el7.noarch
sclo-ror42-rubygem-rails-deprecated_sanitizer-1.0.3-5.el7.noarch
sclo-ror42-rubygem-rails-dom-testing-1.0.5-3.el7.noarch
sclo-ror42-rubygem-rails-html-sanitizer-1.0.3-1.el7.noarch
sclo-ror42-rubygem-railties-4.2.5.1-2.el7.noarch
sclo-ror42-rubygem-sprockets-3.5.2-1.el7.noarch
sclo-ror42-rubygem-sprockets-rails-2.3.3-1.el7.noarch
sclo-ror42-rubygem-sqlite3-1.3.10-2.el7.x86_64
sclo-ror42-rubygem-thread_safe-0.3.5-2.el7.noarch
sclo-ror42-rubygem-turbolinks-2.5.3-2.el7.noarch
sclo-ror42-rubygem-tzinfo-1.2.2-2.el7.noarch
tfm-mod_passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-activerecord-session_store-0.1.2-2.el7.noarch
tfm-rubygem-ancestry-2.0.0-5.el7.noarch
tfm-rubygem-apipie-bindings-0.0.16-2.el7.noarch
tfm-rubygem-apipie-rails-0.3.6-1.el7.noarch
tfm-rubygem-audited-4.2.0-1.el7.noarch
tfm-rubygem-audited-activerecord-4.2.0-2.el7.noarch
tfm-rubygem-awesome_print-1.0.2-13.el7.noarch
tfm-rubygem-bundler_ext-0.3.0-9.el7.noarch
tfm-rubygem-clamp-1.0.0-5.el7.noarch
tfm-rubygem-colorize-0.7.7-4.el7.noarch
tfm-rubygem-deep_cloneable-2.1.1-4.el7.noarch
tfm-rubygem-deface-1.0.2-3.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-1.el7.noarch
tfm-rubygem-excon-0.49.0-3.el7.noarch
tfm-rubygem-facter-2.4.0-3.el7.x86_64
tfm-rubygem-fast_gettext-1.1.0-1.el7.noarch
tfm-rubygem-fog-1.38.0-1.el7.noarch
tfm-rubygem-fog-aws-0.9.1-1.el7.noarch
tfm-rubygem-fog-core-1.36.0-2.el7.noarch
tfm-rubygem-fog-google-0.1.0-2.el7.noarch
tfm-rubygem-fog-json-1.0.2-4.el7.noarch
tfm-rubygem-fog-libvirt-0.2.0-1.el7.noarch
tfm-rubygem-fog-openstack-0.1.2-1.el7.noarch
tfm-rubygem-fog-rackspace-0.1.1-1.el7.noarch
tfm-rubygem-fog-vsphere-0.7.0-1.el7.noarch
tfm-rubygem-fog-xenserver-0.2.3-1.el7.noarch
tfm-rubygem-fog-xml-0.1.2-4.el7.noarch
tfm-rubygem-foreman_discovery-6.0.0-1.fm1_12.el7.noarch
tfm-rubygem-foreman_memcache-0.0.3-4.fm1_12.el7.noarch
tfm-rubygem-formatador-0.2.1-9.el7.noarch
tfm-rubygem-friendly_id-5.1.0-3.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.2.1-3.el7.noarch
tfm-rubygem-hammer_cli-0.7.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.7.0-1.el7.noarch
tfm-rubygem-highline-1.6.21-5.el7.noarch
tfm-rubygem-hirb-0.7.0-9.el7.noarch
tfm-rubygem-hirb-unicode-steakknife-0.0.7-1.el7.noarch
tfm-rubygem-http-cookie-1.0.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-9.el7.noarch
tfm-rubygem-ldap_fluff-0.4.3-1.el7.noarch
tfm-rubygem-little-plugger-1.1.3-21.el7.noarch
tfm-rubygem-locale-2.0.9-11.el7.noarch
tfm-rubygem-logging-1.8.2-4.el7.noarch
tfm-rubygem-mysql2-0.3.19-5.el7.x86_64
tfm-rubygem-net-ldap-0.14.0-1.el7.noarch
tfm-rubygem-netrc-0.7.7-7.el7.noarch
tfm-rubygem-net-scp-1.1.0-6.el7.noarch
tfm-rubygem-net-ssh-3.0.2-2.el7.noarch
tfm-rubygem-oauth-0.4.7-8.el7.noarch
tfm-rubygem-paint-0.8.7-6.el7.noarch
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
tfm-rubygem-protected_attributes-1.1.3-3.el7.noarch
tfm-rubygem-rabl-0.12.0-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-5.el7.noarch
tfm-rubygem-rails-i18n-4.0.8-3.el7.noarch
tfm-rubygem-rails-observers-0.1.2-7.el7.noarch
tfm-rubygem-rbovirt-0.1.2-1.el7.noarch
tfm-rubygem-rbvmomi-1.8.2-4.el7.noarch
tfm-rubygem-responders-2.1.1-1.el7.noarch
tfm-rubygem-rest-client-1.8.0-1.el7.noarch
tfm-rubygem-ruby2ruby-2.1.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.5.2-4.el7.x86_64
tfm-rubygem-ruby_parser-3.6.3-4.el7.noarch
tfm-rubygem-safemode-1.2.4-2.el7.noarch
tfm-rubygem-scoped_search-3.2.2-4.el7.noarch
tfm-rubygem-secure_headers-1.4.1-4.el7.noarch
tfm-rubygem-sexp_processor-4.4.4-3.el7.noarch
tfm-rubygem-table_print-1.5.1-6.el7.noarch
tfm-rubygem-trollop-2.0-5.el7.noarch
tfm-rubygem-unf-0.1.3-5.el7.noarch
tfm-rubygem-unf_ext-0.0.6-6.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-1.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-3.el7.noarch
tfm-rubygem-will_paginate-3.1.0-2.el7.noarch
tfm-rubygem-wirb-1.0.3-3.el7.noarch
tfm-runtime-3.1-3.el7.x86_64
The Foreman manual, under section 3.3.2 ("Software Collections"), says to
install the following packages:
- mod_passenger
- tfm-rubygem-passenger
- tfm-rubygem-passenger-native
- tfm-rubygem-passenger-native-libs
- rubygem-passenger
- rubygem-passenger-native
- rubygem-passenger-native-libs
The 'mod_passenger' package (from the EPEL repo) obsoletes
'rubygem-passenger', 'rubygem-passenger-native' and
'rubygem-passenger-native-libs', so I don't have 'mod_passenger' package
installed -- instead, I have 'tfm-mod_passenger'.
This is the current passenger configuration in the apache config file:
PassengerAppRoot /usr/share/foreman
PassengerRuby /usr/bin/tfm-ruby
PassengerMaxPoolSize 4
PassengerMaxRequestQueueSize 100
PassengerMaxRequests 30
PassengerStartTimeout 600
PassengerMaxPreloaderIdleTime 0
PassengerStatThrottleRate 120
PassengerHighPerformance on
PassengerMaxInstancesPerApp 1
(The last line is there to run 'foreman-tracer'.)
With this passenger configuration, and the ~1250 managed host puppet runs
every hour load-balanced between the four Foreman servers, the Foreman
servers run out of memory after around an hour, unless apache is restarted
before that. The OOM-killer intervenes and kills ruby processes in that
instance -- they are consuming around 11-12GB of RAM when restarted.
Here is the output of 'foreman-tracer rails objects-total all', run for 2
minutes:
OBJECT COUNT
ActiveRecord::Attribute::FromDatabase 141593
String 81506
Hash 81245
ActiveSupport::HashWithIndifferentAccess 51192
Psych::Nodes::Scalar 31317
Psych::ClassLoader 21431
Psych::ScalarScanner 21431
Psych::Nodes::Stream 21430
Psych::Nodes::Document 21430
ActiveRecord::Attribute::FromUser 15835
Psych::Visitors::ToRuby 11153
Psych::Parser 11152
Psych::Handlers::DocumentStream 11152
ActiveSupport::TimeWithZone 10520
Psych::Visitors::YAMLTree 10278
Psych::Emitter 10278
Psych::TreeBuilder 10278
Psych::Visitors::YAMLTree::Registrar 10278
Psych::Visitors::Emitter 10278
StringIO 10277
ActiveRecord::AttributeSet 5238
ActiveRecord::LazyAttributeHash 5238
Nic::Managed 5220
ActiveRecord::Associations::BelongsToAssociation 4788
ActiveModel::Validations::LengthValidator 90
Here is the output of 'foreman-tracer rails objects all', run for 2 minutes:
FILENAME
LINE OBJECT COUNT
t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb
110 String 3093
sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
5 ActiveRecord::Attribute::FromDatabase 3044
/opt/rh/rh-ruby22/root/usr/share/ruby/psych/tree_builder.rb
75 Psych::Nodes::Scalar 2530
/gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
44 ActiveRecord::Relation::Merger 1473
pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb
100 Arel::Attributes::Attribute 1267
/share/gems/gems/activerecord-4.2.5.1/lib/active_record/type/string.rb
35 String 1197
re/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb
620 Time 995
usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb
35 Array 866
/opt/rh/rh-ruby22/root/usr/share/ruby/set.rb
81 Hash 834
s/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
18 ActiveRecord::LazyAttributeHash 793
s/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
19 ActiveRecord::AttributeSet 793
/share/gems/gems/activerecord-4.2.5.1/lib/active_record/type/string.rb
17 String 621
re/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/merger.rb
163 Hash 575
sr/share/gems/gems/dalli-2.7.4/lib/active_support/cache/dalli_store.rb
348 String 549
vesupport-4.2.5.1/lib/active_support/core_ext/hash/transform_values.rb
11 ActiveRecord::Attribute::FromDatabase 498
activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb
52 Hash 429
share/gems/gems/activesupport-4.2.5.1/lib/active_support/subscriber.rb
86 ActiveSupport::Notifications::Event 429
share/gems/gems/activesupport-4.2.5.1/lib/active_support/subscriber.rb
86 Time 429
share/gems/gems/activesupport-4.2.5.1/lib/active_support/subscriber.rb
94 Time 427
share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations.rb
162 cord::Associations::BelongsToAssociation 422
pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb
88 Arel::Nodes::Casted 422
re/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/merger.rb
171 Array 413
re/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/merger.rb
170 Set 413
re/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/merger.rb
185 Set 413
tivesupport-4.2.5.1/lib/active_support/hash_with_indifferent_access.rb
77 ActiveSupport::HashWithIndifferentAccess 409
Here is the output of 'foreman-tracer rails calls all', run for about 40
seconds:
FILENAME LINE METHOD CALLS
/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set/builder.rb
39 [] 11697
root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_set.rb
9 [] 11220
r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
35 value 7056
ot/usr/share/gems/gems/activerecord-4.2.5.1/lib/
active_record/relation/merger.rb 68 nil?
5928
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/request.rb
22 path_info 5250
/opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/request.rb
22 to_s 5249
r/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/path/pattern.rb
164 match 5227
r/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/path/pattern.rb
165 match 5227
r/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/path/pattern.rb
174 to_regexp 5227
hare/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb
165 original_raw_attributes 4893
r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
53 changed_in_place_from? 4203
r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
98 has_been_read? 4203
hare/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb
160 fetch 4203
hare/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb
159 original_raw_attribute 4203
hare/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb
73 attribute_changed_in_place? 4203
share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
95 check_cached_relation 4099
/share/gems/gems/activesupport-4.2.5.1/lib/active_support/per_thread_registry.rb
40 [] 3956
/share/gems/gems/activesupport-4.2.5.1/lib/active_support/per_thread_registry.rb
40 current 3956
/share/gems/gems/activesupport-4.2.5.1/lib/active_support/per_thread_registry.rb
39 instance 3956
hare/gems/gems/thread_safe-0.3.5/lib/thread_safe/non_concurrent_cache_backend.rb
11 [] 2889
ems/activerecord-4.2.5.1/lib/active_record/attribute_methods/before_type_cast.rb
47 to_s 2751
ems/activerecord-4.2.5.1/lib/active_record/attribute_methods/before_type_cast.rb
46 ead_attribute_before_type_cast 2751
h/sclo-ror42/root/usr/share/gems/gems/thread_safe-0.3.5/lib/thread_safe/cache.rb
37 [] 2702
ms/gems/activesupport-4.2.5.1/lib/active_support/hash_with_indifferent_access.rb
261 kind_of? 2420
ms/gems/activesupport-4.2.5.1/lib/active_support/hash_with_indifferent_access.rb
260 convert_key 2420
I tried to set the terminal columns to 100 with 'stty', but it doesn't make
any difference to the output of 'foreman-tracer'. How should I be doing
that?
Thanks
Guy.
On 3 November 2016 at 02:40, Lukas Zapletal <[email protected]> wrote:
> That's expected, the tracer is topped to prevent eating lot of memory,
> so it rather crash than consume more. I have slightly increased the
> limit, can you update the script and run once again, now for the last
> time:
>
> rails objects-total app 100
>
> and
>
> rails calls all 100
>
> Please set minimum font size for your terminal to fit 100 rows. I am
> just curious if we can spot the Model that is causing this.
>
> But I suspect this is the similar issue that someone else just had
> (see replies from today in the list). Hosts were created with too many
> NICs. The suggestion was: "See the 'ignore interfaces with matching
> identifier' option under Settings -> Provisioning." Then you need to
> delete all the NICs from the database to get things rolling again.
>
> LZ
>
> On Wed, Nov 2, 2016 at 2:50 PM, Erez Zarum <[email protected]> wrote:
> > Hey,
> > I think you nailed it :)
> >
> > rails calls all probe failed at some point though.
> >
> > rails calls all
> >
> > FILENAME LINE METHOD CALLS
> > gems/gems/activerecord-4.2.5.1/lib/active_record/
> associations/join_dependency.rb
> > 260 hash 86006
> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/associations.rb
> > 171 association_instance_get 76602
> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/associations.rb
> > 157 association 76602
> > /root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/associations.rb
> > 160 nil? 76602
> > are/gems/gems/activerecord-4.2.5.1/lib/active_record/
> associations/association.rb
> > 63 loaded! 76422
> > e/gems/gems/activesupport-4.2.5.1/lib/active_support/core_
> ext/class/attribute.rb
> > 86 _reflections 60166
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 115 _reflect_on_association 60165
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 116 to_s 60164
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 889 name 60162
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 889 options 60086
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 686 through_reflection 60065
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 889 active_record 60065
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 442 belongs_to? 59980
> > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_
> association.rb
> > 67 stale_state 59979
> > /gems/activerecord-4.2.5.1/lib/active_record/associations/through_
> association.rb
> > 6 through_reflection 59979
> > 42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/reflection.rb
> > 592 collection? 59894
> > ems/gems/activerecord-4.2.5.1/lib/active_record/attribute_
> methods/primary_key.rb
> > 72 primary_key 59802
> > activerecord-4.2.5.1/lib/active_record/associations/
> join_dependency/join_part.rb
> > 18 primary_key 59712
> > gems/gems/activerecord-4.2.5.1/lib/active_record/
> associations/join_dependency.rb
> > 31 column_alias 59711
> > gems/gems/activerecord-4.2.5.1/lib/active_record/
> associations/join_dependency.rb
> > 254 nil? 59710
> > gems/gems/activerecord-4.2.5.1/lib/active_record/
> associations/join_dependency.rb
> > 32 hash 59710
> > /share/gems/gems/activerecord-4.2.5.1/lib/active_record/
> attribute_set/builder.rb
> > 39 [] 58136
> > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/attribute_set.rb
> > 9 [] 58136
> > r42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/attribute.rb
> > 35 value 58078
> > root/usr/share/gems/gems/activerecord-4.2.5.1/lib/
> active_record/attribute_set.rb
> > 30 fetch_value 58042
> > WARNING: Number of errors: 0, skipped probes: 101
> > WARNING: /usr/bin/staprun exited with status: 1
> > Pass 5: run failed. [man error::pass5]
> >
> > rails objects all
> > FILENAME
> > LINE OBJECT COUNT
> > cord-4.2.5.1/lib/active_record/connection_adapters/postgresql/utils.rb
> > 72 rd::ConnectionAdapters::PostgreSQL::Name 7037
> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/table.rb
> > 100 Arel::Attributes::Attribute 4317
> > /opt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel.rb
> > 27 Arel::Nodes::SqlLiteral 4174
> > t/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/result.rb
> > 110 String 3668
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/table_alias.rb
> > 9 Arel::Attributes::Attribute 3095
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> > 4 Arel::Nodes::As 3091
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/alias_predication.rb
> > 4 Arel::Nodes::SqlLiteral 3091
> > activerecord-4.2.5.1/lib/active_record/associations/join_dependency.rb
> > 126 iations::JoinDependency::Aliases::Column 3091
> > usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation.rb
> > 35 Array 2753
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 10 Taxonomy::ActiveRecord_Relation 2655
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 44 ActiveRecord::Relation::Merger 2589
> > hare/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/route.rb
> > 84 Hash 1825
> > s/activerecord-4.2.5.1/lib/active_record/associations/alias_tracker.rb
> > 63 Arel::Table 1666
> > clo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/predications.rb
> > 16 Arel::Nodes::Equality 1436
> > ems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb
> > 106 Taxonomy::ActiveRecord_Relation 1180
> > sr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute.rb
> > 5 ActiveRecord::Attribute::FromDatabase 1180
> > pt/rh/sclo-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes.rb
> > 88 Arel::Nodes::Casted 1152
> > o-ror42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/select_manager.rb
> > 11 Arel::Nodes::SelectStatement 1088
> > or42/root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_core.rb
> > 9 Arel::Nodes::JoinSource 1088
> > root/usr/share/gems/gems/arel-6.0.3/lib/arel/nodes/select_statement.rb
> > 7 Arel::Nodes::SelectCore 1088
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> > 864 Arel::SelectManager 1088
> > /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb
> > 81 Hash 992
> > hare/gems/gems/activerecord-4.2.5.1/lib/active_record/scoping/named.rb
> > 26 Taxonomy::ActiveRecord_Relation 885
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/query_methods.rb
> > 944 Arel::Nodes::And 880
> > /gems/activerecord-4.2.5.1/lib/active_record/relation/spawn_methods.rb
> > 10 Hostgroup::ActiveRecord_Relation 869
> >
> > rails objects-total all
> > OBJECT COUNT
> > Hash 11318
> > Arel::Nodes::SqlLiteral 7566
> > Arel::Attributes::Attribute 7453
> > ActiveRecord::ConnectionAdapters::PostgreSQL::Name 7037
> > Array 6752
> > Taxonomy::ActiveRecord_Relation 4720
> > String 4651
> > ActiveRecord::Associations::JoinDependency::Aliases::Column 3091
> > Arel::Nodes::As 3091
> > ActiveRecord::Relation::Merger 2589
> > Arel::Table 1670
> > Time 1597
> > Arel::Nodes::Equality 1436
> > Hostgroup::ActiveRecord_Relation 1283
> > ActiveRecord::Attribute::FromDatabase 1180
> > Arel::Nodes::Casted 1152
> > Arel::SelectManager 1088
> > Arel::Nodes::JoinSource 1088
> > Arel::Nodes::SelectCore 1088
> > Arel::Nodes::SelectStatement 1088
> > TaxableTaxonomy::ActiveRecord_Relation 1065
> > Arel::Nodes::And 1052
> > Set 992
> > VariableLookupKey::ActiveRecord_Relation 984
> > ActiveRecord::Result 949
> >
> >
> >
> > On Wednesday, November 2, 2016 at 3:17:09 PM UTC+2, Lukas Zapletal wrote:
> >>
> >> Were you running the "foreman-trace rails calls" utility when you
> >> searched the problematic "puppet" query? Because the output looks like
> >> sane output, I can see one hundred of NIC models allocated, but this
> >> should not take 6 Gbs at all.
> >>
> >> I've slightly enhanced the foreman-tracer, can you also try to run
> >> this while doing the problematic query:
> >>
> >> foreman-tracer rails objects-total all
> >>
> >> foreman-tracer rails objects all
> >>
> >> foreman-tracer rails calls all
> >>
> >> We must be allocating some number of objects somewhere, this should
> reveal
> >> them.
> >>
> >> LZ
> >>
> >> On Wed, Nov 2, 2016 at 11:55 AM, Erez Zarum <[email protected]> wrote:
> >> > This is the problem, it's seems like running a query without any field
> >> > is
> >> > causing this problem, it became more aggressive now probably because i
> >> > am
> >> > running it under the "Classes" dashboard which now have much much more
> >> > objects than before.
> >> >
> >> > Running a query without a specific field: "puppet", memory usage from
> 1G
> >> > to
> >> > 5G, a ruby instance takes 100% CPU for about a minute.
> >> > Running a query with a specific field: "name = puppet", memory usage
> >> > remains
> >> > quite the same, small spike in CPU, returns immediately (less than a
> >> > second).
> >> >
> >> >
> >> > On Tuesday, November 1, 2016 at 6:22:01 PM UTC+2, Erez Zarum wrote:
> >> >>
> >> >> I have the "theforeman/puppet" module, when i search under the
> >> >> "Classes"
> >> >> dashboard the word "puppet" it's seems to cause a very high CPU and
> >> >> memory
> >> >> starts to increase up to 6GB from almost 1GB.
> >> >>
> >> >> On Tuesday, November 1, 2016 at 5:19:36 PM UTC+2, Erez Zarum wrote:
> >> >>>
> >> >>> rails objects-total
> >> >>> OBJECT COUNT
> >> >>> ERB 826
> >> >>> Nic::Managed 104
> >> >>> Foreman::Parameters::Caster 38
> >> >>> IPAddr 24
> >> >>> ConfigReportStatusCalculator 24
> >> >>> ActiveSupport::HashWithIndifferentAccess 18
> >> >>> SafeRender 10
> >> >>> Range 9
> >> >>> ActionView::OutputBuffer 8
> >> >>> Classification::GlobalParam 6
> >> >>> ActionDispatch::Request 6
> >> >>> Foreman::Parameters::Validator 6
> >> >>> OpenSSL::ASN1::ASN1Data 6
> >> >>> OpenSSL::ASN1::Sequence 6
> >> >>> Authorizer 5
> >> >>> CertificateExtract 3
> >> >>> OpenSSL::ASN1::ObjectId 3
> >> >>> OpenSSL::ASN1::OctetString 3
> >> >>> OpenSSL::X509::Certificate 3
> >> >>> Classification::ClassParam 2
> >> >>> ActiveModel::Name 2
> >> >>> PuppetFactParser 1
> >> >>> HostStatus::Global 1
> >> >>> PuppetFactImporter 1
> >> >>>
> >> >>> rails objects
> >> >>> FILENAME
> >> >>> LINE OBJECT COUNT
> >> >>> /usr/share/foreman/app/services/safe_render.rb
> >> >>> 39 ERB 826
> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb
> >> >>> 10 Nic::Managed 96
> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >>> 151 Foreman::Parameters::Caster 38
> >> >>> /usr/share/foreman/app/models/config_report.rb
> >> >>> 88 ConfigReportStatusCalculator 18
> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >>> 390 ActiveSupport::HashWithIndifferentAccess 11
> >> >>> /usr/share/foreman/app/models/concerns/encryptable.rb
> >> >>> 50 Range 8
> >> >>> /usr/share/foreman/app/models/host/base.rb
> >> >>> 78 Nic::Managed 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 31 IPAddr 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 18 IPAddr 8
> >> >>> /usr/share/foreman/app/services/classification/base.rb
> >> >>> 8 SafeRender 8
> >> >>> /usr/share/foreman/lib/net/validations.rb
> >> >>> 102 IPAddr 8
> >> >>> /usr/share/foreman/app/models/lookup_value.rb
> >> >>> 51 Foreman::Parameters::Validator 6
> >> >>> /usr/share/foreman/lib/middleware/tagged_logging.rb
> >> >>> 10 ActionDispatch::Request 6
> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >>> 934 Classification::GlobalParam 6
> >> >>> /usr/share/foreman/app/models/host_status/configuration_status.rb
> >> >>> 112 ConfigReportStatusCalculator 6
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 17 OpenSSL::ASN1::ASN1Data 6
> >> >>> /usr/share/foreman/app/helpers/application_helper.rb
> >> >>> 104 Authorizer 5
> >> >>> /usr/share/foreman/app/services/authorizer_cache.rb
> >> >>> 3 ActiveSupport::HashWithIndifferentAccess 5
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 5 OpenSSL::X509::Certificate 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::OctetString 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::Sequence 3
> >> >>>
> >> >>>
> >> >>> /usr/share/foreman/app/controllers/concerns/foreman/
> controller/smart_proxy_auth.rb
> >> >>> 59 CertificateExtract 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 16 OpenSSL::ASN1::ObjectId 3
> >> >>> /usr/share/foreman/app/services/certificate_extract.rb
> >> >>> 17 OpenSSL::ASN1::Sequence 3
> >> >>> /usr/share/foreman/app/models/host/managed.rb
> >> >>> 420 SafeRender 2
> >> >>>
> >> >>> rails calls
> >> >>> FILENAME LINE
> >> >>> METHOD CALLS
> >> >>> /usr/share/foreman/app/models/setting.rb 218
> >> >>> cache 2120
> >> >>> /usr/share/foreman/app/models/setting.rb 72
> >> >>> [] 2086
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 55
> >> >>> exportable_attributes 2054
> >> >>> /usr/share/foreman/app/services/safe_render.rb 12
> >> >>> parse 1138
> >> >>> /usr/share/foreman/app/services/safe_render.rb 31
> >> >>> parse_string 826
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 33
> >> >>> export_attr 810
> >> >>> /usr/share/foreman/app/models/fact_value.rb 7
> >> >>> name 509
> >> >>> /usr/share/foreman/app/models/user.rb 137
> >> >>> admin? 180
> >> >>> /usr/share/foreman/app/models/taxonomy.rb 100
> >> >>> expand 180
> >> >>> /usr/share/foreman/app/models/lookup_keys/lookup_key.rb 92
> >> >>> to_s 112
> >> >>> /usr/share/foreman/app/models/concerns/interface_cloning.rb 9
> >> >>> setup_object_clone 104
> >> >>> /usr/share/foreman/app/models/nic/managed.rb 45
> >> >>> hostname 96
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 34
> >> >>> with_taxonomy_scope 84
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 115
> >> >>> scope_by_taxable_ids 84
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 75
> >> >>> taxable_ids 84
> >> >>> /usr/share/foreman/app/models/host/base.rb 441
> >> >>> get_interface_by_flag 74
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 35
> >> >>> call 66
> >> >>> /usr/share/foreman/app/models/concerns/orchestration/dns.rb 12
> >> >>> dns_ready? 64
> >> >>> /usr/share/foreman/app/models/concerns/taxonomix.rb 131
> >> >>> set_current_taxonomy 62
> >> >>> /usr/share/foreman/app/models/concerns/exportable.rb 14
> >> >>> to_export 62
> >> >>> /usr/share/foreman/app/services/structured_fact_importer.rb 9
> >> >>> normalize_recurse 60
> >> >>> /usr/share/foreman/app/models/host/base.rb 237
> >> >>> primary_interface 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 167
> >> >>> update_generic_matcher 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 72
> >> >>> value_of_key 54
> >> >>> /usr/share/foreman/app/services/classification/base.rb 73
> >> >>> yield 44
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Tuesday, November 1, 2016 at 4:25:31 PM UTC+2, Lukas Zapletal
> >> >>> wrote:
> >> >>>>
> >> >>>> Two or three minutes is just fine, counters do reset after 5
> minutes
> >> >>>> so not longer.
> >> >>>>
> >> >>>> You can also add "rails calls" output as well, that might be
> >> >>>> interesting
> >> >>>> too.
> >> >>>>
> >> >>>> LZ
> >> >>>>
> >> >>>> On Tue, Nov 1, 2016 at 11:58 AM, Erez Zarum <[email protected]>
> >> >>>> wrote:
> >> >>>> > No problem, for how long you think i should have it running to
> give
> >> >>>> > valuable
> >> >>>> > data to work with?
> >> >>>> > I am running only on objects-total and objects (as you mentioned)
> >> >>>> >
> >> >>>> > On Monday, October 31, 2016 at 7:05:28 PM UTC+2, Lukas Zapletal
> >> >>>> > wrote:
> >> >>>> >>
> >> >>>> >> Can you guys try foreman-tracer utility (SystemTap based, will
> >> >>>> >> only
> >> >>>> >> work on CentOS 7 or higher or ST-enabled kernels) on your
> >> >>>> >> production
> >> >>>> >> instance? No changes required in Foreman, setup is quite easy:
> >> >>>> >>
> >> >>>> >> https://github.com/lzap/foreman-tracer
> >> >>>> >>
> >> >>>> >> Interesting statistics would be
> >> >>>> >>
> >> >>>> >> foreman-tracer rails objects-total
> >> >>>> >>
> >> >>>> >> and
> >> >>>> >>
> >> >>>> >> foreman-tracer rails objects
> >> >>>> >>
> >> >>>> >> It's like "top" utility experience, pastebin the bottlenecks
> >> >>>> >> please.
> >> >>>> >>
> >> >>>> >> Thanks
> >> >>>> >>
> >> >>>> >> On Mon, Oct 31, 2016 at 12:00 PM, Erez Zarum <[email protected]
> >
> >> >>>> >> wrote:
> >> >>>> >> > I have just upgraded a foreman deployment to 1.13.1 from
> 1.12.1
> >> >>>> >> > and
> >> >>>> >> > experience the same performance degradation on the foreman web
> >> >>>> >> > app,
> >> >>>> >> > ruby
> >> >>>> >> > processes taking a lot of memory, no new plugins were
> >> >>>> >> > introduced.
> >> >>>> >> > Same as what was written above, I suddenly see ruby instances
> >> >>>> >> > using
> >> >>>> >> > of
> >> >>>> >> > up to
> >> >>>> >> > 4GB RAM/instance as of before i never saw this happens.
> >> >>>> >> >
> >> >>>> >> > On Monday, October 31, 2016 at 10:28:19 AM UTC+2, ohad wrote:
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >> On Mon, Oct 31, 2016 at 1:55 AM, Guy Waugh <
> [email protected]>
> >> >>>> >> >> wrote:
> >> >>>> >> >>>
> >> >>>> >> >>> Hi all,
> >> >>>> >> >>>
> >> >>>> >> >>> We have recently upgraded Foreman from 1.9.2 to 1.12.3 and
> are
> >> >>>> >> >>> finding
> >> >>>> >> >>> that Foreman's ENC/facts/reports components are requiring a
> >> >>>> >> >>> lot
> >> >>>> >> >>> more
> >> >>>> >> >>> resources than the old version.
> >> >>>> >> >>>
> >> >>>> >> >>> We have around 1,700 hosts in Foreman, with 600 of them
> >> >>>> >> >>> checking
> >> >>>> >> >>> in
> >> >>>> >> >>> with
> >> >>>> >> >>> external puppet masters every 30 minutes. Foreman runs under
> >> >>>> >> >>> passenger
> >> >>>> >> >>> and
> >> >>>> >> >>> has a mysql database.
> >> >>>> >> >>>
> >> >>>> >> >>> On 1.9.2, we had 1 x 4 vCPU, 8GB RAM Foreman server handling
> >> >>>> >> >>> the
> >> >>>> >> >>> web
> >> >>>> >> >>> UI,
> >> >>>> >> >>> database and ENC/facts/reports. On 1.12.3, we have expanded
> to
> >> >>>> >> >>> 4
> >> >>>> >> >>> x 8
> >> >>>> >> >>> vCPU,
> >> >>>> >> >>> 32GB RAM Foreman servers handling ENC/facts/reports, and 1
> x 8
> >> >>>> >> >>> vCPU,
> >> >>>> >> >>> 32GB
> >> >>>> >> >>> RAM Foreman server handling the web UI and database. We are
> >> >>>> >> >>> currently
> >> >>>> >> >>> restarting apache on the ENC/fact/report servers every 20
> >> >>>> >> >>> minutes
> >> >>>> >> >>> to
> >> >>>> >> >>> stop
> >> >>>> >> >>> them running out of memory (if we tune passenger more
> >> >>>> >> >>> aggressively,
> >> >>>> >> >>> the
> >> >>>> >> >>> request queue fills up).
> >> >>>> >> >>>
> >> >>>> >> >>> I noticed a couple of other posts in the last few days
> >> >>>> >> >>> regarding
> >> >>>> >> >>> performance. Is anyone running a similarly-sized
> installation
> >> >>>> >> >>> on
> >> >>>> >> >>> version
> >> >>>> >> >>> 1.12/1.13? If so, what performance are you getting? Any tips
> >> >>>> >> >>> for
> >> >>>> >> >>> tuning
> >> >>>> >> >>> passenger or other components?
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> >> Are you using a new plugins? or just plain foreman? any
> >> >>>> >> >> information you
> >> >>>> >> >> could share that would provide some light on where things are
> >> >>>> >> >> slow?
> >> >>>> >> >> (e.g.
> >> >>>> >> >> slow operations, etc).
> >> >>>> >> >>
> >> >>>> >> >> thanks,
> >> >>>> >> >> Ohad
> >> >>>> >> >>>
> >> >>>> >> >>>
> >> >>>> >> >>> Regards
> >> >>>> >> >>> Guy.
> >> >>>> >> >>>
> >> >>>> >> >>> --
> >> >>>> >> >>> You received this message because you are subscribed to the
> >> >>>> >> >>> Google
> >> >>>> >> >>> Groups
> >> >>>> >> >>> "Foreman users" group.
> >> >>>> >> >>> To unsubscribe from this group and stop receiving emails
> from
> >> >>>> >> >>> it,
> >> >>>> >> >>> send
> >> >>>> >> >>> an
> >> >>>> >> >>> email to [email protected].
> >> >>>> >> >>> To post to this group, send email to
> >> >>>> >> >>> [email protected].
> >> >>>> >> >>> Visit this group at
> >> >>>> >> >>> https://groups.google.com/group/foreman-users.
> >> >>>> >> >>> For more options, visit https://groups.google.com/d/optout.
> >> >>>> >> >>
> >> >>>> >> >>
> >> >>>> >> > --
> >> >>>> >> > You received this message because you are subscribed to the
> >> >>>> >> > Google
> >> >>>> >> > Groups
> >> >>>> >> > "Foreman users" group.
> >> >>>> >> > To unsubscribe from this group and stop receiving emails from
> >> >>>> >> > it,
> >> >>>> >> > send
> >> >>>> >> > an
> >> >>>> >> > email to [email protected].
> >> >>>> >> > To post to this group, send email to
> [email protected].
> >> >>>> >> > Visit this group at
> >> >>>> >> > https://groups.google.com/group/foreman-users.
> >> >>>> >> > For more options, visit https://groups.google.com/d/optout.
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> --
> >> >>>> >> Later,
> >> >>>> >> Lukas @lzap Zapletal
> >> >>>> >
> >> >>>> > --
> >> >>>> > You received this message because you are subscribed to the
> Google
> >> >>>> > Groups
> >> >>>> > "Foreman users" group.
> >> >>>> > To unsubscribe from this group and stop receiving emails from it,
> >> >>>> > send
> >> >>>> > an
> >> >>>> > email to [email protected].
> >> >>>> > To post to this group, send email to [email protected].
> >> >>>> > Visit this group at https://groups.google.com/
> group/foreman-users.
> >> >>>> > For more options, visit https://groups.google.com/d/optout.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Later,
> >> >>>> Lukas @lzap Zapletal
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Foreman users" group.
> >> > To unsubscribe from this group and stop receiving emails from it, send
> >> > an
> >> > email to [email protected].
> >> > To post to this group, send email to [email protected].
> >> > Visit this group at https://groups.google.com/group/foreman-users.
> >> > For more options, visit https://groups.google.com/d/optout.
> >>
> >>
> >>
> >> --
> >> Later,
> >> Lukas @lzap Zapletal
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Foreman users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at https://groups.google.com/group/foreman-users.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
>
> --
> You received this message because you are subscribed to the Google Groups
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
