Further, the capsule syncs seem to take swap space (even if RAM is/was available) and never gives it back - to the point where we were publishing content views and tomcat failed, in /var/log/foreman/production.log start getting...
2016-08-04 14:20:15 [app] [I] Imported report for niwa-35649.niwa.local in 0.12 seconds 2016-08-04 14:20:15 [app] [W] Action failed | Katello::Errors::ConnectionRefusedException: A backend service [ Candlepin ] is unreachable | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:90:in `rescue in get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:80:in `get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/resources/candlepin.rb:100:in `get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:17:in `block in <class:Consumer>' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in `instance_eval' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in `run_initializer' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:139:in `lazy_attribute_get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:60:in `block (2 levels) in lazy_accessor' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:38:in `entitlement_status' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/subscription_status.rb:40:in `to_status' | /usr/share/foreman/app/models/host_status/status.rb:64:in `update_status' | /usr/share/foreman/app/models/host_status/status.rb:48:in `refresh' | /usr/share/foreman/app/models/host_status/status.rb:42:in `refresh!' | /usr/share/foreman/app/models/host/managed.rb:858:in `block in refresh_statuses' | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each_key' | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each' and in /var/log/messages Aug 4 14:20:07 wellkatello.niwa.local qdrouterd: Thu Aug 4 14:20:07 2016 ROUTER_LS (info) Router Link Lost - link_id=0 Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* cpuset=/ mems_allowed=0-1 Aug 4 14:20:12 wellkatello.niwa.local kernel: CPU: 1 PID: 7900 Comm: diagnostic_con* Not tainted 3.10.0-327.22.2.el7.x86_64 #1 Aug 4 14:20:12 wellkatello.niwa.local kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880415850b80 00000000555a1641 ffff880004e3fb68 ffffffff816360fc Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880004e3fbf8 ffffffff8163109c ffff88041ac8fc20 ffff88041ac8fc38 Aug 4 14:20:12 wellkatello.niwa.local kernel: 018804fb00000206 fffeffff00000000 0000000000000001 ffffffff81128c03 Aug 4 14:20:12 wellkatello.niwa.local kernel: Call Trace: Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff816360fc>] dump_stack+0x19/0x1b Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8163109c>] dump_header+0x8e/0x214 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81128c03>] ? delayacct_end+0x13/0xb0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116d20e>] oom_kill_process+0x24e/0x3b0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116da36>] out_of_memory+0x4b6/0x4f0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81173c26>] __alloc_pages_nodemask+0xaa6/0xba0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a401>] do_fork+0xe1/0x320 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a6c6>] SyS_clone+0x16/0x20 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646bd9>] stub_clone+0x69/0x90 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646889>] ? system_call_fastpath+0x16/0x1b Aug 4 14:20:12 wellkatello.niwa.local kernel: Mem-Info: Aug 4 14:20:12 wellkatello.niwa.local kernel: Node 0 DMA per-cpu: ... ... Aug 4 14:20:12 wellkatello.niwa.local kernel: Out of memory: Kill process 23388 (java) score 50 or sacrifice child Aug 4 14:20:12 wellkatello.niwa.local kernel: Killed process 23388 (java) total-vm:9879424kB, anon-rss:856892kB, file-rss:0kB Aug 4 14:20:13 wellkatello.niwa.local systemd: tomcat.service: main process exited, code=killed, status=9/KILL Aug 4 14:20:13 wellkatello.niwa.local puppet-master[32492]: Compiled catalog for niwa-35517.niwa.local in environment production in 0.03 seconds Aug 4 14:20:15 wellkatello.niwa.local server: Java virtual machine used: /usr/lib/jvm/jre/bin/java Aug 4 14:20:15 wellkatello.niwa.local server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Aug 4 14:20:15 wellkatello.niwa.local server: main class used: org.apache.catalina.startup.Bootstrap Aug 4 14:20:15 wellkatello.niwa.local server: flags used: Aug 4 14:20:15 wellkatello.niwa.local server: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/to$ Aug 4 14:20:15 wellkatello.niwa.local server: arguments used: stop Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[a4f48860-5627-4931-abfa-9556309ce343] Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[26b8b00d-f769-4452-920f-cf350d5fafd7] Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM org.apache.catalina.startup.Catalina stopServer Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Could not contact localhost:8005. Tomcat may not be running. Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] succeeded in 0.318352089002s: None Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM org.apache.catalina.startup.Catalina stopServer Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Catalina.stop: A reboot clears things up. I'm not sure what to provide to try and troubleshoot this issue :( -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
