Further, the capsule syncs seem to take swap space (even if RAM is/was available) and never gives it back - to the point where we were publishing content views and tomcat failed, in /var/log/foreman/production.log start getting...
2016-08-04 14:20:15 [app] [I] Imported report for niwa-35649.niwa.local in 0.12 seconds 2016-08-04 14:20:15 [app] [W] Action failed | Katello::Errors::ConnectionRefusedException: A backend service [ Candlepin ] is unreachable | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:90:in `rescue in get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:80:in `get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/resources/candlepin.rb:100:in `get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:17:in `block in <class:Consumer>' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in `instance_eval' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in `run_initializer' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:139:in `lazy_attribute_get' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:60:in `block (2 levels) in lazy_accessor' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:38:in `entitlement_status' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/subscription_status.rb:40:in `to_status' | /usr/share/foreman/app/models/host_status/status.rb:64:in `update_status' | /usr/share/foreman/app/models/host_status/status.rb:48:in `refresh' | /usr/share/foreman/app/models/host_status/status.rb:42:in `refresh!' | /usr/share/foreman/app/models/host/managed.rb:858:in `block in refresh_statuses' | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each_key' | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each' and in /var/log/messages Aug 4 14:20:07 wellkatello.niwa.local qdrouterd: Thu Aug 4 14:20:07 2016 ROUTER_LS (info) Router Link Lost - link_id=0 Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* cpuset=/ mems_allowed=0-1 Aug 4 14:20:12 wellkatello.niwa.local kernel: CPU: 1 PID: 7900 Comm: diagnostic_con* Not tainted 3.10.0-327.22.2.el7.x86_64 #1 Aug 4 14:20:12 wellkatello.niwa.local kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880415850b80 00000000555a1641 ffff880004e3fb68 ffffffff816360fc Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880004e3fbf8 ffffffff8163109c ffff88041ac8fc20 ffff88041ac8fc38 Aug 4 14:20:12 wellkatello.niwa.local kernel: 018804fb00000206 fffeffff00000000 0000000000000001 ffffffff81128c03 Aug 4 14:20:12 wellkatello.niwa.local kernel: Call Trace: Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff816360fc>] dump_stack+0x19/0x1b Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8163109c>] dump_header+0x8e/0x214 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81128c03>] ? delayacct_end+0x13/0xb0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116d20e>] oom_kill_process+0x24e/0x3b0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116da36>] out_of_memory+0x4b6/0x4f0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81173c26>] __alloc_pages_nodemask+0xaa6/0xba0 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a401>] do_fork+0xe1/0x320 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a6c6>] SyS_clone+0x16/0x20 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646bd9>] stub_clone+0x69/0x90 Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646889>] ? system_call_fastpath+0x16/0x1b Aug 4 14:20:12 wellkatello.niwa.local kernel: Mem-Info: Aug 4 14:20:12 wellkatello.niwa.local kernel: Node 0 DMA per-cpu: ... ... Aug 4 14:20:12 wellkatello.niwa.local kernel: Out of memory: Kill process 23388 (java) score 50 or sacrifice child Aug 4 14:20:12 wellkatello.niwa.local kernel: Killed process 23388 (java) total-vm:9879424kB, anon-rss:856892kB, file-rss:0kB Aug 4 14:20:13 wellkatello.niwa.local systemd: tomcat.service: main process exited, code=killed, status=9/KILL Aug 4 14:20:13 wellkatello.niwa.local puppet-master[32492]: Compiled catalog for niwa-35517.niwa.local in environment production in 0.03 seconds Aug 4 14:20:15 wellkatello.niwa.local server: Java virtual machine used: /usr/lib/jvm/jre/bin/java Aug 4 14:20:15 wellkatello.niwa.local server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Aug 4 14:20:15 wellkatello.niwa.local server: main class used: org.apache.catalina.startup.Bootstrap Aug 4 14:20:15 wellkatello.niwa.local server: flags used: Aug 4 14:20:15 wellkatello.niwa.local server: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/to$ Aug 4 14:20:15 wellkatello.niwa.local server: arguments used: stop Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[a4f48860-5627-4931-abfa-9556309ce343] Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[26b8b00d-f769-4452-920f-cf350d5fafd7] Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM org.apache.catalina.startup.Catalina stopServer Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Could not contact localhost:8005. Tomcat may not be running. Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] succeeded in 0.318352089002s: None Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM org.apache.catalina.startup.Catalina stopServer Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Catalina.stop: A reboot clears things up. I'm not sure what to provide to try and troubleshoot this issue :( -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
