Dylan, I would be curious to see a foreman-debug report of this. Is there anyway you could prepare one and send it to myself and Eric?
-John John Mitsch Red Hat Engineering (860)-967-7285 irc: jomitsch On Wed, Aug 3, 2016 at 11:09 PM, Dylan Baars <[email protected]> wrote: > Further, the capsule syncs seem to take swap space (even if RAM is/was > available) and never gives it back - to the point where we were publishing > content views and tomcat failed, in /var/log/foreman/production.log start > getting... > > 2016-08-04 14:20:15 [app] [I] Imported report for niwa-35649.niwa.local in > 0.12 seconds > 2016-08-04 14:20:15 [app] [W] Action failed > | Katello::Errors::ConnectionRefusedException: A backend service [ > Candlepin ] is unreachable > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:90:in > `rescue in get' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/http_resource.rb:80:in > `get' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/resources/candlepin.rb:100:in > `get' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:17:in > `block in <class:Consumer>' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in > `instance_eval' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:160:in > `run_initializer' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:139:in > `lazy_attribute_get' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/lib/katello/lazy_accessor.rb:60:in > `block (2 levels) in lazy_accessor' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/services/katello/candlepin/consumer.rb:38:in > `entitlement_status' > | > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.2/app/models/katello/subscription_status.rb:40:in > `to_status' > | /usr/share/foreman/app/models/host_status/status.rb:64:in > `update_status' > | /usr/share/foreman/app/models/host_status/status.rb:48:in `refresh' > | /usr/share/foreman/app/models/host_status/status.rb:42:in `refresh!' > | /usr/share/foreman/app/models/host/managed.rb:858:in `block in > refresh_statuses' > | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each_key' > | /opt/rh/rh-ruby22/root/usr/share/ruby/set.rb:283:in `each' > > and in /var/log/messages > > Aug 4 14:20:07 wellkatello.niwa.local qdrouterd: Thu Aug 4 14:20:07 2016 > ROUTER_LS (info) Router Link Lost - link_id=0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* invoked > oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: diagnostic_con* cpuset=/ > mems_allowed=0-1 > Aug 4 14:20:12 wellkatello.niwa.local kernel: CPU: 1 PID: 7900 Comm: > diagnostic_con* Not tainted 3.10.0-327.22.2.el7.x86_64 #1 > Aug 4 14:20:12 wellkatello.niwa.local kernel: Hardware name: VMware, Inc. > VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 > 04/14/2014 > Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880415850b80 > 00000000555a1641 ffff880004e3fb68 ffffffff816360fc > Aug 4 14:20:12 wellkatello.niwa.local kernel: ffff880004e3fbf8 > ffffffff8163109c ffff88041ac8fc20 ffff88041ac8fc38 > Aug 4 14:20:12 wellkatello.niwa.local kernel: 018804fb00000206 > fffeffff00000000 0000000000000001 ffffffff81128c03 > Aug 4 14:20:12 wellkatello.niwa.local kernel: Call Trace: > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff816360fc>] > dump_stack+0x19/0x1b > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8163109c>] > dump_header+0x8e/0x214 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81128c03>] ? > delayacct_end+0x13/0xb0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116d20e>] > oom_kill_process+0x24e/0x3b0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81088dae>] ? > has_capability_noaudit+0x1e/0x30 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8116da36>] > out_of_memory+0x4b6/0x4f0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81173c26>] > __alloc_pages_nodemask+0xaa6/0xba0 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81078d73>] > copy_process.part.25+0x163/0x1610 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a401>] > do_fork+0xe1/0x320 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff8107a6c6>] > SyS_clone+0x16/0x20 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646bd9>] > stub_clone+0x69/0x90 > Aug 4 14:20:12 wellkatello.niwa.local kernel: [<ffffffff81646889>] ? > system_call_fastpath+0x16/0x1b > Aug 4 14:20:12 wellkatello.niwa.local kernel: Mem-Info: > Aug 4 14:20:12 wellkatello.niwa.local kernel: Node 0 DMA per-cpu: > ... > ... > Aug 4 14:20:12 wellkatello.niwa.local kernel: Out of memory: Kill process > 23388 (java) score 50 or sacrifice child > Aug 4 14:20:12 wellkatello.niwa.local kernel: Killed process 23388 (java) > total-vm:9879424kB, anon-rss:856892kB, file-rss:0kB > Aug 4 14:20:13 wellkatello.niwa.local systemd: tomcat.service: main > process exited, code=killed, status=9/KILL > Aug 4 14:20:13 wellkatello.niwa.local puppet-master[32492]: Compiled > catalog for niwa-35517.niwa.local in environment production in 0.03 seconds > Aug 4 14:20:15 wellkatello.niwa.local server: Java virtual machine used: > /usr/lib/jvm/jre/bin/java > Aug 4 14:20:15 wellkatello.niwa.local server: classpath used: > /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar > Aug 4 14:20:15 wellkatello.niwa.local server: main class used: > org.apache.catalina.startup.Bootstrap > Aug 4 14:20:15 wellkatello.niwa.local server: flags used: > Aug 4 14:20:15 wellkatello.niwa.local server: options used: > -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat > -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp > -Djava.util.logging.config.file=/usr/share/to$ > Aug 4 14:20:15 wellkatello.niwa.local server: arguments used: stop > Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: > Received task: > pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] > Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: > Received task: > pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[a4f48860-5627-4931-abfa-9556309ce343] > Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.strategy:INFO: > Received task: > pulp.server.async.tasks._release_resource[26b8b00d-f769-4452-920f-cf350d5fafd7] > Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM > org.apache.catalina.startup.Catalina stopServer > Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Could not contact > localhost:8005. Tomcat may not be running. > Aug 4 14:20:24 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task > pulp.server.async.tasks._queue_reserved_task[8a0acee7-a3ae-4eba-9b45-f5edb974d71e] > succeeded in 0.318352089002s: None > Aug 4 14:20:24 wellkatello.niwa.local server: Aug 04, 2016 2:20:24 PM > org.apache.catalina.startup.Catalina stopServer > Aug 4 14:20:24 wellkatello.niwa.local server: SEVERE: Catalina.stop: > > A reboot clears things up. > > I'm not sure what to provide to try and troubleshoot this issue :( > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
