[Resin-interest] Resin 4.0.44 killing itself?

2015-06-01 Thread Rick Mann 
Something is killing my resin process. The watchdog log shows this when it 
happens:

[2015/06/01 21:13:45.624] {watchdog-app-0} Watchdog detected close of 
Resin[app-0,pid=3728]
 exit reason: SIGKILL (signal=9)
[2015/06/01 21:13:45.628] {watchdog-app-0} WatchdogChild[app-0] starting
[2015/06/01 21:13:45.677] {watchdog-app-0} Watchdog starting Resin[app-0]

This seems to correlate with invoking an action that spawns some threaded 
activity. Note that this is a webapp I've been running for years on an older 
VM, and a few days ago I moved it to a new VM. ps shows this:

 4985 ?Sl 0:25 /usr/lib/jvm/java-8-oracle/bin/java -Xmx2048m 
-XX:MaxPermSize=256m -Dresin.server=app-0 
-Djava.util.logging.manager=com.caucho.log.LogManagerImpl 
-Djava.system.class.loader=com.caucho.loader.SystemClassLoader 
-Djava.endorsed.dirs=/usr/lib/jvm/java-8-oracle/jre/lib/endorsed:/lz/usr/resin/current//endorsed:/lz/var/www/endorsed
 -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl 
-Djava.awt.headless=true -Djava.awt.headlesslib=true 
-Dresin.home=/lz/usr/resin/current/ -Xss1m -d64 -server 
com.caucho.server.resin.Resin --root-directory /lz/var/www -conf 
/lz/etc/resin/resin.xml -server app-0 -socketwait 50229 -root-directory 
/lz/var/www -log-directory /logs/resin restart
21256 ?Sl11:12 /usr/lib/jvm/java-8-oracle/bin/java 
-Dresin.watchdog=app-0 
-Djava.util.logging.manager=com.caucho.log.LogManagerImpl 
-Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl 
-Djava.awt.headless=true -Djava.awt.headlesslib=true 
-Dresin.home=/lz/usr/resin/current/ -Dresin.root=/lz/var/www -Xrs -Xss256k 
-Xmx32m -d64 -server com.caucho.boot.WatchdogManager -root-directory 
/lz/var/www -conf /lz/etc/resin/resin.xml -log-directory /logs/resin start 
--log-directory /logs/resin

I started resin as root with resinctl start.

I'm not sure what else to look at. Any suggestions?

Thanks!

-- 
Rick Mann
rm...@latencyzero.com



___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Re: [Resin-interest] Resin 4.0.44 killing itself?

2015-06-01 Thread Scott Ferguson 
On 6/1/15 2:19 PM, Rick Mann wrote:
 Something is killing my resin process. The watchdog log shows this when it 
 happens:

 [2015/06/01 21:13:45.624] {watchdog-app-0} Watchdog detected close of 
 Resin[app-0,pid=3728]
   exit reason: SIGKILL (signal=9)
 [2015/06/01 21:13:45.628] {watchdog-app-0} WatchdogChild[app-0] starting
 [2015/06/01 21:13:45.677] {watchdog-app-0} Watchdog starting Resin[app-0]

 This seems to correlate with invoking an action that spawns some threaded 
 activity. Note that this is a webapp I've been running for years on an older 
 VM, and a few days ago I moved it to a new VM. ps shows this:
Is there anything in the resin log?

The only 'kill' is in the watchdog, and it doesn't use a sigkill. It 
just calls the JVM's process.close().

I checked the JNI and there's no kill there.

Anything in /var/log possibly?

-- Scott


   4985 ?Sl 0:25 /usr/lib/jvm/java-8-oracle/bin/java -Xmx2048m 
 -XX:MaxPermSize=256m -Dresin.server=app-0 
 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl 
 -Djava.system.class.loader=com.caucho.loader.SystemClassLoader 
 -Djava.endorsed.dirs=/usr/lib/jvm/java-8-oracle/jre/lib/endorsed:/lz/usr/resin/current//endorsed:/lz/var/www/endorsed
  -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl 
 -Djava.awt.headless=true -Djava.awt.headlesslib=true 
 -Dresin.home=/lz/usr/resin/current/ -Xss1m -d64 -server 
 com.caucho.server.resin.Resin --root-directory /lz/var/www -conf 
 /lz/etc/resin/resin.xml -server app-0 -socketwait 50229 -root-directory 
 /lz/var/www -log-directory /logs/resin restart
 21256 ?Sl11:12 /usr/lib/jvm/java-8-oracle/bin/java 
 -Dresin.watchdog=app-0 
 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl 
 -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl 
 -Djava.awt.headless=true -Djava.awt.headlesslib=true 
 -Dresin.home=/lz/usr/resin/current/ -Dresin.root=/lz/var/www -Xrs -Xss256k 
 -Xmx32m -d64 -server com.caucho.boot.WatchdogManager -root-directory 
 /lz/var/www -conf /lz/etc/resin/resin.xml -log-directory /logs/resin start 
 --log-directory /logs/resin

 I started resin as root with resinctl start.

 I'm not sure what else to look at. Any suggestions?

 Thanks!



___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Re: [Resin-interest] Resin 4.0.44 killing itself?

2015-06-01 Thread Rick Mann 
Ah, good call. I found this in kern.log. Both the old and the new VM have 512 
MB (this is at Rackspace). The older one was running an older Ubuntu, and Java 
7. This is running Java 8. I gotta track down all the possible memory config 
options to resin. But I'm a little troubled that my virtually no-traffic server 
is having memory issues.

Jun  1 22:09:05 tycho2 kernel: [945096.295849] java invoked oom-killer: 
gfp_mask=0x201da, order=0, oom_score_adj=0
Jun  1 22:09:05 tycho2 kernel: [945096.295855] java cpuset=/ mems_allowed=0
Jun  1 22:09:05 tycho2 kernel: [945096.295859] CPU: 0 PID: 4991 Comm: java Not 
tainted 3.13.0-48-generic #80-Ubuntu
Jun  1 22:09:05 tycho2 kernel: [945096.295862] Hardware name: Xen HVM domU, 
BIOS 4.1.1 09/13/2011
Jun  1 22:09:05 tycho2 kernel: [945096.295864]   
88001b253980 81721506 880009799800
Jun  1 22:09:05 tycho2 kernel: [945096.295868]  88001b253a08 
8171bdc1 81067996 88001b2539e0
Jun  1 22:09:05 tycho2 kernel: [945096.295871]  810c75dc 
 88001fbfce28 
Jun  1 22:09:05 tycho2 kernel: [945096.295874] Call Trace:
Jun  1 22:09:05 tycho2 kernel: [945096.295882]  [81721506] 
dump_stack+0x45/0x56
Jun  1 22:09:05 tycho2 kernel: [945096.295886]  [8171bdc1] 
dump_header+0x7f/0x1f1
Jun  1 22:09:05 tycho2 kernel: [945096.295890]  [81067996] ? 
put_online_cpus+0x56/0x80
Jun  1 22:09:05 tycho2 kernel: [945096.295895]  [810c75dc] ? 
rcu_oom_notify+0xcc/0xf0
Jun  1 22:09:05 tycho2 kernel: [945096.295899]  [811529be] 
oom_kill_process+0x1ce/0x330
Jun  1 22:09:05 tycho2 kernel: [945096.295903]  [812d7225] ? 
security_capable_noaudit+0x15/0x20
Jun  1 22:09:05 tycho2 kernel: [945096.295905]  [811530f4] 
out_of_memory+0x414/0x450
Jun  1 22:09:05 tycho2 kernel: [945096.295909]  [81159460] 
__alloc_pages_nodemask+0xa60/0xb80
Jun  1 22:09:05 tycho2 kernel: [945096.295913]  [81197ad3] 
alloc_pages_current+0xa3/0x160
Jun  1 22:09:05 tycho2 kernel: [945096.295917]  [8114f577] 
__page_cache_alloc+0x97/0xc0
Jun  1 22:09:05 tycho2 kernel: [945096.295920]  [81150f85] 
filemap_fault+0x185/0x410
Jun  1 22:09:05 tycho2 kernel: [945096.295924]  [81175d8f] 
__do_fault+0x6f/0x530
Jun  1 22:09:05 tycho2 kernel: [945096.295927]  [81179f32] 
handle_mm_fault+0x482/0xf10
Jun  1 22:09:05 tycho2 kernel: [945096.295930]  [810d7f78] ? 
get_futex_key+0x1d8/0x2c0
Jun  1 22:09:05 tycho2 kernel: [945096.295934]  [8172d534] 
__do_page_fault+0x184/0x560
Jun  1 22:09:05 tycho2 kernel: [945096.295938]  [810a02c5] ? 
set_next_entity+0x95/0xb0
Jun  1 22:09:05 tycho2 kernel: [945096.295942]  [8101260b] ? 
__switch_to+0x16b/0x4d0
Jun  1 22:09:05 tycho2 kernel: [945096.295945]  [8172d92a] 
do_page_fault+0x1a/0x70
Jun  1 22:09:05 tycho2 kernel: [945096.295948]  [81729d68] 
page_fault+0x28/0x30
Jun  1 22:09:05 tycho2 kernel: [945096.295950] Mem-Info:
Jun  1 22:09:05 tycho2 kernel: [945096.295951] Node 0 DMA per-cpu:
Jun  1 22:09:05 tycho2 kernel: [945096.295954] CPU0: hi:0, btch:   1 
usd:   0
Jun  1 22:09:05 tycho2 kernel: [945096.295955] Node 0 DMA32 per-cpu:
Jun  1 22:09:05 tycho2 kernel: [945096.295957] CPU0: hi:  186, btch:  31 
usd:  30
Jun  1 22:09:05 tycho2 kernel: [945096.295962] active_anon:112388 
inactive_anon:156 isolated_anon:0
Jun  1 22:09:05 tycho2 kernel: [945096.295962]  active_file:16 inactive_file:59 
isolated_file:0
Jun  1 22:09:05 tycho2 kernel: [945096.295962]  unevictable:0 dirty:7 
writeback:0 unstable:0
Jun  1 22:09:05 tycho2 kernel: [945096.295962]  free:1137 slab_reclaimable:2500 
slab_unreclaimable:2899
Jun  1 22:09:05 tycho2 kernel: [945096.295962]  mapped:41 shmem:183 
pagetables:1123 bounce:0
Jun  1 22:09:05 tycho2 kernel: [945096.295962]  free_cma:0
Jun  1 22:09:05 tycho2 kernel: [945096.295966] Node 0 DMA free:1952kB min:88kB 
low:108kB high:132kB active_anon:12508kB inactive_anon:12kB active_file:0kB 
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:15988kB managed:15904kB mlocked:0kB dirty:4kB writeback:0kB mapped:16kB 
shmem:20kB slab_reclaimable:728kB slab_unreclaimable:280kB kernel_stack:72kB 
pagetables:108kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:295 all_unreclaimable? yes
Jun  1 22:09:05 tycho2 kernel: [945096.295972] lowmem_reserve[]: 0 466 466 466
Jun  1 22:09:05 tycho2 kernel: [945096.295975] Node 0 DMA32 free:2596kB 
min:2716kB low:3392kB high:4072kB active_anon:437044kB inactive_anon:612kB 
active_file:64kB inactive_file:188kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:503808kB managed:480212kB mlocked:0kB dirty:24kB 
writeback:0kB mapped:148kB shmem:712kB slab_reclaimable:9272kB 
slab_unreclaimable:11316kB kernel_stack:2016kB pagetables:4384kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:403 all_unreclaimable? 
yes