Ah, good call. I found this in kern.log. Both the old and the new VM have 512
MB (this is at Rackspace). The older one was running an older Ubuntu, and Java
7. This is running Java 8. I gotta track down all the possible memory config
options to resin. But I'm a little troubled that my virtually no-traffic server
is having memory issues.
Jun 1 22:09:05 tycho2 kernel: [945096.295849] java invoked oom-killer:
gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 1 22:09:05 tycho2 kernel: [945096.295855] java cpuset=/ mems_allowed=0
Jun 1 22:09:05 tycho2 kernel: [945096.295859] CPU: 0 PID: 4991 Comm: java Not
tainted 3.13.0-48-generic #80-Ubuntu
Jun 1 22:09:05 tycho2 kernel: [945096.295862] Hardware name: Xen HVM domU,
BIOS 4.1.1 09/13/2011
Jun 1 22:09:05 tycho2 kernel: [945096.295864] 0000000000000000
ffff88001b253980 ffffffff81721506 ffff880009799800
Jun 1 22:09:05 tycho2 kernel: [945096.295868] ffff88001b253a08
ffffffff8171bdc1 ffffffff81067996 ffff88001b2539e0
Jun 1 22:09:05 tycho2 kernel: [945096.295871] ffffffff810c75dc
0000000000000000 ffff88001fbfce28 0000000000000000
Jun 1 22:09:05 tycho2 kernel: [945096.295874] Call Trace:
Jun 1 22:09:05 tycho2 kernel: [945096.295882] [<ffffffff81721506>]
dump_stack+0x45/0x56
Jun 1 22:09:05 tycho2 kernel: [945096.295886] [<ffffffff8171bdc1>]
dump_header+0x7f/0x1f1
Jun 1 22:09:05 tycho2 kernel: [945096.295890] [<ffffffff81067996>] ?
put_online_cpus+0x56/0x80
Jun 1 22:09:05 tycho2 kernel: [945096.295895] [<ffffffff810c75dc>] ?
rcu_oom_notify+0xcc/0xf0
Jun 1 22:09:05 tycho2 kernel: [945096.295899] [<ffffffff811529be>]
oom_kill_process+0x1ce/0x330
Jun 1 22:09:05 tycho2 kernel: [945096.295903] [<ffffffff812d7225>] ?
security_capable_noaudit+0x15/0x20
Jun 1 22:09:05 tycho2 kernel: [945096.295905] [<ffffffff811530f4>]
out_of_memory+0x414/0x450
Jun 1 22:09:05 tycho2 kernel: [945096.295909] [<ffffffff81159460>]
__alloc_pages_nodemask+0xa60/0xb80
Jun 1 22:09:05 tycho2 kernel: [945096.295913] [<ffffffff81197ad3>]
alloc_pages_current+0xa3/0x160
Jun 1 22:09:05 tycho2 kernel: [945096.295917] [<ffffffff8114f577>]
__page_cache_alloc+0x97/0xc0
Jun 1 22:09:05 tycho2 kernel: [945096.295920] [<ffffffff81150f85>]
filemap_fault+0x185/0x410
Jun 1 22:09:05 tycho2 kernel: [945096.295924] [<ffffffff81175d8f>]
__do_fault+0x6f/0x530
Jun 1 22:09:05 tycho2 kernel: [945096.295927] [<ffffffff81179f32>]
handle_mm_fault+0x482/0xf10
Jun 1 22:09:05 tycho2 kernel: [945096.295930] [<ffffffff810d7f78>] ?
get_futex_key+0x1d8/0x2c0
Jun 1 22:09:05 tycho2 kernel: [945096.295934] [<ffffffff8172d534>]
__do_page_fault+0x184/0x560
Jun 1 22:09:05 tycho2 kernel: [945096.295938] [<ffffffff810a02c5>] ?
set_next_entity+0x95/0xb0
Jun 1 22:09:05 tycho2 kernel: [945096.295942] [<ffffffff8101260b>] ?
__switch_to+0x16b/0x4d0
Jun 1 22:09:05 tycho2 kernel: [945096.295945] [<ffffffff8172d92a>]
do_page_fault+0x1a/0x70
Jun 1 22:09:05 tycho2 kernel: [945096.295948] [<ffffffff81729d68>]
page_fault+0x28/0x30
Jun 1 22:09:05 tycho2 kernel: [945096.295950] Mem-Info:
Jun 1 22:09:05 tycho2 kernel: [945096.295951] Node 0 DMA per-cpu:
Jun 1 22:09:05 tycho2 kernel: [945096.295954] CPU 0: hi: 0, btch: 1
usd: 0
Jun 1 22:09:05 tycho2 kernel: [945096.295955] Node 0 DMA32 per-cpu:
Jun 1 22:09:05 tycho2 kernel: [945096.295957] CPU 0: hi: 186, btch: 31
usd: 30
Jun 1 22:09:05 tycho2 kernel: [945096.295962] active_anon:112388
inactive_anon:156 isolated_anon:0
Jun 1 22:09:05 tycho2 kernel: [945096.295962] active_file:16 inactive_file:59
isolated_file:0
Jun 1 22:09:05 tycho2 kernel: [945096.295962] unevictable:0 dirty:7
writeback:0 unstable:0
Jun 1 22:09:05 tycho2 kernel: [945096.295962] free:1137 slab_reclaimable:2500
slab_unreclaimable:2899
Jun 1 22:09:05 tycho2 kernel: [945096.295962] mapped:41 shmem:183
pagetables:1123 bounce:0
Jun 1 22:09:05 tycho2 kernel: [945096.295962] free_cma:0
Jun 1 22:09:05 tycho2 kernel: [945096.295966] Node 0 DMA free:1952kB min:88kB
low:108kB high:132kB active_anon:12508kB inactive_anon:12kB active_file:0kB
inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:15988kB managed:15904kB mlocked:0kB dirty:4kB writeback:0kB mapped:16kB
shmem:20kB slab_reclaimable:728kB slab_unreclaimable:280kB kernel_stack:72kB
pagetables:108kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:295 all_unreclaimable? yes
Jun 1 22:09:05 tycho2 kernel: [945096.295972] lowmem_reserve[]: 0 466 466 466
Jun 1 22:09:05 tycho2 kernel: [945096.295975] Node 0 DMA32 free:2596kB
min:2716kB low:3392kB high:4072kB active_anon:437044kB inactive_anon:612kB
active_file:64kB inactive_file:188kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:503808kB managed:480212kB mlocked:0kB dirty:24kB
writeback:0kB mapped:148kB shmem:712kB slab_reclaimable:9272kB
slab_unreclaimable:11316kB kernel_stack:2016kB pagetables:4384kB unstable:0kB
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:403 all_unreclaimable?
yes
Jun 1 22:09:05 tycho2 kernel: [945096.295981] lowmem_reserve[]: 0 0 0 0
Jun 1 22:09:05 tycho2 kernel: [945096.295983] Node 0 DMA: 79*4kB (UM) 3*8kB
(MR) 1*16kB (R) 6*32kB (R) 6*64kB (R) 4*128kB (R) 2*256kB (R) 0*512kB 0*1024kB
0*2048kB 0*4096kB = 1956kB
Jun 1 22:09:05 tycho2 kernel: [945096.295995] Node 0 DMA32: 65*4kB (E) 52*8kB
(UEMR) 62*16kB (UEMR) 5*32kB (EM) 4*64kB (R) 2*128kB (R) 1*256kB (R) 0*512kB
0*1024kB 0*2048kB 0*4096kB = 2596kB
Jun 1 22:09:05 tycho2 kernel: [945096.296006] Node 0 hugepages_total=0
hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 1 22:09:05 tycho2 kernel: [945096.296008] 285 total pagecache pages
Jun 1 22:09:05 tycho2 kernel: [945096.296010] 0 pages in swap cache
Jun 1 22:09:05 tycho2 kernel: [945096.296011] Swap cache stats: add 0, delete
0, find 0/0
Jun 1 22:09:05 tycho2 kernel: [945096.296013] Free swap = 0kB
Jun 1 22:09:05 tycho2 kernel: [945096.296014] Total swap = 0kB
Jun 1 22:09:05 tycho2 kernel: [945096.296015] 129949 pages RAM
Jun 1 22:09:05 tycho2 kernel: [945096.296017] 0 pages HighMem/MovableOnly
Jun 1 22:09:05 tycho2 kernel: [945096.296018] 5899 pages reserved
Jun 1 22:09:05 tycho2 kernel: [945096.296019] [ pid ] uid tgid total_vm
rss nr_ptes swapents oom_score_adj name
Jun 1 22:09:05 tycho2 kernel: [945096.296025] [ 315] 0 315 4868
59 13 0 0 upstart-udev-br
Jun 1 22:09:05 tycho2 kernel: [945096.296028] [ 325] 0 325 12808
140 28 0 -1000 systemd-udevd
Jun 1 22:09:05 tycho2 kernel: [945096.296031] [ 412] 0 412 4521
95 14 0 0 xe-daemon
Jun 1 22:09:05 tycho2 kernel: [945096.296035] [ 534] 0 534 3814
60 11 0 0 upstart-socket-
Jun 1 22:09:05 tycho2 kernel: [945096.296037] [ 602] 0 602 34802
1684 38 0 0 nova-agent
Jun 1 22:09:05 tycho2 kernel: [945096.296040] [ 1062] 0 1062 3818
59 11 0 0 upstart-file-br
Jun 1 22:09:05 tycho2 kernel: [945096.296042] [ 1067] 102 1067 9805
99 23 0 0 dbus-daemon
Jun 1 22:09:05 tycho2 kernel: [945096.296045] [ 1124] 0 1124 10862
90 26 0 0 systemd-logind
Jun 1 22:09:05 tycho2 kernel: [945096.296047] [ 1125] 101 1125 64057
7289 40 0 0 rsyslogd
Jun 1 22:09:05 tycho2 kernel: [945096.296063] [ 1160] 0 1160 4014
41 13 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296087] [ 1162] 0 1162 4014
40 13 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296089] [ 1165] 0 1165 4014
38 13 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296092] [ 1166] 0 1166 4014
40 12 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296095] [ 1168] 0 1168 4014
41 13 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296097] [ 1197] 0 1197 5913
55 17 0 0 cron
Jun 1 22:09:05 tycho2 kernel: [945096.296100] [ 1410] 0 1410 15341
170 34 0 -1000 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296103] [ 7348] 0 7348 4014
41 13 0 0 getty
Jun 1 22:09:05 tycho2 kernel: [945096.296106] [21256] 0 21256 487466
19814 93 0 0 java
Jun 1 22:09:05 tycho2 kernel: [945096.296108] [ 2534] 0 2534 26408
248 56 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296111] [ 2661] 1001 2661 26442
258 53 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296114] [ 2665] 1001 2665 5654
455 17 0 0 bash
Jun 1 22:09:05 tycho2 kernel: [945096.296116] [ 3855] 0 3855 26408
245 57 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296119] [ 3873] 1001 3873 26408
270 55 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296121] [ 3874] 1001 3874 3205
44 11 0 0 sftp-server
Jun 1 22:09:05 tycho2 kernel: [945096.296124] [ 4789] 0 4789 26408
247 55 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296127] [ 4815] 1001 4815 26408
248 53 0 0 sshd
Jun 1 22:09:05 tycho2 kernel: [945096.296129] [ 4816] 1001 4816 5621
420 16 0 0 bash
Jun 1 22:09:05 tycho2 kernel: [945096.296132] [ 4985] 0 4985 1036736
78333 256 0 0 java
Jun 1 22:09:05 tycho2 kernel: [945096.296134] [ 8136] 0 8136 1085
19 8 0 0 sleep
Jun 1 22:09:05 tycho2 kernel: [945096.296137] [ 8146] 0 8146 16297
118 36 0 0 sudo
Jun 1 22:09:05 tycho2 kernel: [945096.296139] [ 8147] 0 8147 1868
24 9 0 0 tail
Jun 1 22:09:05 tycho2 kernel: [945096.296141] Out of memory: Kill process 4985
(java) score 614 or sacrifice child
Jun 1 22:09:05 tycho2 kernel: [945096.296208] Killed process 4985 (java)
total-vm:4146944kB, anon-rss:313332kB, file-rss:0kB
> On Jun 1, 2015, at 14:39 , Scott Ferguson <f...@caucho.com> wrote:
>
> On 6/1/15 2:19 PM, Rick Mann wrote:
>> Something is killing my resin process. The watchdog log shows this when it
>> happens:
>>
>> [2015/06/01 21:13:45.624] {watchdog-app-0} Watchdog detected close of
>> Resin[app-0,pid=3728]
>> exit reason: SIGKILL (signal=9)
>> [2015/06/01 21:13:45.628] {watchdog-app-0} WatchdogChild[app-0] starting
>> [2015/06/01 21:13:45.677] {watchdog-app-0} Watchdog starting Resin[app-0]
>>
>> This seems to correlate with invoking an action that spawns some threaded
>> activity. Note that this is a webapp I've been running for years on an older
>> VM, and a few days ago I moved it to a new VM. ps shows this:
> Is there anything in the resin log?
>
> The only 'kill' is in the watchdog, and it doesn't use a sigkill. It
> just calls the JVM's process.close().
>
> I checked the JNI and there's no kill there.
>
> Anything in /var/log possibly?
>
> -- Scott
>
>>
>> 4985 ? Sl 0:25 /usr/lib/jvm/java-8-oracle/bin/java -Xmx2048m
>> -XX:MaxPermSize=256m -Dresin.server=app-0
>> -Djava.util.logging.manager=com.caucho.log.LogManagerImpl
>> -Djava.system.class.loader=com.caucho.loader.SystemClassLoader
>> -Djava.endorsed.dirs=/usr/lib/jvm/java-8-oracle/jre/lib/endorsed:/lz/usr/resin/current//endorsed:/lz/var/www/endorsed
>> -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl
>> -Djava.awt.headless=true -Djava.awt.headlesslib=true
>> -Dresin.home=/lz/usr/resin/current/ -Xss1m -d64 -server
>> com.caucho.server.resin.Resin --root-directory /lz/var/www -conf
>> /lz/etc/resin/resin.xml -server app-0 -socketwait 50229 -root-directory
>> /lz/var/www -log-directory /logs/resin restart
>> 21256 ? Sl 11:12 /usr/lib/jvm/java-8-oracle/bin/java
>> -Dresin.watchdog=app-0
>> -Djava.util.logging.manager=com.caucho.log.LogManagerImpl
>> -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl
>> -Djava.awt.headless=true -Djava.awt.headlesslib=true
>> -Dresin.home=/lz/usr/resin/current/ -Dresin.root=/lz/var/www -Xrs -Xss256k
>> -Xmx32m -d64 -server com.caucho.boot.WatchdogManager -root-directory
>> /lz/var/www -conf /lz/etc/resin/resin.xml -log-directory /logs/resin start
>> --log-directory /logs/resin
>>
>> I started resin as root with "resinctl start."
>>
>> I'm not sure what else to look at. Any suggestions?
>>
>> Thanks!
>>
>
>
> _______________________________________________
> resin-interest mailing list
> resin-interest@caucho.com
> http://maillist.caucho.com/mailman/listinfo/resin-interest
--
Rick Mann
rm...@latencyzero.com
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
