This issue has been harassing our cluster. Any thoughts would be
appreciated.
I've attached the syslog showing the issue. I've cut out several sections
of oom kills of node processes, however I've left in the last two.
Previous oom kill messages are similar.
These oom kills occur before the system locks up, as can be seen in the
attached log message.
What on earth is going on? https://issues.apache.org/jira/browse/MESOS-662
describes kernel hangs due to OOM situations. I'm not saying this is
happening again; I am wondering if there is an other issue that is causing
a lockup.
Thoughts?
Jun 24 00:40:20 dalmesos01 kernel: [78508.015523] type=1400
audit(1435106420.483:9699): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=29037 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:20 dalmesos01 kernel: [78508.015604] type=1400
audit(1435106420.483:9700): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=29036 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:40 dalmesos01 kernel: [78527.739905] type=1400
audit(1435106440.228:9701): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27130 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:40 dalmesos01 kernel: [78527.967288] type=1400
audit(1435106440.456:9702): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27129 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:41 dalmesos01 kernel: [78528.957358] type=1400
audit(1435106441.448:9703): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27061 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:41 dalmesos01 kernel: [78528.957386] type=1400
audit(1435106441.448:9704): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27061 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:41 dalmesos01 kernel: [78529.040507] node invoked oom-killer:
gfp_mask=0xd0, order=0, oom_score_adj=0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040514] node
cpuset=2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b
mems_allowed=0-1
Jun 24 00:40:41 dalmesos01 kernel: [78529.040520] CPU: 2 PID: 30851 Comm: node
Not tainted 3.13.0-43-generic #72-Ubuntu
Jun 24 00:40:41 dalmesos01 kernel: [78529.040522] Hardware name: Cisco Systems
Inc UCSC-C240-M3S/UCSC-C240-M3S, BIOS C240M3.1.5.7.0.042820140452 04/28/2014
Jun 24 00:40:41 dalmesos01 kernel: [78529.040524] ffff881fc7317400
ffff880f57ae3c48 ffffffff81720bf6 ffff881fe96d6000
Jun 24 00:40:41 dalmesos01 kernel: [78529.040530] ffff880f57ae3cd0
ffffffff8171b4b1 ffffffff81c867e0 0000000000000000
Jun 24 00:40:41 dalmesos01 kernel: [78529.040538] 0000000000000015
ffff880f57ae3ca0 ffffffff811522a7 ffffffffffffff04
Jun 24 00:40:41 dalmesos01 kernel: [78529.040546] Call Trace:
Jun 24 00:40:41 dalmesos01 kernel: [78529.040557] [<ffffffff81720bf6>]
dump_stack+0x45/0x56
Jun 24 00:40:41 dalmesos01 kernel: [78529.040563] [<ffffffff8171b4b1>]
dump_header+0x7f/0x1f1
Jun 24 00:40:41 dalmesos01 kernel: [78529.040569] [<ffffffff811522a7>] ?
find_lock_task_mm+0x27/0x70
Jun 24 00:40:41 dalmesos01 kernel: [78529.040574] [<ffffffff811526de>]
oom_kill_process+0x1ce/0x330
Jun 24 00:40:41 dalmesos01 kernel: [78529.040581] [<ffffffff811b0852>] ?
mem_cgroup_iter+0x92/0x340
Jun 24 00:40:41 dalmesos01 kernel: [78529.040586] [<ffffffff811b491c>]
mem_cgroup_oom_synchronize+0x51c/0x560
Jun 24 00:40:41 dalmesos01 kernel: [78529.040591] [<ffffffff811b3e50>] ?
mem_cgroup_charge_common+0xa0/0xa0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040596] [<ffffffff81152e64>]
pagefault_out_of_memory+0x14/0x80
Jun 24 00:40:41 dalmesos01 kernel: [78529.040600] [<ffffffff81719aa1>]
mm_fault_error+0x8e/0x180
Jun 24 00:40:41 dalmesos01 kernel: [78529.040606] [<ffffffff8172cf31>]
__do_page_fault+0x4a1/0x560
Jun 24 00:40:41 dalmesos01 kernel: [78529.040611] [<ffffffff810a0255>] ?
set_next_entity+0x95/0xb0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040616] [<ffffffff81012609>] ?
__switch_to+0x169/0x4c0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040621] [<ffffffff8172d00a>]
do_page_fault+0x1a/0x70
Jun 24 00:40:41 dalmesos01 kernel: [78529.040626] [<ffffffff81729468>]
page_fault+0x28/0x30
Jun 24 00:40:41 dalmesos01 kernel: [78529.040633] Task in
/docker/2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b killed
as a result of limit of
/docker/2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b
Jun 24 00:40:41 dalmesos01 kernel: [78529.040637] memory: usage 557056kB, limit
557056kB, failcnt 25928
Jun 24 00:40:41 dalmesos01 kernel: [78529.040640] memory+swap: usage 557056kB,
limit 1048576kB, failcnt 0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040642] kmem: usage 0kB, limit
18014398509481983kB, failcnt 0
Jun 24 00:40:41 dalmesos01 kernel: [78529.040644] Memory cgroup stats for
/docker/2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b:
cache:688KB rss:556368KB rss_huge:141312KB mapped_file:0KB writeback:0KB
swap:0KB inactive_anon:133676KB active_anon:423156KB inactive_file:0KB
active_file:0KB unevictable:0KB
Jun 24 00:40:41 dalmesos01 kernel: [78529.040677] [ pid ] uid tgid total_vm
rss nr_ptes swapents oom_score_adj name
Jun 24 00:40:41 dalmesos01 kernel: [78529.040798] [26845] 0 26845 1080
159 7 0 0 sh
Jun 24 00:40:41 dalmesos01 kernel: [78529.040803] [26896] 0 26896 5009
364 15 0 0 startup.sh
Jun 24 00:40:41 dalmesos01 kernel: [78529.040806] [26897] 0 26897 176126
13860 67 0 0 node
Jun 24 00:40:41 dalmesos01 kernel: [78529.040860] [30849] 0 30849 298247
64703 231 0 0 node
Jun 24 00:40:41 dalmesos01 kernel: [78529.040863] [30851] 0 30851 303829
65076 232 0 0 node
Jun 24 00:40:41 dalmesos01 kernel: [78529.040866] Memory cgroup out of memory:
Kill process 30860 (node) score 468 or sacrifice child
Jun 24 00:40:41 dalmesos01 kernel: [78529.195125] Killed process 30860 (node)
total-vm:1215316kB, anon-rss:253024kB, file-rss:7280kB
Jun 24 00:40:41 dalmesos01 kernel: [78529.349875] node invoked oom-killer:
gfp_mask=0xd0, order=0, oom_score_adj=0
Jun 24 00:40:41 dalmesos01 kernel: [78529.349878] node
cpuset=2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b
mems_allowed=0-1
Jun 24 00:40:41 dalmesos01 kernel: [78529.349881] CPU: 1 PID: 30849 Comm: node
Not tainted 3.13.0-43-generic #72-Ubuntu
Jun 24 00:40:41 dalmesos01 kernel: [78529.349882] Hardware name: Cisco Systems
Inc UCSC-C240-M3S/UCSC-C240-M3S, BIOS C240M3.1.5.7.0.042820140452 04/28/2014
Jun 24 00:40:41 dalmesos01 kernel: [78529.349884] ffff881fc7317400
ffff880f7fbc1c48 ffffffff81720bf6 ffff881fcd663000
Jun 24 00:40:41 dalmesos01 kernel: [78529.349889] ffff880f7fbc1cd0
ffffffff8171b4b1 0000000000000202 0000000000000000
Jun 24 00:40:41 dalmesos01 kernel: [78529.349893] ffff880f7fbc1c90
ffff880f7fbc1ca0 ffffffff811522a7 0000000000000001
Jun 24 00:40:41 dalmesos01 kernel: [78529.349896] Call Trace:
Jun 24 00:40:41 dalmesos01 kernel: [78529.349902] [<ffffffff81720bf6>]
dump_stack+0x45/0x56
Jun 24 00:40:41 dalmesos01 kernel: [78529.349906] [<ffffffff8171b4b1>]
dump_header+0x7f/0x1f1
Jun 24 00:40:41 dalmesos01 kernel: [78529.349909] [<ffffffff811522a7>] ?
find_lock_task_mm+0x27/0x70
Jun 24 00:40:41 dalmesos01 kernel: [78529.349912] [<ffffffff811526de>]
oom_kill_process+0x1ce/0x330
Jun 24 00:40:41 dalmesos01 kernel: [78529.349916] [<ffffffff812d6ce5>] ?
security_capable_noaudit+0x15/0x20
Jun 24 00:40:41 dalmesos01 kernel: [78529.349921] [<ffffffff811b491c>]
mem_cgroup_oom_synchronize+0x51c/0x560
Jun 24 00:40:41 dalmesos01 kernel: [78529.349924] [<ffffffff811b3e50>] ?
mem_cgroup_charge_common+0xa0/0xa0
Jun 24 00:40:41 dalmesos01 kernel: [78529.349927] [<ffffffff81152e64>]
pagefault_out_of_memory+0x14/0x80
Jun 24 00:40:41 dalmesos01 kernel: [78529.349929] [<ffffffff81719aa1>]
mm_fault_error+0x8e/0x180
Jun 24 00:40:41 dalmesos01 kernel: [78529.349932] [<ffffffff8172cf31>]
__do_page_fault+0x4a1/0x560
Jun 24 00:40:41 dalmesos01 kernel: [78529.349937] [<ffffffff8117f69d>] ?
vma_merge+0x23d/0x340
Jun 24 00:40:41 dalmesos01 kernel: [78529.349940] [<ffffffff81180774>] ?
do_brk+0x1d4/0x360
Jun 24 00:40:41 dalmesos01 kernel: [78529.349943] [<ffffffff8172d00a>]
do_page_fault+0x1a/0x70
Jun 24 00:40:41 dalmesos01 kernel: [78529.349946] [<ffffffff81729468>]
page_fault+0x28/0x30
Jun 24 00:40:41 dalmesos01 kernel: [78529.349948] Task in / killed as a result
of limit of
/docker/2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b
Jun 24 00:40:41 dalmesos01 kernel: [78529.349951] memory: usage 557056kB, limit
557056kB, failcnt 25928
Jun 24 00:40:41 dalmesos01 kernel: [78529.349952] memory+swap: usage 557056kB,
limit 1048576kB, failcnt 0
Jun 24 00:40:41 dalmesos01 kernel: [78529.349954] kmem: usage 0kB, limit
18014398509481983kB, failcnt 0
Jun 24 00:40:41 dalmesos01 kernel: [78529.349955] Memory cgroup stats for
/docker/2c58e5f20c73377fd45c81d06acf1d2ca2326e0facf4d786c6b9243176ad7c4b:
cache:688KB rss:556368KB rss_huge:141312KB mapped_file:0KB writeback:0KB
swap:0KB inactive_anon:135580KB active_anon:421380KB inactive_file:0KB
active_file:0KB unevictable:0KB
Jun 24 00:40:41 dalmesos01 kernel: [78529.349969] [ pid ] uid tgid total_vm
rss nr_ptes swapents oom_score_adj name
Jun 24 00:40:41 dalmesos01 kernel: [78529.350005] [26845] 0 26845 1080
159 7 0 0 sh
Jun 24 00:40:41 dalmesos01 kernel: [78529.350008] [26896] 0 26896 5009
364 15 0 0 startup.sh
Jun 24 00:40:41 dalmesos01 kernel: [78529.350010] [26897] 0 26897 176126
13860 67 0 0 node
Jun 24 00:40:41 dalmesos01 kernel: [78529.350023] [30849] 0 30849 298247
64703 231 0 0 node
Jun 24 00:40:41 dalmesos01 kernel: [78529.350024] Memory cgroup out of memory:
Kill process 30859 (node) score 469 or sacrifice child
Jun 24 00:40:42 dalmesos01 kernel: [78530.116248] type=1400
audit(1435106442.609:9705): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27265 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:42 dalmesos01 kernel: [78530.116265] type=1400
audit(1435106442.609:9706): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27264 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:46 dalmesos01 kernel: [78534.137584] type=1400
audit(1435106446.633:9707): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27444 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:50 dalmesos01 kernel: [78538.016569] type=1400
audit(1435106450.513:9708): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=29038 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:40:50 dalmesos01 kernel: [78538.016591] type=1400
audit(1435106450.513:9709): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=29039 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:41:09 dalmesos01 kernel: [78551.357390] ------------[ cut here
]------------
Jun 24 00:41:09 dalmesos01 kernel: [78551.357401] WARNING: CPU: 2 PID: 30861 at
/build/buildd/linux-3.13.0/kernel/watchdog.c:245
watchdog_overflow_callback+0x9c/0xd0()
Jun 24 00:41:09 dalmesos01 kernel: [78551.357403] Watchdog detected hard LOCKUP
on cpu 2
Jun 24 00:41:09 dalmesos01 kernel: [78551.357405] Modules linked in: veth
xt_nat xt_tcpudp xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables
nf_nat nf_conntrack bridge stp llc nfsv3 nfsd auth_rpcgss nfs_acl aufs nfs
lockd sunrpc fscache gpio_ich joydev intel_rapl x86_pkg_temp_thermal
intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd lpc_ich
sb_edac edac_core ipmi_si wmi acpi_power_meter mac_hid lp parport xfs libcrc32c
dm_round_robin ses dm_multipath enclosure hid_generic scsi_dh fnic libfcoe
libfc megaraid_sas usbhid scsi_transport_fc hid enic scsi_tgt
Jun 24 00:41:09 dalmesos01 kernel: [78551.357457] CPU: 2 PID: 30861 Comm: node
Not tainted 3.13.0-43-generic #72-Ubuntu
Jun 24 00:41:09 dalmesos01 kernel: [78551.357458] Hardware name: Cisco Systems
Inc UCSC-C240-M3S/UCSC-C240-M3S, BIOS C240M3.1.5.7.0.042820140452 04/28/2014
Jun 24 00:41:09 dalmesos01 kernel: [78551.357460] 0000000000000009
ffff88103fe45c20 ffffffff81720bf6 ffff88103fe45c68
Jun 24 00:41:09 dalmesos01 kernel: [78551.357465] ffff88103fe45c58
ffffffff810677cd ffff8810293a8000 0000000000000000
Jun 24 00:41:09 dalmesos01 kernel: [78551.357469] ffff88103fe45d88
0000000000000000 ffff88103fe45ef8 ffff88103fe45cb8
Jun 24 00:41:09 dalmesos01 kernel: [78551.357473] Call Trace:
Jun 24 00:41:09 dalmesos01 kernel: [78551.357474] <NMI> [<ffffffff81720bf6>]
dump_stack+0x45/0x56
Jun 24 00:41:09 dalmesos01 kernel: [78551.357484] [<ffffffff810677cd>]
warn_slowpath_common+0x7d/0xa0
Jun 24 00:41:09 dalmesos01 kernel: [78551.357486] [<ffffffff8106783c>]
warn_slowpath_fmt+0x4c/0x50
Jun 24 00:41:09 dalmesos01 kernel: [78551.357492] [<ffffffff810bbb0c>] ?
print_time.part.8+0x6c/0x90
Jun 24 00:41:09 dalmesos01 kernel: [78551.357495] [<ffffffff8110dac0>] ?
restart_watchdog_hrtimer+0x50/0x50
Jun 24 00:41:09 dalmesos01 kernel: [78551.357499] [<ffffffff8110db5c>]
watchdog_overflow_callback+0x9c/0xd0
Jun 24 00:41:09 dalmesos01 kernel: [78551.357503] [<ffffffff8114581e>]
__perf_event_overflow+0x8e/0x240
Jun 24 00:41:09 dalmesos01 kernel: [78551.357508] [<ffffffff81029248>] ?
x86_perf_event_set_period+0xe8/0x150
Jun 24 00:41:09 dalmesos01 kernel: [78551.357511] [<ffffffff81146334>]
perf_event_overflow+0x14/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357515] [<ffffffff810306ad>]
intel_pmu_handle_irq+0x1ed/0x3f0
Jun 24 00:41:09 dalmesos01 kernel: [78551.357519] [<ffffffff81188411>] ?
unmap_kernel_range_noflush+0x11/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357524] [<ffffffff8172ab5b>]
perf_event_nmi_handler+0x2b/0x50
Jun 24 00:41:09 dalmesos01 kernel: [78551.357527] [<ffffffff8172a378>]
nmi_handle.isra.3+0x88/0x180
Jun 24 00:41:09 dalmesos01 kernel: [78551.357529] [<ffffffff8172a5d9>]
do_nmi+0x169/0x340
Jun 24 00:41:09 dalmesos01 kernel: [78551.357532] [<ffffffff817297e1>]
end_repeat_nmi+0x1e/0x2e
Jun 24 00:41:09 dalmesos01 kernel: [78551.357538] [<ffffffff81371d43>] ?
__write_lock_failed+0x13/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357541] [<ffffffff81371d43>] ?
__write_lock_failed+0x13/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357544] [<ffffffff81371d43>] ?
__write_lock_failed+0x13/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357545] <<EOE>>
[<ffffffff8172895e>] _raw_write_lock_irq+0x1e/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357550] [<ffffffff81065dae>]
copy_process.part.26+0xc3e/0x16b0
Jun 24 00:41:09 dalmesos01 kernel: [78551.357553] [<ffffffff810669f5>]
do_fork+0xd5/0x340
Jun 24 00:41:09 dalmesos01 kernel: [78551.357555] [<ffffffff81066ce6>]
SyS_clone+0x16/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78551.357559] [<ffffffff81731a09>]
stub_clone+0x69/0x90
Jun 24 00:41:09 dalmesos01 kernel: [78551.357562] [<ffffffff817316ad>] ?
system_call_fastpath+0x1a/0x1f
Jun 24 00:41:09 dalmesos01 kernel: [78551.357564] ---[ end trace
01146e4ab621d97d ]---
Jun 24 00:41:09 dalmesos01 kernel: [78557.209385] BUG: soft lockup - CPU#1
stuck for 23s! [node:30849]
Jun 24 00:41:09 dalmesos01 kernel: [78557.286474] Modules linked in: veth
xt_nat xt_tcpudp xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables x_tables
nf_nat nf_conntrack bridge stp llc nfsv3 nfsd auth_rpcgss nfs_acl aufs nfs
lockd sunrpc fscache gpio_ich joydev intel_rapl x86_pkg_temp_thermal
intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd lpc_ich
sb_edac edac_core ipmi_si wmi acpi_power_meter mac_hid lp parport xfs libcrc32c
dm_round_robin ses dm_multipath enclosure hid_generic scsi_dh fnic libfcoe
libfc megaraid_sas usbhid scsi_transport_fc hid enic scsi_tgt
Jun 24 00:41:09 dalmesos01 kernel: [78557.286526] CPU: 1 PID: 30849 Comm: node
Tainted: G W 3.13.0-43-generic #72-Ubuntu
Jun 24 00:41:09 dalmesos01 kernel: [78557.286528] Hardware name: Cisco Systems
Inc UCSC-C240-M3S/UCSC-C240-M3S, BIOS C240M3.1.5.7.0.042820140452 04/28/2014
Jun 24 00:41:09 dalmesos01 kernel: [78557.286530] task: ffff881fcd663000 ti:
ffff880f7fbc0000 task.ti: ffff880f7fbc0000
Jun 24 00:41:09 dalmesos01 kernel: [78557.286531] RIP:
0010:[<ffffffff811525f9>] [<ffffffff811525f9>] oom_kill_process+0xe9/0x330
Jun 24 00:41:09 dalmesos01 kernel: [78557.286538] RSP: 0000:ffff880f7fbc1ce0
EFLAGS: 00000206
Jun 24 00:41:09 dalmesos01 kernel: [78557.286539] RAX: ffff881fe96d6310 RBX:
ffff88202633b650 RCX: 0000000000000006
Jun 24 00:41:09 dalmesos01 kernel: [78557.286541] RDX: 0000000000000007 RSI:
ffff881fe96d6000 RDI: ffffffff81c07040
Jun 24 00:41:09 dalmesos01 kernel: [78557.286542] RBP: ffff880f7fbc1d30 R08:
0000000000000086 R09: ffff881fc7317400
Jun 24 00:41:09 dalmesos01 kernel: [78557.286544] R10: 0000000000000050 R11:
ffffc9001d02bff8 R12: ffff88202633b000
Jun 24 00:41:09 dalmesos01 kernel: [78557.286545] R13: 0000000000000246 R14:
0000000000000000 R15: ffffffff81a63e20
Jun 24 00:41:09 dalmesos01 kernel: [78557.286547] FS: 00007f3e61a26740(0000)
GS:ffff88103fe20000(0000) knlGS:0000000000000000
Jun 24 00:41:09 dalmesos01 kernel: [78557.286549] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033
Jun 24 00:41:09 dalmesos01 kernel: [78557.286550] CR2: 0000000008d98d38 CR3:
000000200ed02000 CR4: 00000000000407e0
Jun 24 00:41:09 dalmesos01 kernel: [78557.286552] Stack:
Jun 24 00:41:09 dalmesos01 kernel: [78557.286553] ffff881fe96d6000
ffffffff812d6ce5 ffff88202633b000 ffff881fc7317400
Jun 24 00:41:09 dalmesos01 kernel: [78557.286558] ffff881fe96d63a0
ffff881fc7317400 0000000000022000 ffff88202633b000
Jun 24 00:41:09 dalmesos01 kernel: [78557.286562] 0000000000000000
0000000000000000 ffff880f7fbc1dd0 ffffffff811b491c
Jun 24 00:41:09 dalmesos01 kernel: [78557.286565] Call Trace:
Jun 24 00:41:09 dalmesos01 kernel: [78557.286572] [<ffffffff812d6ce5>] ?
security_capable_noaudit+0x15/0x20
Jun 24 00:41:09 dalmesos01 kernel: [78557.286578] [<ffffffff811b491c>]
mem_cgroup_oom_synchronize+0x51c/0x560
Jun 24 00:41:09 dalmesos01 kernel: [78557.286582] [<ffffffff811b3e50>] ?
mem_cgroup_charge_common+0xa0/0xa0
Jun 24 00:41:09 dalmesos01 kernel: [78557.286585] [<ffffffff81152e64>]
pagefault_out_of_memory+0x14/0x80
Jun 24 00:41:09 dalmesos01 kernel: [78557.286589] [<ffffffff81719aa1>]
mm_fault_error+0x8e/0x180
Jun 24 00:41:09 dalmesos01 kernel: [78557.286594] [<ffffffff8172cf31>]
__do_page_fault+0x4a1/0x560
Jun 24 00:41:09 dalmesos01 kernel: [78557.286599] [<ffffffff8117f69d>] ?
vma_merge+0x23d/0x340
Jun 24 00:41:09 dalmesos01 kernel: [78557.286602] [<ffffffff81180774>] ?
do_brk+0x1d4/0x360
Jun 24 00:41:09 dalmesos01 kernel: [78557.286605] [<ffffffff8172d00a>]
do_page_fault+0x1a/0x70
Jun 24 00:41:09 dalmesos01 kernel: [78557.286609] [<ffffffff81729468>]
page_fault+0x28/0x30
Jun 24 00:41:09 dalmesos01 kernel: [78557.286610] Code: 45 31 ed e8 1a 65 5d 00
48 c7 c7 40 70 c0 81 e8 ae 62 5d 00 4c 8b 4d c8 48 89 5d b0 48 89 5d c0 48 8b
75 b0 48 8b 86 10 03 00 00 <4c> 8d a6 10 03 00 00 49 39 c4 4c 8d b8 e0 fc ff ff
75 17 eb 6b
Jun 24 00:41:10 dalmesos01 kernel: [78557.719642] type=1400
audit(1435106470.239:9710): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27130 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:41:10 dalmesos01 kernel: [78557.954570] type=1400
audit(1435106470.475:9711): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27128 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:41:11 dalmesos01 kernel: [78558.957417] type=1400
audit(1435106471.479:9712): apparmor="DENIED" operation="ptrace"
profile="docker-default" pid=27060 comm="node" requested_mask="trace"
denied_mask="trace" peer="docker-default"
Jun 24 00:41:11 dalmesos01 kernel: [78558.957418] audit: audit_lost=331
audit_rate_limit=0 audit_backlog_limit=64
Jun 24 00:41:11 dalmesos01 kernel: [78558.957421] audit: printk limit exceeded
