This morning I was seeing bunches of errors in the server.log. I think I
tracked them to a syslog/tcp input. My rsyslog entry on the client is as
follows.
# Graylog
$template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION%
%TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID%
%STRUCTURED-DATA% %msg%\n"
*.* @@xxx.xxx.xxx.xxx:12204;GRAYLOGRFC5424
It seems the cause was memory errors on a compute node. The question is if
this is a graylog bug or expected behavior. There were a series of these
"com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'xxx':
was expecting ('true', 'false' or 'null') at [Source:" errors. I'm running
the current versions of graylog-server and elasticsearch. The token 'xxx'
is a random character(s) and then a massive bunch of garbage characters
will follow the error.
>From /var/log/messages:
May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ?
nv_rx_process_optimized+0x126/0x2a0 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
scheduler_tick+0xcc/0x260
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
free_pcppages_bulk+0x392/0x460
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
handle_IRQ_event+0x60/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? do_IRQ+0x75/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
ret_from_intr+0x0/0x11
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ?
submit_bh+0x60/0x1f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ?
__block_write_full_page+0x1c8/0x330
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ?
end_buffer_async_write+0x0/0x190
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ?
block_write_full_page_endio+0xe0/0x120
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ?
buffer_unmapped+0x0/0x20 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ?
block_write_full_page+0x15/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ?
ext3_ordered_writepage+0x1ed/0x240 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ?
__writepage+0x17/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ?
write_cache_pages+0x1fd/0x4c0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ?
__ext4_journal_stop+0x68/0xa0 [ext4]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ?
__writepage+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ?
generic_writepages+0x24/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ?
do_writepages+0x35/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ?
writeback_single_inode+0xdd/0x290
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ?
writeback_sb_inodes+0xbd/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ?
writeback_inodes_wb+0xab/0x1b0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ?
wb_writeback+0x2f3/0x410
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ?
thread_return+0x4e/0x76e
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ?
del_timer_sync+0x22/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ?
wb_do_writeback+0xbb/0x240
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ?
bdi_writeback_task+0x63/0x1b0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ?
bit_waitqueue+0x17/0xd0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
bdi_start_fn+0x0/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ?
bdi_start_fn+0x86/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
bdi_start_fn+0x0/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ?
kthread+0x96/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ?
child_rip+0xa/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ?
child_rip+0x0/0x20
May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ?
nv_rx_process_optimized+0x126/0x2a0 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
scheduler_tick+0xcc/0x260
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
free_pcppages_bulk+0x392/0x460
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
handle_IRQ_event+0x60/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? do_IRQ+0x75/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
ret_from_intr+0x0/0x11
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ?
submit_bh+0x60/0x1f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ?
__block_write_full_page+0x1c8/0x330
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ?
end_buffer_async_write+0x0/0x190
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ?
block_write_full_page_endio+0xe0/0x120
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ?
buffer_unmapped+0x0/0x20 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ?
block_write_full_page+0x15/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ?
ext3_ordered_writepage+0x1ed/0x240 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ?
__writepage+0x17/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ?
write_cache_pages+0x1fd/0x4c0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ?
__ext4_journal_stop+0x68/0xa0 [ext4]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ?
__writepage+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ?
generic_writepages+0x24/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ?
do_writepages+0x35/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ?
writeback_single_inode+0xdd/0x290
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ?
writeback_sb_inodes+0xbd/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ?
writeback_inodes_wb+0xab/0x1b0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ?
wb_writeback+0x2f3/0x410
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ?
thread_return+0x4e/0x76e
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ?
del_timer_sync+0x22/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ?
wb_do_writeback+0xbb/0x240
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ?
bdi_writeback_task+0x63/0x1b0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ?
bit_waitqueue+0x17/0xd0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
bdi_start_fn+0x0/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ?
bdi_start_fn+0x86/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
bdi_start_fn+0x0/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ?
kthread+0x96/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ?
child_rip+0xa/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ? kthread+0x0/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ?
child_rip+0x0/0x20
May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ?
check_preempt_wakeup+0x1c0/0x260
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105572c>] ?
check_preempt_curr+0x7c/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025a08b>] ?
nv_tx_done_optimized+0x5b/0x200 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
scheduler_tick+0xcc/0x260
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
free_pcppages_bulk+0x392/0x460
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
handle_IRQ_event+0x60/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? do_IRQ+0x75/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
ret_from_intr+0x0/0x11
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ?
ext3_get_branch+0xed/0x140 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ?
ext3_get_branch+0x98/0x140 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ?
ext3_get_blocks_handle+0xdf/0xe20 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ?
zone_statistics+0x99/0xc0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ?
alloc_buffer_head+0x49/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ?
alloc_page_buffers+0x86/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ?
ext3_get_block+0xc7/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ?
__block_prepare_write+0x1d3/0x530
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ?
block_write_begin_newtrunc+0x5c/0xd0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ?
block_write_begin+0x43/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ?
ext3_write_begin+0x20b/0x2c0 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
generic_file_buffered_write+0x123/0x2e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
__generic_file_aio_write+0x260/0x490
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
generic_file_aio_write+0x88/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
do_sync_write+0xfa/0x140
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
autoremove_wake_function+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
security_file_permission+0x16/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
vfs_write+0xb8/0x1a0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
sys_write+0x51/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ?
check_preempt_wakeup+0x1c0/0x260
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81043780>] ?
gart_map_page+0x0/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a758>] ?
irq_exit+0x48/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? do_IRQ+0x75/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
handle_IRQ_event+0x60/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ? do_IRQ+0x75/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
ret_from_intr+0x0/0x11
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ?
ext3_get_branch+0xed/0x140 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ?
ext3_get_branch+0x98/0x140 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ?
ext3_get_blocks_handle+0xdf/0xe20 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ?
zone_statistics+0x99/0xc0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ?
alloc_buffer_head+0x49/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ?
alloc_page_buffers+0x86/0xf0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ?
ext3_get_block+0xc7/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ?
__block_prepare_write+0x1d3/0x530
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ?
block_write_begin_newtrunc+0x5c/0xd0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ?
block_write_begin+0x43/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ?
ext3_write_begin+0x20b/0x2c0 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
ext3_get_block+0x0/0x120 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
generic_file_buffered_write+0x123/0x2e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
__generic_file_aio_write+0x260/0x490
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
generic_file_aio_write+0x88/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
do_sync_write+0xfa/0x140
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
autoremove_wake_function+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
security_file_permission+0x16/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
vfs_write+0xb8/0x1a0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
sys_write+0x51/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
tick_program_event+0x2a/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
tick_program_event+0x2a/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ?
smp_apic_timer_interrupt+0x4a/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ?
apic_timer_interrupt+0x13/0x20
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff81138fe7>] ?
shrink_page_list.clone.3+0xf7/0x650
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fe0>] ?
shrink_page_list.clone.3+0xf0/0x650
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ?
mem_cgroup_lru_del_list+0x2b/0xb0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ?
isolate_lru_pages.clone.0+0xd7/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ?
wait_iff_congested+0x6d/0xe0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ?
shrink_inactive_list+0x343/0x830
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ?
shrink_mem_cgroup_zone+0x3ae/0x610
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ?
mem_cgroup_iter+0xfd/0x280
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ?
shrink_zone+0x63/0xb0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ?
do_try_to_free_pages+0x115/0x610
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ?
get_page_from_freelist+0x15c/0x870
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ?
try_to_free_pages+0x92/0x120
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ?
__alloc_pages_nodemask+0x47e/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ?
alloc_pages_current+0xaa/0x110
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ?
__page_cache_alloc+0x87/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ?
grab_cache_page_write_begin+0x8e/0xc0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ?
ext3_write_begin+0x8f/0x2c0 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
unlock_page+0x27/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
generic_file_buffered_write+0x123/0x2e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
__generic_file_aio_write+0x260/0x490
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
generic_file_aio_write+0x88/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
do_sync_write+0xfa/0x140
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
autoremove_wake_function+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
security_file_permission+0x16/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
vfs_write+0xb8/0x1a0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
sys_write+0x51/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
order:2, mode:0x20
May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
2.6.32-431.11.2.el6.x86_64 #1
May 1 13:08:56 compute-0-21 kernel: Call Trace:
May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
__alloc_pages_nodemask+0x74a/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
tick_program_event+0x2a/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
kmem_getpages+0x62/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
fallback_alloc+0x1ba/0x270
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
cache_grow+0x2cf/0x320
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
____cache_alloc_node+0x99/0x160
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
kmem_cache_alloc_node_trace+0x90/0x200
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
__kmalloc_node+0x4d/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
__alloc_skb+0x7a/0x180
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
dev_alloc_skb+0x1d/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
nv_napi_poll+0x8c/0x610 [forcedeth]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
net_rx_action+0x103/0x2f0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
__do_softirq+0xc1/0x1e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
tick_program_event+0x2a/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
call_softirq+0x1c/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
do_softirq+0x65/0xa0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
irq_exit+0x85/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ?
smp_apic_timer_interrupt+0x4a/0x60
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ?
apic_timer_interrupt+0x13/0x20
May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff8109b1d4>] ?
__wake_up_bit+0x4/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
unlock_page+0x27/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fb0>] ?
shrink_page_list.clone.3+0xc0/0x650
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ?
mem_cgroup_lru_del_list+0x2b/0xb0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ?
isolate_lru_pages.clone.0+0xd7/0x170
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ?
wait_iff_congested+0x6d/0xe0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ?
shrink_inactive_list+0x343/0x830
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ?
shrink_mem_cgroup_zone+0x3ae/0x610
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ?
mem_cgroup_iter+0xfd/0x280
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ?
shrink_zone+0x63/0xb0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ?
do_try_to_free_pages+0x115/0x610
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ?
get_page_from_freelist+0x15c/0x870
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ?
try_to_free_pages+0x92/0x120
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811411c0>] ?
next_zone+0x30/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ?
__alloc_pages_nodemask+0x47e/0x8d0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ?
alloc_pages_current+0xaa/0x110
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ?
__page_cache_alloc+0x87/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ?
grab_cache_page_write_begin+0x8e/0xc0
May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ?
ext3_write_begin+0x8f/0x2c0 [ext3]
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
unlock_page+0x27/0x30
May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
generic_file_buffered_write+0x123/0x2e0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
__generic_file_aio_write+0x260/0x490
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
generic_file_aio_write+0x88/0x100
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
do_sync_write+0xfa/0x140
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
autoremove_wake_function+0x0/0x40
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
security_file_permission+0x16/0x20
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
vfs_write+0xb8/0x1a0
May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
sys_write+0x51/0x90
May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
