Hmm, oh bugger, it seems the kernel errors are not the issue. The question
remains what is. I still see the errors every few minutes. The errors make
reference to GELF, and I only have one GELF tcp input from 2 Windows boxes
running NXlog. The errors seem to have started with the last graylog-server
update.
On Friday, May 1, 2015 at 4:29:49 PM UTC-7, Mark Moorcroft wrote:
>
>
> This morning I was seeing bunches of errors in the server.log. I think I
> tracked them to a syslog/tcp input. My rsyslog entry on the client is as
> follows.
>
> # Graylog
> $template GRAYLOGRFC5424,"<%PRI%>%PROTOCOL-VERSION%
> %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID%
> %STRUCTURED-DATA% %msg%\n"
> *.* @@xxx.xxx.xxx.xxx:12204;GRAYLOGRFC5424
>
> It seems the cause was memory errors on a compute node. The question is if
> this is a graylog bug or expected behavior. There were a series of these
> "com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'xxx':
> was expecting ('true', 'false' or 'null') at [Source:" errors. I'm running
> the current versions of graylog-server and elasticsearch. The token 'xxx'
> is a random character(s) and then a massive bunch of garbage characters
> will follow the error.
>
> From /var/log/messages:
>
> May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ?
> nv_rx_process_optimized+0x126/0x2a0 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
> scheduler_tick+0xcc/0x260
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
> free_pcppages_bulk+0x392/0x460
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
> handle_IRQ_event+0x60/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ?
> do_IRQ+0x75/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
> ret_from_intr+0x0/0x11
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ?
> submit_bh+0x60/0x1f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ?
> __block_write_full_page+0x1c8/0x330
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ?
> end_buffer_async_write+0x0/0x190
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ?
> block_write_full_page_endio+0xe0/0x120
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ?
> buffer_unmapped+0x0/0x20 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ?
> block_write_full_page+0x15/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ?
> ext3_ordered_writepage+0x1ed/0x240 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ?
> __writepage+0x17/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ?
> write_cache_pages+0x1fd/0x4c0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ?
> __ext4_journal_stop+0x68/0xa0 [ext4]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ?
> __writepage+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ?
> generic_writepages+0x24/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ?
> do_writepages+0x35/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ?
> writeback_single_inode+0xdd/0x290
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ?
> writeback_sb_inodes+0xbd/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ?
> writeback_inodes_wb+0xab/0x1b0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ?
> wb_writeback+0x2f3/0x410
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ?
> thread_return+0x4e/0x76e
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ?
> del_timer_sync+0x22/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ?
> wb_do_writeback+0xbb/0x240
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ?
> bdi_writeback_task+0x63/0x1b0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ?
> bit_waitqueue+0x17/0xd0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
> bdi_start_fn+0x0/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ?
> bdi_start_fn+0x86/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
> bdi_start_fn+0x0/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ?
> kthread+0x96/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ?
> child_rip+0xa/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ?
> kthread+0x0/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ?
> child_rip+0x0/0x20
> May 1 13:08:56 compute-0-21 kernel: flush-8:0: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 444, comm: flush-8:0 Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025bc76>] ?
> nv_rx_process_optimized+0x126/0x2a0 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
> scheduler_tick+0xcc/0x260
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
> free_pcppages_bulk+0x392/0x460
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
> handle_IRQ_event+0x60/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ?
> do_IRQ+0x75/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
> ret_from_intr+0x0/0x11
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff811bdd20>] ?
> submit_bh+0x60/0x1f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0598>] ?
> __block_write_full_page+0x1c8/0x330
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bf560>] ?
> end_buffer_async_write+0x0/0x190
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c07e0>] ?
> block_write_full_page_endio+0xe0/0x120
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c4b30>] ?
> buffer_unmapped+0x0/0x20 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0835>] ?
> block_write_full_page+0x15/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c56dd>] ?
> ext3_ordered_writepage+0x1ed/0x240 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336c7>] ?
> __writepage+0x17/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113498d>] ?
> write_cache_pages+0x1fd/0x4c0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa0203e28>] ?
> __ext4_journal_stop+0x68/0xa0 [ext4]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811336b0>] ?
> __writepage+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134c74>] ?
> generic_writepages+0x24/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81134cb5>] ?
> do_writepages+0x35/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b50cd>] ?
> writeback_single_inode+0xdd/0x290
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b54cd>] ?
> writeback_sb_inodes+0xbd/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b562b>] ?
> writeback_inodes_wb+0xab/0x1b0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5a23>] ?
> wb_writeback+0x2f3/0x410
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81527f30>] ?
> thread_return+0x4e/0x76e
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81084d92>] ?
> del_timer_sync+0x22/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5bfb>] ?
> wb_do_writeback+0xbb/0x240
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811b5de3>] ?
> bdi_writeback_task+0x63/0x1b0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b117>] ?
> bit_waitqueue+0x17/0xd0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
> bdi_start_fn+0x0/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81143a26>] ?
> bdi_start_fn+0x86/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811439a0>] ?
> bdi_start_fn+0x0/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109aee6>] ?
> kthread+0x96/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c20a>] ?
> child_rip+0xa/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109ae50>] ?
> kthread+0x0/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c200>] ?
> child_rip+0x0/0x20
> May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ?
> check_preempt_wakeup+0x1c0/0x260
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105572c>] ?
> check_preempt_curr+0x7c/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025a08b>] ?
> nv_tx_done_optimized+0x5b/0x200 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105dd5c>] ?
> scheduler_tick+0xcc/0x260
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112eef2>] ?
> free_pcppages_bulk+0x392/0x460
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
> handle_IRQ_event+0x60/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ?
> do_IRQ+0x75/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
> ret_from_intr+0x0/0x11
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ?
> ext3_get_branch+0xed/0x140 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ?
> ext3_get_branch+0x98/0x140 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ?
> ext3_get_blocks_handle+0xdf/0xe20 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ?
> zone_statistics+0x99/0xc0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ?
> alloc_buffer_head+0x49/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ?
> alloc_page_buffers+0x86/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ?
> ext3_get_block+0xc7/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ?
> __block_prepare_write+0x1d3/0x530
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ?
> block_write_begin_newtrunc+0x5c/0xd0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ?
> block_write_begin+0x43/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ?
> ext3_write_begin+0x20b/0x2c0 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
> generic_file_buffered_write+0x123/0x2e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
> __generic_file_aio_write+0x260/0x490
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
> generic_file_aio_write+0x88/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
> do_sync_write+0xfa/0x140
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
> autoremove_wake_function+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
> security_file_permission+0x16/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
> vfs_write+0xb8/0x1a0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
> sys_write+0x51/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
> May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8105a360>] ?
> check_preempt_wakeup+0x1c0/0x260
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81043780>] ?
> gart_map_page+0x0/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a758>] ?
> irq_exit+0x48/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ?
> do_IRQ+0x75/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810e6eb0>] ?
> handle_IRQ_event+0x60/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81531605>] ?
> do_IRQ+0x75/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b9d3>] ?
> ret_from_intr+0x0/0x11
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffffa02c51dd>] ?
> ext3_get_branch+0xed/0x140 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c5188>] ?
> ext3_get_branch+0x98/0x140 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c721f>] ?
> ext3_get_blocks_handle+0xdf/0xe20 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811422c9>] ?
> zone_statistics+0x99/0xc0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811bd919>] ?
> alloc_buffer_head+0x49/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811be416>] ?
> alloc_page_buffers+0x86/0xf0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c8027>] ?
> ext3_get_block+0xc7/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c0ed3>] ?
> __block_prepare_write+0x1d3/0x530
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c153c>] ?
> block_write_begin_newtrunc+0x5c/0xd0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811c1943>] ?
> block_write_begin+0x43/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c999b>] ?
> ext3_write_begin+0x20b/0x2c0 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c7f60>] ?
> ext3_get_block+0x0/0x120 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
> generic_file_buffered_write+0x123/0x2e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
> __generic_file_aio_write+0x260/0x490
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
> generic_file_aio_write+0x88/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
> do_sync_write+0xfa/0x140
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
> autoremove_wake_function+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
> security_file_permission+0x16/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
> vfs_write+0xb8/0x1a0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
> sys_write+0x51/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
> May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
> tick_program_event+0x2a/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
> tick_program_event+0x2a/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ?
> smp_apic_timer_interrupt+0x4a/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ?
> apic_timer_interrupt+0x13/0x20
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff81138fe7>] ?
> shrink_page_list.clone.3+0xf7/0x650
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fe0>] ?
> shrink_page_list.clone.3+0xf0/0x650
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ?
> mem_cgroup_lru_del_list+0x2b/0xb0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ?
> isolate_lru_pages.clone.0+0xd7/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ?
> wait_iff_congested+0x6d/0xe0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ?
> shrink_inactive_list+0x343/0x830
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ?
> shrink_mem_cgroup_zone+0x3ae/0x610
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ?
> mem_cgroup_iter+0xfd/0x280
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ?
> shrink_zone+0x63/0xb0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ?
> do_try_to_free_pages+0x115/0x610
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ?
> get_page_from_freelist+0x15c/0x870
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ?
> try_to_free_pages+0x92/0x120
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ?
> __alloc_pages_nodemask+0x47e/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ?
> alloc_pages_current+0xaa/0x110
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ?
> __page_cache_alloc+0x87/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ?
> grab_cache_page_write_begin+0x8e/0xc0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ?
> ext3_write_begin+0x8f/0x2c0 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
> unlock_page+0x27/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
> generic_file_buffered_write+0x123/0x2e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
> __generic_file_aio_write+0x260/0x490
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
> generic_file_aio_write+0x88/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
> do_sync_write+0xfa/0x140
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
> autoremove_wake_function+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
> security_file_permission+0x16/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
> vfs_write+0xb8/0x1a0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
> sys_write+0x51/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
> May 1 13:08:56 compute-0-21 kernel: l906.exe: page allocation failure.
> order:2, mode:0x20
> May 1 13:08:56 compute-0-21 kernel: Pid: 1390, comm: l906.exe Not tainted
> 2.6.32-431.11.2.el6.x86_64 #1
> May 1 13:08:56 compute-0-21 kernel: Call Trace:
> May 1 13:08:56 compute-0-21 kernel: <IRQ> [<ffffffff8112f9da>] ?
> __alloc_pages_nodemask+0x74a/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
> tick_program_event+0x2a/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116e492>] ?
> kmem_getpages+0x62/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116f0aa>] ?
> fallback_alloc+0x1ba/0x270
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116eaff>] ?
> cache_grow+0x2cf/0x320
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116ee29>] ?
> ____cache_alloc_node+0x99/0x160
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8116fff0>] ?
> kmem_cache_alloc_node_trace+0x90/0x200
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117020d>] ?
> __kmalloc_node+0x4d/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145033a>] ?
> __alloc_skb+0x7a/0x180
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8145090d>] ?
> dev_alloc_skb+0x1d/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025c728>] ?
> nv_alloc_rx_optimized+0x198/0x270 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa025d80c>] ?
> nv_napi_poll+0x8c/0x610 [forcedeth]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81460fb3>] ?
> net_rx_action+0x103/0x2f0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a8e1>] ?
> __do_softirq+0xc1/0x1e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff810ac9ba>] ?
> tick_program_event+0x2a/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100c30c>] ?
> call_softirq+0x1c/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100fa75>] ?
> do_softirq+0x65/0xa0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8107a795>] ?
> irq_exit+0x85/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff815316ca>] ?
> smp_apic_timer_interrupt+0x4a/0x60
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100bb93>] ?
> apic_timer_interrupt+0x13/0x20
> May 1 13:08:56 compute-0-21 kernel: <EOI> [<ffffffff8109b1d4>] ?
> __wake_up_bit+0x4/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
> unlock_page+0x27/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81138fb0>] ?
> shrink_page_list.clone.3+0xc0/0x650
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117834b>] ?
> mem_cgroup_lru_del_list+0x2b/0xb0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811397d7>] ?
> isolate_lru_pages.clone.0+0xd7/0x170
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81142b5d>] ?
> wait_iff_congested+0x6d/0xe0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81139f23>] ?
> shrink_inactive_list+0x343/0x830
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113a7be>] ?
> shrink_mem_cgroup_zone+0x3ae/0x610
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8117a01d>] ?
> mem_cgroup_iter+0xfd/0x280
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113aa83>] ?
> shrink_zone+0x63/0xb0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113abe5>] ?
> do_try_to_free_pages+0x115/0x610
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112d2dc>] ?
> get_page_from_freelist+0x15c/0x870
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8113b2b2>] ?
> try_to_free_pages+0x92/0x120
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811411c0>] ?
> next_zone+0x30/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8112f70e>] ?
> __alloc_pages_nodemask+0x47e/0x8d0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81167aaa>] ?
> alloc_pages_current+0xaa/0x110
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fd47>] ?
> __page_cache_alloc+0x87/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81120a5e>] ?
> grab_cache_page_write_begin+0x8e/0xc0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffffa02c981f>] ?
> ext3_write_begin+0x8f/0x2c0 [ext3]
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8111fa37>] ?
> unlock_page+0x27/0x30
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff811202c3>] ?
> generic_file_buffered_write+0x123/0x2e0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121d20>] ?
> __generic_file_aio_write+0x260/0x490
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81121fd8>] ?
> generic_file_aio_write+0x88/0x100
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81188d4a>] ?
> do_sync_write+0xfa/0x140
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8109b290>] ?
> autoremove_wake_function+0x0/0x40
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81226496>] ?
> security_file_permission+0x16/0x20
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189048>] ?
> vfs_write+0xb8/0x1a0
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff81189941>] ?
> sys_write+0x51/0x90
> May 1 13:08:56 compute-0-21 kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
