Hi Niels,
4 out of 5 Rackspace slave VM's hung overnight. Rebooted one of
them and it didn't come back. Checking out it's console (they
have an in-browser Java applet for it) showed a kernel traceback.
Scrollback for the console (took some effort ;>) is attached.
It's showing a bunch of XFS messages from the system shutdown,
but then (down a page or two) a traceback starts.
Does this look like a potential XFS bug, or is it more hinting
something else is wrong? eg improper cleanup script, or we
need kernel settings adjusted, or ?
+ Justin
XFS (dm-16): Mounting Filesystem
XFS (dm-17): Mounting Filesystem
XFS (dm-16): Starting recovery (logdev: internal)
XFS (dm-16): Ending recovery (logdev: internal)
XFS (dm-17): Starting recovery (logdev: internal)
XFS (dm-17): Ending recovery (logdev: internal)
XFS (dm-4): Mounting Filesystem
XFS (dm-4): Ending clean mount
XFS (dm-5): Mounting Filesystem
XFS (dm-5): Starting recovery (logdev: internal)
XFS (dm-5): Ending recovery (logdev: internal)
XFS (dm-6): Mounting Filesystem
XFS (dm-6): Starting recovery (logdev: internal)
XFS (dm-6): Ending recovery (logdev: internal)
XFS (dm-7): Mounting Filesystem
XFS (dm-7): Starting recovery (logdev: internal)
XFS (dm-7): Ending recovery (logdev: internal)
XFS (dm-4): Mounting Filesystem
XFS (dm-4): Ending clean mount
XFS (dm-9): Mounting Filesystem
XFS (dm-9): Ending clean mount
XFS (dm-10): Mounting Filesystem
XFS (dm-10): Starting recovery (logdev: internal)
XFS (dm-10): Ending recovery (logdev: internal)
XFS (dm-11): Mounting Filesystem
XFS (dm-11): Starting recovery (logdev: internal)
XFS (dm-11): Ending recovery (logdev: internal)
XFS (dm-12): Mounting Filesystem
XFS (dm-12): Starting recovery (logdev: internal)
XFS (dm-12): Ending recovery (logdev: internal)
XFS (dm-13): Mounting Filesystem
XFS (dm-13): Starting recovery (logdev: internal)
XFS (dm-13): Ending recovery (logdev: internal)
XFS (dm-14): Mounting Filesystem
XFS (dm-14): Starting recovery (logdev: internal)
XFS (dm-14): Ending recovery (logdev: internal)
XFS (dm-15): Mounting Filesystem
XFS (dm-15): Starting recovery (logdev: internal)
XFS (dm-15): Ending recovery (logdev: internal)
XFS (dm-16): Mounting Filesystem
XFS (dm-16): Starting recovery (logdev: internal)
XFS (dm-16): Ending recovery (logdev: internal)
XFS (dm-17): Mounting Filesystem
XFS (dm-17): Starting recovery (logdev: internal)
XFS (dm-17): Ending recovery (logdev: internal)
XFS (dm-10): Mounting Filesystem
XFS (dm-10): Starting recovery (logdev: internal)
XFS (dm-10): Ending recovery (logdev: internal)
XFS (dm-11): Mounting Filesystem
XFS (dm-11): Starting recovery (logdev: internal)
XFS (dm-11): Ending recovery (logdev: internal)
XFS (dm-4): Mounting Filesystem
XFS (dm-4): Ending clean mount
XFS (dm-9): Mounting Filesystem
XFS (dm-9): Ending clean mount
XFS (dm-14): Mounting Filesystem
XFS (dm-14): Ending clean mount
XFS (dm-15): Mounting Filesystem
XFS (dm-15): Starting recovery (logdev: internal)
XFS (dm-15): Ending recovery (logdev: internal)
XFS (dm-16): Mounting Filesystem
XFS (dm-16): Starting recovery (logdev: internal)
XFS (dm-16): Ending recovery (logdev: internal)
XFS (loop1): Mounting Filesystem
XFS (loop1): Ending clean mount
XFS (loop2): Mounting Filesystem
XFS (loop2): Ending clean mount
INFO: task xfsalloc/0:463 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfsalloc/0 D ffff880004b1b840 0 463 2 0x00000000
ffff88003eb27a50 0000000000000246 0000000000000000 0000000000000000
ffffffff810074fd ffff88003eb27a20 ffffffff81007ca2 0000000000000000
ffff8800045705f8 ffff88003eb27fd8 000000000000fbc8 ffff8800045705f8
Call Trace:
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffffa011bf62>] xlog_wait+0x72/0x90 [xfs]
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffffa011d8c3>] ? xlog_state_release_iclog+0xb3/0xf0 [xfs]
[<ffffffffa011ddde>] _xfs_log_force+0x1ae/0x240 [xfs]
[<ffffffffa011e038>] xfs_log_force+0x38/0x90 [xfs]
[<ffffffffa00e3008>] xfs_alloc_ag_vextent_size+0x1b8/0x630 [xfs]
[<ffffffffa00e404d>] xfs_alloc_ag_vextent+0xad/0x100 [xfs]
[<ffffffffa00e4a9c>] xfs_alloc_vextent+0x2bc/0x610 [xfs]
[<ffffffffa00ef197>] xfs_bmap_btalloc+0x267/0x700 [xfs]
[<ffffffffa00ef63e>] xfs_bmap_alloc+0xe/0x10 [xfs]
[<ffffffffa00f5622>] __xfs_bmapi_allocate+0x102/0x340 [xfs]
[<ffffffff81527bfe>] ? thread_return+0x4e/0x760
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffffa00f59a0>] xfs_bmapi_allocate_worker+0x70/0xa0 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffff81094980>] worker_thread+0x170/0x2a0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff8109af00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff81094810>] ? worker_thread+0x0/0x2a0
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff81094810>] ? worker_thread+0x0/0x2a0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task flush-7:0:23981 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-7:0 D ffff880004b1b840 0 23981 2 0x00000080
ffff88003eb6f970 0000000000000246 00000000ffffffff 003eccac90e64182
ffff88003eb6f8f0 ffff88003c28d2f0 0000000001063a6a ffffffffac679ecd
ffff88003e018638 ffff88003eb6ffd8 000000000000fbc8 ffff88003e018638
Call Trace:
[<ffffffff810a6d01>] ? ktime_get_ts+0xb1/0xf0
[<ffffffff8111f590>] ? sync_page+0x0/0x50
[<ffffffff81528383>] io_schedule+0x73/0xc0
[<ffffffff8111f5cd>] sync_page+0x3d/0x50
[<ffffffff81528c1a>] __wait_on_bit_lock+0x5a/0xc0
[<ffffffff8111f567>] __lock_page+0x67/0x70
[<ffffffff8109af80>] ? wake_bit_function+0x0/0x50
[<ffffffff81135875>] ? pagevec_lookup_tag+0x25/0x40
[<ffffffff811347cd>] write_cache_pages+0x3cd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff811b4c8d>] writeback_single_inode+0xdd/0x290
[<ffffffff811b508d>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b51eb>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b55e3>] wb_writeback+0x2f3/0x410
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81084a02>] ? del_timer_sync+0x22/0x30
[<ffffffff811b58a5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b59a3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109ad87>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143616>] bdi_start_fn+0x86/0x100
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task loop1:15077 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
loop1 D ffff880004b1b840 0 15077 2 0x00000080
ffff88003e461500 0000000000000246 ffff88003e4614b0 ffffffffa00ed42d
000000000000000c ffff88003d513dc0 ffffea00005b7920 00000004005b78f8
ffff88003e454638 ffff88003e461fd8 000000000000fbc8 ffff88003e454638
Call Trace:
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff81095171>] ? __queue_work+0x41/0x50
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffffa00f5908>] xfs_bmapi_allocate+0xa8/0xd0 [xfs]
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffffa00f62b6>] xfs_bmapi+0x8e6/0xee0 [xfs]
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffffa0131fc7>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
[<ffffffffa0118b30>] xfs_iomap_write_allocate+0x140/0x370 [xfs]
[<ffffffffa0133a83>] xfs_map_blocks+0x193/0x250 [xfs]
[<ffffffffa0134569>] xfs_vm_writepage+0x1f9/0x590 [xfs]
[<ffffffff81133337>] __writepage+0x17/0x40
[<ffffffff811345fd>] write_cache_pages+0x1fd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff8111fd7b>] __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff810037be>] ? xen_end_context_switch+0x1e/0x30
[<ffffffff8111fdda>] filemap_write_and_wait_range+0x5a/0x90
[<ffffffff811ba6ce>] vfs_fsync_range+0x7e/0x100
[<ffffffff811ba7bd>] vfs_fsync+0x1d/0x20
[<ffffffff8137bcd7>] do_bio_filebacked+0x317/0x330
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bdc1>] loop_thread+0xd1/0x270
[<ffffffff8109af00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bcf0>] ? loop_thread+0x0/0x270
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task loop2:15087 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
loop2 D ffff880004b1b840 0 15087 2 0x00000080
ffff88003c6a5500 0000000000000246 ffff88003c6a54b0 ffffffffa00ed42d
000000000000000b ffff88003e6c4df0 ffffea00009af4b8 00000007009af490
ffff880006a3b058 ffff88003c6a5fd8 000000000000fbc8 ffff880006a3b058
Call Trace:
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff81095171>] ? __queue_work+0x41/0x50
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffffa00f5908>] xfs_bmapi_allocate+0xa8/0xd0 [xfs]
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffffa00f62b6>] xfs_bmapi+0x8e6/0xee0 [xfs]
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffffa0131fc7>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
[<ffffffffa0118b30>] xfs_iomap_write_allocate+0x140/0x370 [xfs]
[<ffffffffa0133a83>] xfs_map_blocks+0x193/0x250 [xfs]
[<ffffffffa0134569>] xfs_vm_writepage+0x1f9/0x590 [xfs]
[<ffffffff81133337>] __writepage+0x17/0x40
[<ffffffff811345fd>] write_cache_pages+0x1fd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff8111fd7b>] __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff810037be>] ? xen_end_context_switch+0x1e/0x30
[<ffffffff8111fdda>] filemap_write_and_wait_range+0x5a/0x90
[<ffffffff811ba6ce>] vfs_fsync_range+0x7e/0x100
[<ffffffff811ba7bd>] vfs_fsync+0x1d/0x20
[<ffffffff8137bcd7>] do_bio_filebacked+0x317/0x330
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bdc1>] loop_thread+0xd1/0x270
[<ffffffff8109af00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bcf0>] ? loop_thread+0x0/0x270
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task flush-7:2:15212 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-7:2 D ffff880004b1b840 0 15212 2 0x00000080
ffff88003e1e7440 0000000000000246 0000000000000000 ffff88003eb20080
ffff88003eb20080 ffff880004b1b840 0000000000000000 0000000000000000
ffff88002cf17098 ffff88003e1e7fd8 000000000000fbc8 ffff88002cf17098
Call Trace:
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff81095171>] ? __queue_work+0x41/0x50
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffffa00f5908>] xfs_bmapi_allocate+0xa8/0xd0 [xfs]
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffffa00f62b6>] xfs_bmapi+0x8e6/0xee0 [xfs]
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffffa0131fc7>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
[<ffffffffa0118b30>] xfs_iomap_write_allocate+0x140/0x370 [xfs]
[<ffffffffa0133a83>] xfs_map_blocks+0x193/0x250 [xfs]
[<ffffffffa0134569>] xfs_vm_writepage+0x1f9/0x590 [xfs]
[<ffffffff81133337>] __writepage+0x17/0x40
[<ffffffff811345fd>] write_cache_pages+0x1fd/0x4c0
[<ffffffff81135875>] ? pagevec_lookup_tag+0x25/0x40
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff811b4c8d>] writeback_single_inode+0xdd/0x290
[<ffffffff811b508d>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b51eb>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b55e3>] wb_writeback+0x2f3/0x410
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81084a02>] ? del_timer_sync+0x22/0x30
[<ffffffff811b58a5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b59a3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109ad87>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143616>] bdi_start_fn+0x86/0x100
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task flush-7:1:15213 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-7:1 D ffff880004b1b840 0 15213 2 0x00000080
ffff88003dc014a0 0000000000000246 0000000000000000 ffff88003eb20080
ffff88003eb20080 ffff880004b1b840 0000000000000000 0000000000000000
ffff88002cf16638 ffff88003dc01fd8 000000000000fbc8 ffff88002cf16638
Call Trace:
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff81095171>] ? __queue_work+0x41/0x50
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffffa00f5908>] xfs_bmapi_allocate+0xa8/0xd0 [xfs]
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffffa00f62b6>] xfs_bmapi+0x8e6/0xee0 [xfs]
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffffa0131fc7>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
[<ffffffffa0118b30>] xfs_iomap_write_allocate+0x140/0x370 [xfs]
[<ffffffffa0133a83>] xfs_map_blocks+0x193/0x250 [xfs]
[<ffffffffa0134569>] xfs_vm_writepage+0x1f9/0x590 [xfs]
[<ffffffff81133337>] __writepage+0x17/0x40
[<ffffffff811345fd>] write_cache_pages+0x1fd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff811b4c8d>] writeback_single_inode+0xdd/0x290
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff811b508d>] writeback_sb_inodes+0xbd/0x170
[<ffffffff81007c49>] ? xen_irq_enable_direct_end+0x0/0x7
[<ffffffff811b5464>] wb_writeback+0x174/0x410
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81084a02>] ? del_timer_sync+0x22/0x30
[<ffffffff811b57bb>] wb_do_writeback+0xbb/0x240
[<ffffffff811b59a3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109ad87>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143616>] bdi_start_fn+0x86/0x100
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task umount:15333 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
umount D ffff880004b1b840 0 15333 14904 0x00000080
ffff88003deabc28 0000000000000286 ffff88003deabbd8 ffffffff810097cc
ffff88002cf160b8 0000000000000000 0000000000eabbe8 ffff880004b193c0
ffff8800131abaf8 ffff88003deabfd8 000000000000fbc8 ffff8800131abaf8
Call Trace:
[<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
[<ffffffff81527bfe>] ? thread_return+0x4e/0x760
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff8106a39b>] ? enqueue_task_fair+0xfb/0x100
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffff811b4185>] writeback_inodes_sb_nr+0x75/0xa0
[<ffffffff811b41f3>] writeback_inodes_sb+0x43/0x50
[<ffffffff811ba95e>] __sync_filesystem+0x3e/0x90
[<ffffffff811bab9a>] sync_filesystem+0x3a/0x70
[<ffffffff8118aed7>] generic_shutdown_super+0x27/0xe0
[<ffffffff8118afc1>] kill_block_super+0x31/0x50
[<ffffffff8118b797>] deactivate_super+0x57/0x80
[<ffffffff811aa73f>] mntput_no_expire+0xbf/0x110
[<ffffffff811ab28b>] sys_umount+0x7b/0x3a0
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task xfsalloc/0:463 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfsalloc/0 D ffff880004b1b840 0 463 2 0x00000000
ffff88003eb27a50 0000000000000246 0000000000000000 0000000000000000
ffffffff810074fd ffff88003eb27a20 ffffffff81007ca2 0000000000000000
ffff8800045705f8 ffff88003eb27fd8 000000000000fbc8 ffff8800045705f8
Call Trace:
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffffa011bf62>] xlog_wait+0x72/0x90 [xfs]
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffffa011d8c3>] ? xlog_state_release_iclog+0xb3/0xf0 [xfs]
[<ffffffffa011ddde>] _xfs_log_force+0x1ae/0x240 [xfs]
[<ffffffffa011e038>] xfs_log_force+0x38/0x90 [xfs]
[<ffffffffa00e3008>] xfs_alloc_ag_vextent_size+0x1b8/0x630 [xfs]
[<ffffffffa00e404d>] xfs_alloc_ag_vextent+0xad/0x100 [xfs]
[<ffffffffa00e4a9c>] xfs_alloc_vextent+0x2bc/0x610 [xfs]
[<ffffffffa00ef197>] xfs_bmap_btalloc+0x267/0x700 [xfs]
[<ffffffffa00ef63e>] xfs_bmap_alloc+0xe/0x10 [xfs]
[<ffffffffa00f5622>] __xfs_bmapi_allocate+0x102/0x340 [xfs]
[<ffffffff81527bfe>] ? thread_return+0x4e/0x760
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffffa00f59a0>] xfs_bmapi_allocate_worker+0x70/0xa0 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffff81094980>] worker_thread+0x170/0x2a0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff8109af00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff81094810>] ? worker_thread+0x0/0x2a0
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff81094810>] ? worker_thread+0x0/0x2a0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task flush-7:0:23981 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-7:0 D ffff880004b1b840 0 23981 2 0x00000080
ffff88003eb6f970 0000000000000246 00000000ffffffff 003eccac90e64182
ffff88003eb6f8f0 ffff88003c28d2f0 0000000001063a6a ffffffffac679ecd
ffff88003e018638 ffff88003eb6ffd8 000000000000fbc8 ffff88003e018638
Call Trace:
[<ffffffff810a6d01>] ? ktime_get_ts+0xb1/0xf0
[<ffffffff8111f590>] ? sync_page+0x0/0x50
[<ffffffff81528383>] io_schedule+0x73/0xc0
[<ffffffff8111f5cd>] sync_page+0x3d/0x50
[<ffffffff81528c1a>] __wait_on_bit_lock+0x5a/0xc0
[<ffffffff8111f567>] __lock_page+0x67/0x70
[<ffffffff8109af80>] ? wake_bit_function+0x0/0x50
[<ffffffff81135875>] ? pagevec_lookup_tag+0x25/0x40
[<ffffffff811347cd>] write_cache_pages+0x3cd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff811b4c8d>] writeback_single_inode+0xdd/0x290
[<ffffffff811b508d>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b51eb>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b55e3>] wb_writeback+0x2f3/0x410
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81084a02>] ? del_timer_sync+0x22/0x30
[<ffffffff811b58a5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b59a3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109ad87>] ? bit_waitqueue+0x17/0xd0
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143616>] bdi_start_fn+0x86/0x100
[<ffffffff81143590>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
INFO: task loop1:15077 blocked for more than 120 seconds.
Not tainted 2.6.32-431.17.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
loop1 D ffff880004b1b840 0 15077 2 0x00000080
ffff88003e461500 0000000000000246 ffff88003e4614b0 ffffffffa00ed42d
000000000000000c ffff88003d513dc0 ffffea00005b7920 00000004005b78f8
ffff88003e454638 ffff88003e461fd8 000000000000fbc8 ffff88003e454638
Call Trace:
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffff81528a95>] schedule_timeout+0x215/0x2e0
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81528713>] wait_for_common+0x123/0x180
[<ffffffff81061d00>] ? default_wake_function+0x0/0x20
[<ffffffff81095171>] ? __queue_work+0x41/0x50
[<ffffffff8152882d>] wait_for_completion+0x1d/0x20
[<ffffffffa00f5908>] xfs_bmapi_allocate+0xa8/0xd0 [xfs]
[<ffffffffa00ed42d>] ? xfs_bmap_search_multi_extents+0xad/0x120 [xfs]
[<ffffffffa00f5930>] ? xfs_bmapi_allocate_worker+0x0/0xa0 [xfs]
[<ffffffffa00f62b6>] xfs_bmapi+0x8e6/0xee0 [xfs]
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffffa0131fc7>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
[<ffffffffa0118b30>] xfs_iomap_write_allocate+0x140/0x370 [xfs]
[<ffffffffa0133a83>] xfs_map_blocks+0x193/0x250 [xfs]
[<ffffffffa0134569>] xfs_vm_writepage+0x1f9/0x590 [xfs]
[<ffffffff81133337>] __writepage+0x17/0x40
[<ffffffff811345fd>] write_cache_pages+0x1fd/0x4c0
[<ffffffff81133320>] ? __writepage+0x0/0x40
[<ffffffff811348e4>] generic_writepages+0x24/0x30
[<ffffffffa01338ad>] xfs_vm_writepages+0x5d/0x80 [xfs]
[<ffffffff81134911>] do_writepages+0x21/0x40
[<ffffffff8111fd7b>] __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff810037be>] ? xen_end_context_switch+0x1e/0x30
[<ffffffff8111fdda>] filemap_write_and_wait_range+0x5a/0x90
[<ffffffff811ba6ce>] vfs_fsync_range+0x7e/0x100
[<ffffffff811ba7bd>] vfs_fsync+0x1d/0x20
[<ffffffff8137bcd7>] do_bio_filebacked+0x317/0x330
[<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
[<ffffffff81007ca2>] ? check_events+0x12/0x20
[<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bdc1>] loop_thread+0xd1/0x270
[<ffffffff8109af00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8152a63c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8137bcf0>] ? loop_thread+0x0/0x270
[<ffffffff8109ab56>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8100b294>] ? int_ret_from_sys_call+0x7/0x1b
[<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Stopping atd: [ OK ]
Stopping sshd: [ OK ]
Shutting down postfix: [ OK ]
Stopping crond: [ OK ]
Shutting down ntpd: [ OK ]
Stopping block device availability: Deactivating block devices:
[ OK ]
Unmounting loopback filesystems: umount: /d: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
--
GlusterFS - http://www.gluster.org
An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.
My personal twitter: twitter.com/realjustinclift
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
