Hello Herbert,
Thanks for your help!
Here is the process stack I get when a "dd" process is hanging:
[<ffffffff81199f56>] __find_get_block_slow+0xc6/0x150[<ffffffffa070d8e9>]
ocfs2_metadata_cache_unlock+0x19/0x30 [ocfs2][<ffffffffa070dcd7>]
ocfs2_buffer_cached+0xa7/0x1a0 [ocfs2][<ffffffffa070e7ec>]
ocfs2_set_buffer_uptodate+0x2c/0x100 [ocfs2][<ffffffffffffffff>]
0xffffffffffffffff
Regarding the full stack dump, I found relevant lines by comparing full stack
dumps while the system was frozen and not frozen. Here are the diff that were
identified.
kworker/12:1 S ffff881fc9af65e0 0 90 2 0x00000000
ffff881fc9af9e40 0000000000000046 ffff881fc9af9de0 ffffffff81057602
0000000000012200 ffff881fc9af9fd8 ffff881fc9af8010 0000000000012200
ffff881fc9af9fd8 0000000000012200 ffff881982b4e680 ffff881fc9af6040Call Trace:
[<ffffffff81057602>] ? complete+0x52/0x60 [<ffffffff81085887>] ?
move_linked_works+0x67/0x90 [<ffffffff81503eaf>] schedule+0x3f/0x60
[<ffffffff81088a1e>] worker_thread+0x24e/0x3c0 [<ffffffff810887d0>] ?
manage_workers+0x120/0x120 [<ffffffff8108d546>] kthread+0x96/0xa0
[<ffffffff8150f304>] kernel_thread_helper+0x4/0x10 [<ffffffff8108d4b0>] ?
kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8150f300>] ? gs_change+0x13/0x13
dd R running task 0 25826 14680 0x00000080
ffff881fc4de7388 ffffffff8110c65e ffff881fc4de7388 ffffffff81505e7e
ffff881fc4de73d8 ffffffff81199f56 ffff881fc4de6010 0000000000012200
ffff881fc4de7fd8 0000000000000000 ffff881fc4de73c8 ffffffff81505e7eCall Trace:
[<ffffffff8110c65e>] ? find_get_page+0x1e/0xa0 [<ffffffff81505e7e>] ?
_raw_spin_lock+0xe/0x20 [<ffffffff81199f56>] __find_get_block_slow+0xc6/0x150
[<ffffffff81505e7e>] ? _raw_spin_lock+0xe/0x20 [<ffffffffa06be462>] ?
ocfs2_inode_cache_unlock+0x12/0x20 [ocfs2] [<ffffffffa070d8e9>]
ocfs2_metadata_cache_unlock+0x19/0x30 [ocfs2] [<ffffffffa070dcd7>]
ocfs2_buffer_cached+0xa7/0x1a0 [ocfs2] [<ffffffffa070e7ec>]
ocfs2_set_buffer_uptodate+0x2c/0x100 [ocfs2] [<ffffffffa06be482>] ?
ocfs2_inode_cache_io_unlock+0x12/0x20 [ocfs2] [<ffffffffa06e9a85>] ?
ocfs2_block_group_find_clear_bits+0xf5/0x180 [ocfs2] [<ffffffffa06ea568>] ?
ocfs2_cluster_group_search+0xa8/0x230 [ocfs2] [<ffffffffa06eac13>] ?
ocfs2_read_group_descriptor+0x73/0xb0 [ocfs2] [<ffffffffa06ebe20>] ?
ocfs2_search_chain+0x100/0x730 [ocfs2] [<ffffffffa06ec7ec>] ?
ocfs2_claim_suballoc_bits+0x39c/0x570 [ocfs2] [<ffffffffa01b16ff>] ?
do_get_write_access+0x35f/0x600 [jbd2] [<ffffffffa06eca69>] ?
__ocfs2_claim_clusters+0xa9/0x340 [ocfs2] [<ffffffffa070d8e9>] ?
ocfs2_metadata_cache_unlock+0x19/0x30 [ocfs2] [<ffffffffa06ecd1d>] ?
ocfs2_claim_clusters+0x1d/0x20 [ocfs2] [<ffffffffa06cafdf>] ?
ocfs2_local_alloc_new_window+0x6f/0x340 [ocfs2] [<ffffffffa06cb413>] ?
ocfs2_local_alloc_slide_window+0x163/0x5c0 [ocfs2] [<ffffffffa06cb9c7>] ?
ocfs2_reserve_local_alloc_bits+0x157/0x340 [ocfs2] [<ffffffffa06ca2b8>] ?
ocfs2_alloc_should_use_local+0x68/0xd0 [ocfs2] [<ffffffffa06ee475>] ?
ocfs2_reserve_clusters_with_limit+0xb5/0x320 [ocfs2] [<ffffffffa06ef3a8>] ?
ocfs2_reserve_clusters+0x18/0x20 [ocfs2] [<ffffffffa06efdce>] ?
ocfs2_lock_allocators+0x1fe/0x2b0 [ocfs2] [<ffffffffa069d7a3>] ?
ocfs2_write_begin_nolock+0x913/0x1100 [ocfs2] [<ffffffffa06be482>] ?
ocfs2_inode_cache_io_unlock+0x12/0x20 [ocfs2] [<ffffffffa070d959>] ?
ocfs2_metadata_cache_io_unlock+0x19/0x30 [ocfs2] [<ffffffffa069f628>] ?
ocfs2_read_blocks+0x2f8/0x6c0 [ocfs2] [<ffffffffa06c7460>] ?
ocfs2_journal_access_eb+0x20/0x20 [ocfs2] [<ffffffffa069e086>] ?
ocfs2_write_begin+0xf6/0x220 [ocfs2] [<ffffffff8110be43>] ?
generic_perform_write+0xc3/0x1c0 [<ffffffffa06b94ad>] ?
ocfs2_prepare_inode_for_write+0x10d/0x710 [ocfs2] [<ffffffff8110bf6b>] ?
generic_file_buffered_write_iter+0x2b/0x60 [<ffffffffa06ba677>] ?
ocfs2_file_write_iter+0x367/0x9b0 [ocfs2] [<ffffffffa06bad48>] ?
ocfs2_file_aio_write+0x88/0xa0 [ocfs2] [<ffffffff8116c7c2>] ?
do_sync_write+0xe2/0x120 [<ffffffff811feae3>] ?
security_file_permission+0x23/0x90 [<ffffffff8116cd58>] ? vfs_write+0xc8/0x190
[<ffffffff8116cf21>] ? sys_write+0x51/0x90 [<ffffffff810cc1bb>] ?
audit_syscall_exit+0x25b/0x290 [<ffffffff8150e1c2>] ?
system_call_fastpath+0x16/0x1b
Thanks,
Jeff
Date: Thu, 25 Oct 2012 18:39:31 -0700
From: herbert.van.den.be...@oracle.com
To: jpaterso...@hotmail.com
CC: ocfs2-users@oss.oracle.com
Subject: Re: [Ocfs2-users] OCFS2 hanging on writes
Hello Jeff,
You might want to check what the writer process is waiting on when
it's frozen. The wchan column of ps might be enough, but if not,
then perhaps a kernel stack trace of the process from
/proc/<pid>/stack or from echo t > /proc/sysrq-trigger .
The latter will show other blocked processes as well, which may be
helpful in determining the cause of the freeze.
Thanks,
Herbert.
On 10/25/2012 06:32 PM, Jeff Paterson wrote:
Hello,
I would need help with our
OCFS2 (1.8.0) filesystem. We are having problems with
it since a couple days. When we write onto it, it
hangs.
The "hanging pattern" is
easily reproductible. If I write a 1GB file on the
filesystem, it does the following:
- write ~200 MB of
data on the disk in 1 second
- freeze for about
10 seconds
- write ~200 MB of
data on the disk in 1 second
- freeze for about
10 seconds
- write ~200 MB of
data on the disk in 1 second
- freeze for about
10 seconds
(and so on)
When the freezes occur:
- other writes
operations (from other processes) on the same node
also freeze
- writes operations
on other nodes are not affected by the freezes on
another node
Read
operations (on any cluster node, even the one with
frozen writes) don't seem to be affected by the
freezes. One sure thing, read operations alone don't cause
the
filesystem freeze.
For info, before the problem began to appear we
could sustain 640 MB/s writes without any freeze.
I tried to mount the filesystem on a single node
to avoid issues that could happen with inter-node
communications and the problem was still there.
Filesystem details
The filesystem has 18 TB and it is currently
72% full.
Mount options are the following:
rw,nodev,_netdev,noatime,errors=panic,data=writeback,noacl,nouser_xattr,commit=60,heartbeat=local
All Features: backup-super
strict-journal-super sparse extended-slotmap
inline-data metaecc indexed-dirs refcount
discontig-bg unwritten
There is nothing special in the systems logs
beside application errors caused by the freezes.
Would a fsck.ocfs2 help? How long would it take
for 18 TB?
Is there a flag I can enable in debugfs.ocfs2 to
get a better idea of what is happening and why it is
freezing like that?
Any help would be greatly appreciated.
Thanks in advance,
Jeff
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-users
