Re: Disk output lockup 2.6.12_rc2 2.6.11.7
Mikael Andersson wrote:
> During heavy io-load a lockup occurs that appears to prevent any disk
> output from taking place. fs is reiserfs on two device-mapper mirrored
> 200G maxtor disks. After the lockup occurs you can to things like 'ls',
> but echo > test.txt will hang.
fs is now ext3
>
> A typical workload producing the error is doing:
> rsync of large (1GB) over 100Mbit ethernet
> simultaneous compilation / gunzip
Or almost anything that writes something to the disk.
>
> I've disabled preemption, and tried with and without acpi enabled, with
> and without smp support (it was smp by default so i switched it off).
> Also tried with another nic (rtl8139) since i got an nv_stop_tx:
> TransmitterStatus remained busy<6> in the logs. I also tried 2.6.11.7
> with the same result.
Tried converting to ext3, some problem, albeit the lockups are less
severe. More of the locked processes can be killed and echo > test.txt
works. So _some_ io gets through.
The output from sysrq-T is somewhat less confusing though, it appears
then hung processes is somehow being hung in __generic_unplug_device, i
had a look at the assembler, but couldn't make heads or tails of it. the
code at __generic_unplug_device+19 was test %eax,%eax immediately
preceded by a callq to the test instruction. Obviously something magic
(by my eyes) is going on here.
Also tried 2.6.12_rc3-mm3
I'd really like to find a solution to this since it kinda borks the
nice an shiny machine if it can't handle large files without getting
into trouble.
I've been working on this for two days, have been trying to find
similar bug reports, trying a lot of different kernels and kernel
options to no avail.
I'm a little out of options right now, any ideas for something to try,
patches to test, or some help in understanding what's happening ?
kmirrord/0 D 81003f1bccd8 0 978 9 1731 977 (L-TLB)
Call Trace:
{cache_alloc_refill+1222}
{io_schedule+15}
--
kjournald D 81003e94bcd8 0 1748 1 2060 953 (L-TLB)
Call Trace:
{__generic_unplug_device+19}
{generic_unplug_device+189}
--
rsync D 00701553dccf 0 6903 6901 (NOTLB)
Call Trace:
{__generic_unplug_device+19}
{generic_unplug_device+189}
--
x86_64-pc-lin D 006dc7d23e49 0 13785 13742 (NOTLB)
Call Trace:
{generic_unplug_device+189}
{dm_unplug_all+29}
/Mikael Andersson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Disk output lockup amd64 nforce4 device-mapper 2.6.12_rc2
During heavy io-load a lockup occurs that appears to prevent any disk
output from taking place. fs is reiserfs on two device-mapper mirrored
200G maxtor disks. After the lockup occurs you can to things like 'ls',
but echo > test.txt will hang.
A typical workload producing the error is doing:
rsync of large (1GB) over 100Mbit ethernet
simultaneous compilation / gunzip
I've disabled preemption, and tried with and without acpi enabled, with
and without smp support (it was smp by default so i switched it off).
Also tried with another nic (rtl8139) since i got an nv_stop_tx:
TransmitterStatus remained busy<6> in the logs. I also tried 2.6.11.7
with the same result.
This produces the lockup after 10-40 minutes. I've got a complete log
from alt-sysreq-T if anyone wants to look at it. The following is a dump
for the bash and kmirrord process when trying echo > test.txt.
The output for kmirrord is included because it's exactly as specified
below both before and after echo > test.txt is invoked, this looked
peculiar to me.
Later runs showed what was to my eyes similar results, some callpaths
weren't exactly the same, but kmirrord was somewhere in mempool_alloc
and the bash process was in do_journal_end, whatever that means :)
bash D c238e000 0 6529 6381 (NOTLB)
81003dfc1bd8 0086
810001b6ba60
6db6db6db6db6db7 801acc46
810001b6ba98 81003d8c27e0
27cd 805a61c0
Call Trace:
{__d_lookup+358} {queue_log_writer+115}
{default_wake_function+0}
{do_journal_end+604}
{search_by_entry_key+50}
{do_journal_begin_r+77}
{do_journal_begin_r+575}
{journal_begin+204}
{reiserfs_create+183}
{vfs_create+188}
{open_namei+454}
{filp_open+39}
{get_unused_fd+223}
{getname+138}
{sys_open+76}
{system_call+126}
kmirrord/0 D 81003f1bccd8 0 978 9 1731 977 (L-TLB)
81003f931ab8 0046
0200 8016a2d6
810036d18e80 0001
00733f931c30 81003f8d4030
952e 810036cf67e0
Call Trace:
{cache_alloc_refill+1222}
{io_schedule+15}
{mempool_alloc+707} {bvec_get_page+0}
{autoremove_wake_function+0}
{autoremove_wake_function+0}
{__rh_alloc+44}
{rh_inc_pending+67}
{do_work+2955}
{do_work+3000}
{thread_return+0}
{do_work+0}
{worker_thread+715}
{default_wake_function+0}
{default_wake_function+0}
{worker_thread+0}
{kthread+205} {keventd_create_kthread+0}
{child_rip+8} {keventd_create_kthread+0}
{kthread+0}
{child_rip+0}
/Mikael Andersson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
