With or without trunking seem to generate the same problem
Without trunking I got
BUG: soft lockup detected on CPU#0!
Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210}
<ffffffff80289151>{update_process_times+66}
<ffffffff802713fe>{smp_local_timer_interrupt+35}
<ffffffff80271463>{smp_apic_timer_interrupt+65}
<ffffffff8025f54c>{apic_timer_interrupt+132} <EOI>
<ffffffff80224b87>{tcp_sendmsg+0}
<ffffffff80413bba>{inet_ioctl+0}
<ffffffff88141216>{:iscsi_trgt:is_data_available+62}
<ffffffff881419e7>{:iscsi_trgt:istd+1460}
<ffffffff80403ea6>{tcp_sendpage+0}
<ffffffff8027fef6>{__wake_up_common+67}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff88141433>{:iscsi_trgt:istd+0}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff80231a7d>{kthread+200} <ffffffff8025f8a2>{child_rip+8}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0}
BUG: soft lockup detected on CPU#0!
Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210}
<ffffffff80289151>{update_process_times+66}
<ffffffff802713fe>{smp_local_timer_interrupt+35}
<ffffffff80271463>{smp_apic_timer_interrupt+65}
<ffffffff8025f54c>{apic_timer_interrupt+132} <EOI>
<ffffffff80224b87>{tcp_sendmsg+0}
<ffffffff881411c0>{:iscsi_trgt:nthread_wakeup+35}
<ffffffff881411b3>{:iscsi_trgt:nthread_wakeup+22}
<ffffffff8814219a>{:iscsi_trgt:istd+3431}
<ffffffff80403ea6>{tcp_sendpage+0}
<ffffffff8027fef6>{__wake_up_common+67}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff88141433>{:iscsi_trgt:istd+0}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff80231a7d>{kthread+200}
<ffffffff8025f8a2>{child_rip+8}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff802319b5>{kthread+0}
<ffffffff8025f89a>{child_rip+0}
Re-enabling trunking again and I get
BUG: soft lockup detected on CPU#0!
Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210}
<ffffffff80289151>{update_process_times+66}
<ffffffff802713fe>{smp_local_timer_interrupt+35}
<ffffffff80271463>{smp_apic_timer_interrupt+65}
<ffffffff8025f54c>{apic_timer_interrupt+132} <EOI>
<ffffffff80254356>{tcp_ioctl+0}
<ffffffff8020af50>{__might_sleep+30}
<ffffffff802326d7>{lock_sock+28}
<ffffffff80263257>{_spin_lock_bh+9}
<ffffffff8022fd23>{release_sock+15}
<ffffffff802543a2>{tcp_ioctl+76}
<ffffffff80413c44>{inet_ioctl+138}
<ffffffff88141216>{:iscsi_trgt:is_data_available+62}
<ffffffff8814125a>{:iscsi_trgt:do_recv+41}
<ffffffff8023081f>{qdisc_restart+24}
<ffffffff8022eaa6>{dev_queue_xmit+510}
<ffffffff8807c266>{:bonding:bond_dev_queue_xmit+489}
<ffffffff8023277e>{lock_sock+195}
<ffffffff8807fd96>{:bonding:bond_xmit_roundrobin+154}
<ffffffff80232136>{__tcp_push_pending_frames+1367}
<ffffffff8022fd23>{release_sock+15}
<ffffffff80225551>{tcp_sendmsg+2506}
<ffffffff80236f84>{do_sock_write+199}
<ffffffff803dbac1>{sock_writev+220}
<ffffffff8025db21>{cache_alloc_refill+237}
<ffffffff80220d80>{tcp_transmit_skb+1579}
<ffffffff80408067>{tcp_retransmit_skb+1352}
<ffffffff80254356>{tcp_ioctl+0}
<ffffffff8024f5a4>{finish_wait+52}
<ffffffff803e0d10>{sk_stream_wait_memory+458}
<ffffffff80291608>{autoremove_wake_function+0}
<ffffffff80291608>{autoremove_wake_function+0}
<ffffffff8022fd23>{release_sock+15}
<ffffffff80246a25>{try_to_wake_up+955}
<ffffffff88141609>{:iscsi_trgt:istd+470}
<ffffffff80403ea6>{tcp_sendpage+0}
<ffffffff8027fef6>{__wake_up_common+67}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff88141433>{:iscsi_trgt:istd+0}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff80231a7d>{kthread+200}
<ffffffff8025f8a2>{child_rip+8}
<ffffffff8029131c>{keventd_create_kthread+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff8027308f>{flat_send_IPI_mask+0}
<ffffffff802319b5>{kthread+0}
<ffffffff8025f89a>{child_rip+0}
Without trunking though the write performance after this doesn't seem to
be affected (still at about 80-90MB rather than down at less than 10MB)
-----Original Message-----
From: Rafiu Fakunle [mailto:[EMAIL PROTECTED]
Sent: Monday, 27 November 2006 12:27 p.m.
To: Dave Watkins
Cc: [email protected]
Subject: Re: [OF-users] iSCSI bug?
Dave Watkins wrote:
> Sorry about that, I remembered as soon as I sent it that I hadn't
> included version. It's x86_64 version 2.2 (did a conary updateall from
> 2.1 beta. Uname -r gives 2.6.17.14-0.3.smp.x86_64.
>
> I'll try with a UP kernel although it will take some time as I have to
> rebuild the e1000 module from the UP kernel sources.
Try without the network trunking anyway in the meantime. Would be an
interesting test.
R.
> I'll let you know
> if I can reproduce on the UP kernel.
>
> I don't think it's related to that ticket as they are all writes
anyway
> and they only see the problem on large files.
>
> Dave
>
> -----Original Message-----
> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED]
> Sent: Monday, 27 November 2006 11:40 a.m.
> To: Dave Watkins
> Cc: [email protected]
> Subject: Re: [OF-users] iSCSI bug?
>
> Hi Dave,
>
> Excellent test and bug report.
>
> I wonder whether it may be related to this:
>
> https://project.openfiler.com/tracker/ticket/435
>
> Can you try to reproduce with a UP kernel pls.
>
> Also I need the output of `uname -r`
>
> Thx,
>
> R.
>
> FTR: this is running r58 from IET svn
>
>
> Dave Watkins wrote:
>
>> Hi All
>>
>> I think I've found a bug in the iscsi target software in my
>> benchmarking/testing.
>>
>> Some background on the hardware first in case it may be related.
>>
>> Dual core/dual opteron with 2GB of ram
>> 3ware 8006 2 port raid card for OS drives
>> 3ware 9550SX card for data drives
>> Dual GB Broadcom on-board NIC's teamed into bond0 (management)
>> Quad port Intel PCI-E GB NIC with all 4 ports teamed into bond1 (main
>> iscsi data network)
>> 4 x 250GB WD SATA HDD's in RAID5
>>
>> Of note here is that I have had to replace the e1000 driver with the
>> latest from Intel to support the quad port card
>>
>> I have made some volumes and mounted them on various windows servers
>>
> and
>
>> have been using iobench to tune performance of the system. When using
>>
> a
>
>> read only test pattern I see this
>>
>> BUG: soft lockup detected on CPU#0!
>>
>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210}
>> <ffffffff80289151>{update_process_times+66}
>> <ffffffff802713fe>{smp_local_timer_interrupt+35}
>> <ffffffff80271463>{smp_apic_timer_interrupt+65}
>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI>
>> <ffffffff88141486>{:iscsi_trgt:istd+83}
>> <ffffffff88141476>{:iscsi_trgt:istd+67}
>> <ffffffff80403ea6>{tcp_sendpage+0}
>> <ffffffff8027fef6>{__wake_up_common+67}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff88141433>{:iscsi_trgt:istd+0}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff80231a7d>{kthread+200}
>> <ffffffff8025f8a2>{child_rip+8}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0}
>> BUG: soft lockup detected on CPU#0!
>>
>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210}
>> <ffffffff80289151>{update_process_times+66}
>> <ffffffff802713fe>{smp_local_timer_interrupt+35}
>> <ffffffff80271463>{smp_apic_timer_interrupt+65}
>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI>
>> <ffffffff802631ec>{_spin_unlock_irqrestore+8}
>> <ffffffff80246a25>{try_to_wake_up+955}
>> <ffffffff881411cc>{:iscsi_trgt:nthread_wakeup+47}
>> <ffffffff8814219a>{:iscsi_trgt:istd+3431}
>> <ffffffff80403ea6>{tcp_sendpage+0}
>> <ffffffff8027fef6>{__wake_up_common+67}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff88141433>{:iscsi_trgt:istd+0}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff80231a7d>{kthread+200}
>> <ffffffff8025f8a2>{child_rip+8}
>> <ffffffff8029131c>{keventd_create_kthread+0}
>> <ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0}
>>
>> Doing write only based patterns this doesn't come up. After this
>> performance of the system dives (from about 110MB/sec of iscsi
>> performance to about 10MB/sec).
>>
>> This is fairly reproducible here so if you need anymore information
>>
> just
>
>> ask.
>>
>> Dave
>>
>>
>>
>>
>
------------------------------------------------------------------------
>
>> _______________________________________________
>> Openfiler-users mailing list
>> [email protected]
>> https://lists.openfiler.com/mailman/listinfo/openfiler-users
>>
>>
>
>
_______________________________________________
Openfiler-users mailing list
[email protected]
https://lists.openfiler.com/mailman/listinfo/openfiler-users
