Still the same, both with bonding enabled and disabled unfortunatly
-----Original Message----- From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 November 2006 2:52 p.m. To: Rafiu Fakunle Cc: Dave Watkins; [email protected] Subject: Re: [OF-users] iSCSI bug? Rafiu Fakunle wrote: > Dave Watkins wrote: >> Ok, UP is fine. To be sure it wasn't the e1000 driver I also tried using >> only the Broadcom NIC's as well. Under UP there is no error, under SMP >> the error reoccurs even with e1000 not loaded and no bonding. >> >> Hope this helps >> > Immensely. I'm just doing up a changeset for you. http://www.openfiler.com/download/PACKAGES/iscsi_trgt-kernel-0.4.14.ccs conary update iscsi_trgt-kernel-0.4.14.ccs --replace-files Then test again with 2.6.17.14-0.3.smp.x86_64 (with and without bonding) Thx, R. > > R. > >> Dave >> >> -----Original Message----- >> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >> November 2006 1:15 p.m. >> To: Dave Watkins >> Cc: [email protected] >> Subject: Re: [OF-users] iSCSI bug? >> >> OK, and UP without trunking? >> >> R. >> >> Dave Watkins wrote: >> >>> With or without trunking seem to generate the same problem >>> >>> Without trunking I got >>> BUG: soft lockup detected on CPU#0! >>> >>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>> <ffffffff80289151>{update_process_times+66} >>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>> <ffffffff80224b87>{tcp_sendmsg+0} >>> <ffffffff80413bba>{inet_ioctl+0} >>> <ffffffff88141216>{:iscsi_trgt:is_data_available+62} >>> <ffffffff881419e7>{:iscsi_trgt:istd+1460} >>> <ffffffff80403ea6>{tcp_sendpage+0} >>> <ffffffff8027fef6>{__wake_up_common+67} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff80231a7d>{kthread+200} <ffffffff8025f8a2>{child_rip+8} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0} >>> BUG: soft lockup detected on CPU#0! >>> >>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>> <ffffffff80289151>{update_process_times+66} >>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>> <ffffffff80224b87>{tcp_sendmsg+0} >>> <ffffffff881411c0>{:iscsi_trgt:nthread_wakeup+35} >>> <ffffffff881411b3>{:iscsi_trgt:nthread_wakeup+22} >>> <ffffffff8814219a>{:iscsi_trgt:istd+3431} >>> <ffffffff80403ea6>{tcp_sendpage+0} >>> <ffffffff8027fef6>{__wake_up_common+67} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff80231a7d>{kthread+200} >>> <ffffffff8025f8a2>{child_rip+8} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff802319b5>{kthread+0} >>> <ffffffff8025f89a>{child_rip+0} >>> >>> Re-enabling trunking again and I get >>> BUG: soft lockup detected on CPU#0! >>> >>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>> <ffffffff80289151>{update_process_times+66} >>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>> <ffffffff80254356>{tcp_ioctl+0} >>> <ffffffff8020af50>{__might_sleep+30} >>> <ffffffff802326d7>{lock_sock+28} >>> <ffffffff80263257>{_spin_lock_bh+9} >>> <ffffffff8022fd23>{release_sock+15} >>> <ffffffff802543a2>{tcp_ioctl+76} >>> <ffffffff80413c44>{inet_ioctl+138} >>> <ffffffff88141216>{:iscsi_trgt:is_data_available+62} >>> <ffffffff8814125a>{:iscsi_trgt:do_recv+41} >>> <ffffffff8023081f>{qdisc_restart+24} >>> <ffffffff8022eaa6>{dev_queue_xmit+510} >>> <ffffffff8807c266>{:bonding:bond_dev_queue_xmit+489} >>> <ffffffff8023277e>{lock_sock+195} >>> <ffffffff8807fd96>{:bonding:bond_xmit_roundrobin+154} >>> <ffffffff80232136>{__tcp_push_pending_frames+1367} >>> <ffffffff8022fd23>{release_sock+15} >>> <ffffffff80225551>{tcp_sendmsg+2506} >>> <ffffffff80236f84>{do_sock_write+199} >>> <ffffffff803dbac1>{sock_writev+220} >>> <ffffffff8025db21>{cache_alloc_refill+237} >>> <ffffffff80220d80>{tcp_transmit_skb+1579} >>> <ffffffff80408067>{tcp_retransmit_skb+1352} >>> <ffffffff80254356>{tcp_ioctl+0} >>> <ffffffff8024f5a4>{finish_wait+52} >>> <ffffffff803e0d10>{sk_stream_wait_memory+458} >>> <ffffffff80291608>{autoremove_wake_function+0} >>> <ffffffff80291608>{autoremove_wake_function+0} >>> <ffffffff8022fd23>{release_sock+15} >>> <ffffffff80246a25>{try_to_wake_up+955} >>> <ffffffff88141609>{:iscsi_trgt:istd+470} >>> <ffffffff80403ea6>{tcp_sendpage+0} >>> <ffffffff8027fef6>{__wake_up_common+67} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff80231a7d>{kthread+200} >>> <ffffffff8025f8a2>{child_rip+8} >>> <ffffffff8029131c>{keventd_create_kthread+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff8027308f>{flat_send_IPI_mask+0} >>> <ffffffff802319b5>{kthread+0} >>> <ffffffff8025f89a>{child_rip+0} >>> >>> Without trunking though the write performance after this doesn't seem >>> >> to >> >>> be affected (still at about 80-90MB rather than down at less than >>> >> 10MB) >> >>> -----Original Message----- >>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >>> November 2006 12:27 p.m. >>> To: Dave Watkins >>> Cc: [email protected] >>> Subject: Re: [OF-users] iSCSI bug? >>> >>> Dave Watkins wrote: >>> >>>> Sorry about that, I remembered as soon as I sent it that I hadn't >>>> included version. It's x86_64 version 2.2 (did a conary updateall >>>> >> from >> >>>> 2.1 beta. Uname -r gives 2.6.17.14-0.3.smp.x86_64. >>>> >>>> I'll try with a UP kernel although it will take some time as I have >>>> >> to >> >>>> rebuild the e1000 module from the UP kernel sources. >>> Try without the network trunking anyway in the meantime. Would be an >>> interesting test. >>> >>> R. >>> >>> >>> >>>> I'll let you know >>>> if I can reproduce on the UP kernel. >>>> >>>> I don't think it's related to that ticket as they are all writes >>>> >>> anyway >>> >>>> and they only see the problem on large files. >>>> >>>> Dave >>>> >>>> -----Original Message----- >>>> From: Rafiu Fakunle [mailto:[EMAIL PROTECTED] Sent: Monday, 27 >>>> November 2006 11:40 a.m. >>>> To: Dave Watkins >>>> Cc: [email protected] >>>> Subject: Re: [OF-users] iSCSI bug? >>>> >>>> Hi Dave, >>>> >>>> Excellent test and bug report. >>>> >>>> I wonder whether it may be related to this: >>>> >>>> https://project.openfiler.com/tracker/ticket/435 >>>> >>>> Can you try to reproduce with a UP kernel pls. >>>> >>>> Also I need the output of `uname -r` >>>> >>>> Thx, >>>> >>>> R. >>>> >>>> FTR: this is running r58 from IET svn >>>> >>>> >>>> Dave Watkins wrote: >>>> >>>>> Hi All >>>>> >>>>> I think I've found a bug in the iscsi target software in my >>>>> benchmarking/testing. >>>>> >>>>> Some background on the hardware first in case it may be related. >>>>> Dual core/dual opteron with 2GB of ram >>>>> 3ware 8006 2 port raid card for OS drives >>>>> 3ware 9550SX card for data drives >>>>> Dual GB Broadcom on-board NIC's teamed into bond0 (management) >>>>> Quad port Intel PCI-E GB NIC with all 4 ports teamed into bond1 >>>>> >> (main >> >>>>> iscsi data network) >>>>> 4 x 250GB WD SATA HDD's in RAID5 >>>>> >>>>> Of note here is that I have had to replace the e1000 driver with the >>>>> latest from Intel to support the quad port card >>>>> >>>>> I have made some volumes and mounted them on various windows servers >>>>> >>>> and >>>> >>>>> have been using iobench to tune performance of the system. When >>>>> >> using >> >>>>> >>>> a >>>> >>>>> read only test pattern I see this >>>>> >>>>> BUG: soft lockup detected on CPU#0! >>>>> >>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>> <ffffffff80289151>{update_process_times+66} >>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>> <ffffffff88141486>{:iscsi_trgt:istd+83} >>>>> <ffffffff88141476>{:iscsi_trgt:istd+67} >>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff80231a7d>{kthread+200} >>>>> <ffffffff8025f8a2>{child_rip+8} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0} >>>>> BUG: soft lockup detected on CPU#0! >>>>> >>>>> Call Trace: <IRQ> <ffffffff8029f73c>{softlockup_tick+210} >>>>> <ffffffff80289151>{update_process_times+66} >>>>> <ffffffff802713fe>{smp_local_timer_interrupt+35} >>>>> <ffffffff80271463>{smp_apic_timer_interrupt+65} >>>>> <ffffffff8025f54c>{apic_timer_interrupt+132} <EOI> >>>>> <ffffffff802631ec>{_spin_unlock_irqrestore+8} >>>>> <ffffffff80246a25>{try_to_wake_up+955} >>>>> <ffffffff881411cc>{:iscsi_trgt:nthread_wakeup+47} >>>>> <ffffffff8814219a>{:iscsi_trgt:istd+3431} >>>>> <ffffffff80403ea6>{tcp_sendpage+0} >>>>> <ffffffff8027fef6>{__wake_up_common+67} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff88141433>{:iscsi_trgt:istd+0} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff80231a7d>{kthread+200} >>>>> <ffffffff8025f8a2>{child_rip+8} >>>>> <ffffffff8029131c>{keventd_create_kthread+0} >>>>> <ffffffff802319b5>{kthread+0} <ffffffff8025f89a>{child_rip+0} >>>>> >>>>> Doing write only based patterns this doesn't come up. After this >>>>> performance of the system dives (from about 110MB/sec of iscsi >>>>> performance to about 10MB/sec). >>>>> >>>>> This is fairly reproducible here so if you need anymore information >>>>> >>>> just >>>> >>>>> ask. >>>>> >>>>> Dave >>>>> >>>>> >>>>> >> ------------------------------------------------------------------------ >> >>> >>>> >>>>> _______________________________________________ >>>>> Openfiler-users mailing list >>>>> [email protected] >>>>> https://lists.openfiler.com/mailman/listinfo/openfiler-users >>>>> >>>> >>> >> >> > > _______________________________________________ > Openfiler-users mailing list > [email protected] > https://lists.openfiler.com/mailman/listinfo/openfiler-users _______________________________________________ Openfiler-users mailing list [email protected] https://lists.openfiler.com/mailman/listinfo/openfiler-users
