Re: [PREEMPT-RT] [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On 2017-05-17 17:07:34 [+0200], To Chad Dupuis wrote: > > > Sebastian, can you add this change to your patch set? > > > > Are sure that you can reliably reproduce the issue and fix it with the > > patch above? Because this patch: > > oh. Okay. Now it clicked. It can fix the issue but it is still possible, > that CPU0 goes down between your check for it and schedule_work_on() > returning. Let my think of something… Oh wait. I already thought about this: it may take bnx2fc_percpu from CPU7 and run the worker on CPU3. The job isn't lost, because the worker does: | static void bnx2fc_percpu_io_work(struct work_struct *work_s) | { | struct bnx2fc_percpu_s *p; … | p = container_of(work_s, struct bnx2fc_percpu_s, work); | | spin_lock_bh(>fp_work_lock); and so will access bnx2fc_percpu of CPU7 running on CPU3. So I *think* that your patch should make no difference and there should be no leak if schedule_work_on() is invoked on an offline CPU. Sebastian
Re: [PREEMPT-RT] [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On 2017-05-17 17:01:53 [+0200], To Chad Dupuis wrote: > On 2017-05-12 11:55:52 [-0400], Chad Dupuis wrote: > > Ok, I believe I've found the issue here. The machine that the test has > > performed on had many more possible CPUs than active CPUs. We calculate > > which CPU to the work time on in bnx2fc_process_new_cqes() like this: > > > > unsigned int cpu = wqe % num_possible_cpus(); > > > > Since not all CPUs are active, we were trying to schedule work on > > non-active CPUs which meant that the upper layers were never notified of > > the completion. With this change: > > > > diff --git a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > index c2288d6..6f08e43 100644 > > --- a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > +++ b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > @@ -1042,7 +1042,12 @@ static int bnx2fc_process_new_cqes(struct > > bnx2fc_rport *tgt) > > /* Pending work request completion */ > > struct bnx2fc_work *work = NULL; > > struct bnx2fc_percpu_s *fps = NULL; > > - unsigned int cpu = wqe % num_possible_cpus(); > > + unsigned int cpu = wqe % num_active_cpus(); > > + > > + /* Sanity check cpu to make sure it's online */ > > + if (!cpu_active(cpu)) > > + /* Default to CPU 0 */ > > + cpu = 0; > > > > work = bnx2fc_alloc_work(tgt, wqe); > > if (work) { > > > > The issue is fixed. > > > > Sebastian, can you add this change to your patch set? > > Are sure that you can reliably reproduce the issue and fix it with the > patch above? Because this patch: oh. Okay. Now it clicked. It can fix the issue but it is still possible, that CPU0 goes down between your check for it and schedule_work_on() returning. Let my think of something… Sebastian
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On Wed, 17 May 2017, 11:01am, Sebastian Andrzej Siewior wrote: > On 2017-05-12 11:55:52 [-0400], Chad Dupuis wrote: > > Ok, I believe I've found the issue here. The machine that the test has > > performed on had many more possible CPUs than active CPUs. We calculate > > which CPU to the work time on in bnx2fc_process_new_cqes() like this: > > > > unsigned int cpu = wqe % num_possible_cpus(); > > > > Since not all CPUs are active, we were trying to schedule work on > > non-active CPUs which meant that the upper layers were never notified of > > the completion. With this change: > > > > diff --git a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > index c2288d6..6f08e43 100644 > > --- a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > +++ b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > > @@ -1042,7 +1042,12 @@ static int bnx2fc_process_new_cqes(struct > > bnx2fc_rport *tgt) > > /* Pending work request completion */ > > struct bnx2fc_work *work = NULL; > > struct bnx2fc_percpu_s *fps = NULL; > > - unsigned int cpu = wqe % num_possible_cpus(); > > + unsigned int cpu = wqe % num_active_cpus(); > > + > > + /* Sanity check cpu to make sure it's online */ > > + if (!cpu_active(cpu)) > > + /* Default to CPU 0 */ > > + cpu = 0; > > > > work = bnx2fc_alloc_work(tgt, wqe); > > if (work) { > > > > The issue is fixed. > > > > Sebastian, can you add this change to your patch set? > > Are sure that you can reliably reproduce the issue and fix it with the > patch above? Because this patch: Yes it was reproducible each time we would start the FCoE interface. With the above patch, our sanity test passed with no issues seen. > > diff --git a/init/main.c b/init/main.c > index b0c11cbf5ddf..483a971b1fd2 100644 > --- a/init/main.c > +++ b/init/main.c > @@ -997,6 +997,12 @@ static int __ref kernel_init(void *unused) > "See Linux Documentation/admin-guide/init.rst for guidance."); > } > > +static void workfn(struct work_struct *work) > +{ > + pr_err("%s() %d\n", __func__, raw_smp_processor_id()); > +} > +static DECLARE_WORK(work, workfn); > + > static noinline void __init kernel_init_freeable(void) > { > /* > @@ -1040,6 +1046,15 @@ static noinline void __init kernel_init_freeable(void) > > (void) sys_dup(0); > (void) sys_dup(0); > + { > + > + cpu_down(3); > + pr_err("%s() num possible: %d\n", __func__, > num_possible_cpus()); > + pr_err("%s() num online : %d\n", __func__, > num_online_cpus()); > + pr_err("%s() 3 active: %d\n", __func__, cpu_active(3)); > + schedule_work_on(3, ); > + ssleep(5); > + } > /* > * check if there is an early userspace init. If yes, let it do all > * the work > > produces this output: > [1.960313] Unregister pv shared memory for cpu 3 > [1.997000] kernel_init_freeable() num possible: 8 > [1.998073] kernel_init_freeable() num online : 7 > [1.999125] kernel_init_freeable() 3 active: 0 > [2.000337] workfn() 1 > > which means, CPU3 is offline and work runs on CPU1 instead. So it does > already what you suggest except that chances are, that it is not run on > CPU0 in this case (but on another CPU). > > So it either takes some time for wait_for_completion(_req->tm_done); > to come back _or_ there is a leak somewhere where a complete() is > somehow missing / racing against something. > > Sebastian >
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On 2017-05-12 11:55:52 [-0400], Chad Dupuis wrote: > Ok, I believe I've found the issue here. The machine that the test has > performed on had many more possible CPUs than active CPUs. We calculate > which CPU to the work time on in bnx2fc_process_new_cqes() like this: > > unsigned int cpu = wqe % num_possible_cpus(); > > Since not all CPUs are active, we were trying to schedule work on > non-active CPUs which meant that the upper layers were never notified of > the completion. With this change: > > diff --git a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > index c2288d6..6f08e43 100644 > --- a/drivers/scsi/bnx2fc/bnx2fc_hwi.c > +++ b/drivers/scsi/bnx2fc/bnx2fc_hwi.c > @@ -1042,7 +1042,12 @@ static int bnx2fc_process_new_cqes(struct > bnx2fc_rport *tgt) > /* Pending work request completion */ > struct bnx2fc_work *work = NULL; > struct bnx2fc_percpu_s *fps = NULL; > - unsigned int cpu = wqe % num_possible_cpus(); > + unsigned int cpu = wqe % num_active_cpus(); > + > + /* Sanity check cpu to make sure it's online */ > + if (!cpu_active(cpu)) > + /* Default to CPU 0 */ > + cpu = 0; > > work = bnx2fc_alloc_work(tgt, wqe); > if (work) { > > The issue is fixed. > > Sebastian, can you add this change to your patch set? Are sure that you can reliably reproduce the issue and fix it with the patch above? Because this patch: diff --git a/init/main.c b/init/main.c index b0c11cbf5ddf..483a971b1fd2 100644 --- a/init/main.c +++ b/init/main.c @@ -997,6 +997,12 @@ static int __ref kernel_init(void *unused) "See Linux Documentation/admin-guide/init.rst for guidance."); } +static void workfn(struct work_struct *work) +{ + pr_err("%s() %d\n", __func__, raw_smp_processor_id()); +} +static DECLARE_WORK(work, workfn); + static noinline void __init kernel_init_freeable(void) { /* @@ -1040,6 +1046,15 @@ static noinline void __init kernel_init_freeable(void) (void) sys_dup(0); (void) sys_dup(0); + { + + cpu_down(3); + pr_err("%s() num possible: %d\n", __func__, num_possible_cpus()); + pr_err("%s() num online : %d\n", __func__, num_online_cpus()); + pr_err("%s() 3 active: %d\n", __func__, cpu_active(3)); + schedule_work_on(3, ); + ssleep(5); + } /* * check if there is an early userspace init. If yes, let it do all * the work produces this output: [1.960313] Unregister pv shared memory for cpu 3 [1.997000] kernel_init_freeable() num possible: 8 [1.998073] kernel_init_freeable() num online : 7 [1.999125] kernel_init_freeable() 3 active: 0 [2.000337] workfn() 1 which means, CPU3 is offline and work runs on CPU1 instead. So it does already what you suggest except that chances are, that it is not run on CPU0 in this case (but on another CPU). So it either takes some time for wait_for_completion(_req->tm_done); to come back _or_ there is a leak somewhere where a complete() is somehow missing / racing against something. Sebastian
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On Tue, 9 May 2017, 11:18am, James Bottomley wrote: > On Tue, 2017-05-09 at 10:17 -0400, Chad Dupuis wrote: > > On Mon, 8 May 2017, 10:04pm, Martin K. Petersen wrote: > > > > > > > > Sebastian, > > > > > > > Martin, do you see any chance to get this merged? Chad replied to > > the > > > > list that he is going to test it on 2017-04-10, didn't respond to > > the > > > > ping 10 days later. The series stalled last time in the same way. > > > > > > I am very reluctant to merge something when a driver has an active > > > maintainer and that person has not acked the change. > > > > > > That said, Chad: You have been sitting on this for quite a while. > > Please > > > make it a priority. In exchange for veto rights you do have to > > provide > > > timely feedback on anything that touches your driver. > > > > > > Thanks! > > > > > > > We did do some testing and hit a calltrace during device discovery: > > > > [ 1332.551799] INFO: task scsi_eh_15:1970 blocked for more than 120 > > seconds. > > [ 1332.551804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables > > this message. > > [ 1332.551807] scsi_eh_15 D 880823488c14 0 1970 2 > > 0x0080 > > [ 1332.551813] 881053a17cb0 0046 88084d693ec0 > > 881053a17fd8 > > [ 1332.551817] 881053a17fd8 881053a17fd8 88084d693ec0 > > 880823488d48 > > [ 1332.551821] 880823488d50 7fff 88084d693ec0 > > 880823488c14 > > [ 1332.551825] Call Trace: > > [ 1332.551838] [] schedule+0x29/0x70 > > [ 1332.551844] [] schedule_timeout+0x239/0x2d0 > > [ 1332.551850] [] ? console_unlock+0x208/0x400 > > [ 1332.551855] [] ? vprintk_emit+0x3c4/0x510 > > [ 1332.551861] [] ? > > lock_timer_base.isra.33+0x2b/0x50 > > [ 1332.551866] [] wait_for_completion+0x116/0x170 > > [ 1332.551874] [] ? wake_up_state+0x20/0x20 > > [ 1332.551885] [] bnx2fc_abts_cleanup+0x3d/0x62 > > [bnx2fc] > > [ 1332.551892] [] bnx2fc_eh_abort+0x470/0x580 > > [bnx2fc] > > [ 1332.551900] [] scsi_error_handler+0x59f/0x8b0 > > [ 1332.551904] [] ? scsi_eh_get_sense+0x250/0x250 > > [ 1332.551911] [] kthread+0xcf/0xe0 > > [ 1332.551916] [] ? > > kthread_create_on_node+0x140/0x140 > > [ 1332.551923] [] ret_from_fork+0x58/0x90 > > [ 1332.551928] [] ? > > kthread_create_on_node+0x140/0x140 > > Reporting this when you found it would have been helpful ... > > That said, it does look like a genuine hang in the workqueues, so it > rather invalidates the current patch set. > > > To be honest, I'm reluctant to merge these patches on bnx2fc as the > > I/O path on this driver has been stable for quite some time and given > > that it's an older driver I'm not looking to make changes there. > > OK, so find a way to achieve both sets of goals because there's a limit > to how long we allow "stable" drivers to hold up infrastructure changes > within the kernel. The main goal of the current patch set is to remove > the cpu hotplug calls from the drivers because they want to remove them > from the kernel. This is rather complex because you're using per cpu > work queues so you currently have to manage starting and stopping them > as the CPUs come up or go down ... getting rid of that for standard > kernel infrastructure will make the driver easier to keep in > maintenance mode for longer. > > James > Ok, I believe I've found the issue here. The machine that the test has performed on had many more possible CPUs than active CPUs. We calculate which CPU to the work time on in bnx2fc_process_new_cqes() like this: unsigned int cpu = wqe % num_possible_cpus(); Since not all CPUs are active, we were trying to schedule work on non-active CPUs which meant that the upper layers were never notified of the completion. With this change: diff --git a/drivers/scsi/bnx2fc/bnx2fc_hwi.c b/drivers/scsi/bnx2fc/bnx2fc_hwi.c index c2288d6..6f08e43 100644 --- a/drivers/scsi/bnx2fc/bnx2fc_hwi.c +++ b/drivers/scsi/bnx2fc/bnx2fc_hwi.c @@ -1042,7 +1042,12 @@ static int bnx2fc_process_new_cqes(struct bnx2fc_rport *tgt) /* Pending work request completion */ struct bnx2fc_work *work = NULL; struct bnx2fc_percpu_s *fps = NULL; - unsigned int cpu = wqe % num_possible_cpus(); + unsigned int cpu = wqe % num_active_cpus(); + + /* Sanity check cpu to make sure it's online */ + if (!cpu_active(cpu)) + /* Default to CPU 0 */ + cpu = 0; work = bnx2fc_alloc_work(tgt, wqe); if (work) { The issue is fixed. Sebastian, can you add this change to your patch set?
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
Chad, > To be honest, I'm reluctant to merge these patches on bnx2fc as the > I/O path on this driver has been stable for quite some time and given > that it's an older driver I'm not looking to make changes there. I understand that the driver is in maintenance mode. However, the Linux kernel and its interfaces are not. We do not offer any guarantees in that department. So even if you do not intend to add new features to bnx2fc, you will still need to keep it current wrt. the ongoing changes in kernel interfaces (for better and for worse). -- Martin K. Petersen Oracle Linux Engineering
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On Tue, 2017-05-09 at 10:17 -0400, Chad Dupuis wrote: > On Mon, 8 May 2017, 10:04pm, Martin K. Petersen wrote: > > > > > Sebastian, > > > > > Martin, do you see any chance to get this merged? Chad replied to > the > > > list that he is going to test it on 2017-04-10, didn't respond to > the > > > ping 10 days later. The series stalled last time in the same way. > > > > I am very reluctant to merge something when a driver has an active > > maintainer and that person has not acked the change. > > > > That said, Chad: You have been sitting on this for quite a while. > Please > > make it a priority. In exchange for veto rights you do have to > provide > > timely feedback on anything that touches your driver. > > > > Thanks! > > > > We did do some testing and hit a calltrace during device discovery: > > [ 1332.551799] INFO: task scsi_eh_15:1970 blocked for more than 120 > seconds. > [ 1332.551804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables > this message. > [ 1332.551807] scsi_eh_15 D 880823488c14 0 1970 2 > 0x0080 > [ 1332.551813] 881053a17cb0 0046 88084d693ec0 > 881053a17fd8 > [ 1332.551817] 881053a17fd8 881053a17fd8 88084d693ec0 > 880823488d48 > [ 1332.551821] 880823488d50 7fff 88084d693ec0 > 880823488c14 > [ 1332.551825] Call Trace: > [ 1332.551838] [] schedule+0x29/0x70 > [ 1332.551844] [] schedule_timeout+0x239/0x2d0 > [ 1332.551850] [] ? console_unlock+0x208/0x400 > [ 1332.551855] [] ? vprintk_emit+0x3c4/0x510 > [ 1332.551861] [] ? > lock_timer_base.isra.33+0x2b/0x50 > [ 1332.551866] [] wait_for_completion+0x116/0x170 > [ 1332.551874] [] ? wake_up_state+0x20/0x20 > [ 1332.551885] [] bnx2fc_abts_cleanup+0x3d/0x62 > [bnx2fc] > [ 1332.551892] [] bnx2fc_eh_abort+0x470/0x580 > [bnx2fc] > [ 1332.551900] [] scsi_error_handler+0x59f/0x8b0 > [ 1332.551904] [] ? scsi_eh_get_sense+0x250/0x250 > [ 1332.551911] [] kthread+0xcf/0xe0 > [ 1332.551916] [] ? > kthread_create_on_node+0x140/0x140 > [ 1332.551923] [] ret_from_fork+0x58/0x90 > [ 1332.551928] [] ? > kthread_create_on_node+0x140/0x140 Reporting this when you found it would have been helpful ... That said, it does look like a genuine hang in the workqueues, so it rather invalidates the current patch set. > To be honest, I'm reluctant to merge these patches on bnx2fc as the > I/O path on this driver has been stable for quite some time and given > that it's an older driver I'm not looking to make changes there. OK, so find a way to achieve both sets of goals because there's a limit to how long we allow "stable" drivers to hold up infrastructure changes within the kernel. The main goal of the current patch set is to remove the cpu hotplug calls from the drivers because they want to remove them from the kernel. This is rather complex because you're using per cpu work queues so you currently have to manage starting and stopping them as the CPUs come up or go down ... getting rid of that for standard kernel infrastructure will make the driver easier to keep in maintenance mode for longer. James
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On Mon, 8 May 2017, 10:04pm, Martin K. Petersen wrote: > > Sebastian, > > > Martin, do you see any chance to get this merged? Chad replied to the > > list that he is going to test it on 2017-04-10, didn't respond to the > > ping 10 days later. The series stalled last time in the same way. > > I am very reluctant to merge something when a driver has an active > maintainer and that person has not acked the change. > > That said, Chad: You have been sitting on this for quite a while. Please > make it a priority. In exchange for veto rights you do have to provide > timely feedback on anything that touches your driver. > > Thanks! > We did do some testing and hit a calltrace during device discovery: [ 1332.551799] INFO: task scsi_eh_15:1970 blocked for more than 120 seconds. [ 1332.551804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1332.551807] scsi_eh_15 D 880823488c14 0 1970 2 0x0080 [ 1332.551813] 881053a17cb0 0046 88084d693ec0 881053a17fd8 [ 1332.551817] 881053a17fd8 881053a17fd8 88084d693ec0 880823488d48 [ 1332.551821] 880823488d50 7fff 88084d693ec0 880823488c14 [ 1332.551825] Call Trace: [ 1332.551838] [] schedule+0x29/0x70 [ 1332.551844] [] schedule_timeout+0x239/0x2d0 [ 1332.551850] [] ? console_unlock+0x208/0x400 [ 1332.551855] [] ? vprintk_emit+0x3c4/0x510 [ 1332.551861] [] ? lock_timer_base.isra.33+0x2b/0x50 [ 1332.551866] [] wait_for_completion+0x116/0x170 [ 1332.551874] [] ? wake_up_state+0x20/0x20 [ 1332.551885] [] bnx2fc_abts_cleanup+0x3d/0x62 [bnx2fc] [ 1332.551892] [] bnx2fc_eh_abort+0x470/0x580 [bnx2fc] [ 1332.551900] [] scsi_error_handler+0x59f/0x8b0 [ 1332.551904] [] ? scsi_eh_get_sense+0x250/0x250 [ 1332.551911] [] kthread+0xcf/0xe0 [ 1332.551916] [] ? kthread_create_on_node+0x140/0x140 [ 1332.551923] [] ret_from_fork+0x58/0x90 [ 1332.551928] [] ? kthread_create_on_node+0x140/0x140 To be honest, I'm reluctant to merge these patches on bnx2fc as the I/O path on this driver has been stable for quite some time and given that it's an older driver I'm not looking to make changes there.
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
Sebastian, > Martin, do you see any chance to get this merged? Chad replied to the > list that he is going to test it on 2017-04-10, didn't respond to the > ping 10 days later. The series stalled last time in the same way. I am very reluctant to merge something when a driver has an active maintainer and that person has not acked the change. That said, Chad: You have been sitting on this for quite a while. Please make it a priority. In exchange for veto rights you do have to provide timely feedback on anything that touches your driver. Thanks! -- Martin K. Petersen Oracle Linux Engineering
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On 2017-04-10 19:12:49 [+0200], To Martin K . Petersen wrote: > This is a repost to get the patches applied against v4.11-rc6. mkp's scsi > for-next tree can be merged with no conflicts. … Martin, do you see any chance to get this merged? Chad replied to the list that he is going to test it on 2017-04-10, didn't respond to the ping 10 days later. The series stalled last time in the same way. > The whole series is also available at > git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/hotplug-staging.git > scsi/bnx2_v2 Sebastian
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On 2017-04-10 14:20:41 [-0400], Chad Dupuis wrote: > Sebastian, will take a look. This [0] commit in tip's smp/hotplug branch is staged for the next merge window. It will trigger a lockdep warning on the recursive get_online_cpus() invocation in the two drivers. It is fixed/avoided by the series. [0] https://git.kernel.org/tip/tip/c/d215aab82d81974f438bfbc80aa437132f3c37c3 Sebastian
Re: [REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
On Mon, 10 Apr 2017, 5:12pm -, Sebastian Andrzej Siewior wrote: > This is a repost to get the patches applied against v4.11-rc6. mkp's scsi > for-next tree can be merged with no conflicts. > > The last repost [0] was not merged and stalled after Martin pinged Chad > [1]. He didn't even reply after tglx pinged him approx two weeks later. > > Johannes Thumshirn was so kind to test the FCoE part of the old series > [2]. I did not add a tested-by tag due to the rebase. > > If my memory is correct then my first attempt on this (with hotplug > threads back then) was around December 2015. > > The whole series is also available at > git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/hotplug-staging.git > scsi/bnx2_v2 > > [0] http://www.spinics.net/lists/linux-scsi/msg102143.html > [1] http://www.spinics.net/lists/linux-scsi/msg102295.html > [2] http://www.spinics.net/lists/linux-scsi/msg102716.html > > Sebastian > Sebastian, will take a look.
[REEEEPOST] bnx2i + bnx2fc: convert to generic workqueue (#3)
This is a repost to get the patches applied against v4.11-rc6. mkp's scsi for-next tree can be merged with no conflicts. The last repost [0] was not merged and stalled after Martin pinged Chad [1]. He didn't even reply after tglx pinged him approx two weeks later. Johannes Thumshirn was so kind to test the FCoE part of the old series [2]. I did not add a tested-by tag due to the rebase. If my memory is correct then my first attempt on this (with hotplug threads back then) was around December 2015. The whole series is also available at git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/hotplug-staging.git scsi/bnx2_v2 [0] http://www.spinics.net/lists/linux-scsi/msg102143.html [1] http://www.spinics.net/lists/linux-scsi/msg102295.html [2] http://www.spinics.net/lists/linux-scsi/msg102716.html Sebastian