Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 07.10.2013 19:09, John Baldwin wrote:
On Sunday, October 06, 2013 3:30:42 am Alexander Motin wrote:
On 02.10.2013 20:30, John Baldwin wrote:
On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote:
On 07.09.2013 02:02, Jeremie Le Hen wrote:
On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote:
On 06.09.2013 11:06, Jeremie Le Hen wrote:
On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote:
On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin
wrote:
I've found and fixed possible double request completion, that could
cause
such symptoms if happened. Updated patch located as usual:
http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch
With this new one I cannot boot any more (I also updated the source
tree). This is a hand transcripted version:
Trying to mount root from zfs:zroot/root []...
panic: Batch flag already set
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper()
kdb_backtrace()
vpanic()
kassert_panic()
xpt_batch_start()
ata_interrupt()
softclock_call_cc()
softclock()
ithread_loop()
fork_exit()
fork_trampoline()
Thank you for the report. I see my fault. It is probably specific to
ata(4) driver only. I've workarounded that in new patch version, but
probably that area needs some rethinking.
http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch
I'm not sure you needed a confirmation, but it boots. Thanks :).
I didn't quite understand the thread; is direct dispatch enabled for
amd64? ISTR you said only i386 but someone else posted the macro for
amd64.
Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64.
FYI, I tested mfi with this patch set and mfid worked fine for handling
g_up
directly:
Index: dev/mfi/mfi_disk.c
===
--- dev/mfi/mfi_disk.c (revision 257407)
+++ dev/mfi/mfi_disk.c (working copy)
@@ -162,6 +162,7 @@
sc->ld_disk->d_unit = sc->ld_unit;
sc->ld_disk->d_sectorsize = secsize;
sc->ld_disk->d_mediasize = sectors * secsize;
+ sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION;
if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) {
sc->ld_disk->d_fwheads = 255;
sc->ld_disk->d_fwsectors = 63;
Thank you for the feedback. But looking on mfi driver sources I would
say that it calls biodone() from mfi_disk_complete() from cm_complete()
method, which is called while holding mfi_io_lock mutex. I guess that if
on top of mfi device would be some GEOM class, supporting direct
dispatch and sending new requests down on previous request completion
(or retrying requests), that could cause recursive mfi_io_lock
acquisition. That is exactly the cause why I've added this flag. May be
it is a bit paranoid, but it is better to be safe then sorry.
Another good reason to drop the lock before calling biodone() would be
reducing the lock hold time. Otherwise it may just increase lock
congestion there and destroy all benefits of the direct dispatch.
Ah, interesting. What is your policy for such drivers? Should they be
left using g_up, should they drop the lock around biodone when completeing
multiple requests in an interrupt? Should they try to batch them by
waiting and doing biodone at the end after dropping the lock?
In CAM's da and ada drivers I've managed to drop the periph lock before
calling biodone(). New CAM locking design allowed to do it safe.
Possibility of dropping CAM SIM (HBA driver) locks on completion though
depends on HBA design. For example, for ATA/SATA drivers I haven't found
safe way to drop the lock in place, so I delay commands completions up
to the moment when lock can be safely dropped and then process
completions in a batch. For SCSI HBAs with more separated request and
response queues I guess that may be easier. But now all SCSI drivers are
still queuing requests completion to separate CAM threads to decouple
the locks that way. I've just added more of them to spread the load
between cores. But I hope that, for example, mps driver could be
reworked to allow both using multiple MSI-X interrupt threads and
lock-less request completion.
Whether it is possible to do it safe for mfi I don't know. For cases
when dropping the locks is not possible, there is g_up thread to use.
For setups with single controller in a system with single interrupt
thread queuing to g_up thread may even be better, helping to split load
between two CPUs instead of one. But that is in case if HBA is fast or
ineffective enough to saturate one CPU.
--
Alexander Motin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Sunday, October 06, 2013 3:30:42 am Alexander Motin wrote:
> On 02.10.2013 20:30, John Baldwin wrote:
> > On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote:
> >> On 07.09.2013 02:02, Jeremie Le Hen wrote:
> >>> On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote:
> On 06.09.2013 11:06, Jeremie Le Hen wrote:
> > On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote:
> >> On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin
> > wrote:
> >>> I've found and fixed possible double request completion, that could
> > cause
> >>> such symptoms if happened. Updated patch located as usual:
> >>>
http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch
> >>>
> > With this new one I cannot boot any more (I also updated the source
> > tree). This is a hand transcripted version:
> >
> > Trying to mount root from zfs:zroot/root []...
> > panic: Batch flag already set
> > cpuid = 1
> > KDB: stack backtrace:
> > db_trace_self_wrapper()
> > kdb_backtrace()
> > vpanic()
> > kassert_panic()
> > xpt_batch_start()
> > ata_interrupt()
> > softclock_call_cc()
> > softclock()
> > ithread_loop()
> > fork_exit()
> > fork_trampoline()
>
> Thank you for the report. I see my fault. It is probably specific to
> ata(4) driver only. I've workarounded that in new patch version, but
> probably that area needs some rethinking.
>
> http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch
> >>>
> >>> I'm not sure you needed a confirmation, but it boots. Thanks :).
> >>>
> >>> I didn't quite understand the thread; is direct dispatch enabled for
> >>> amd64? ISTR you said only i386 but someone else posted the macro for
> >>> amd64.
> >>
> >> Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64.
> >
> > FYI, I tested mfi with this patch set and mfid worked fine for handling
g_up
> > directly:
> >
> > Index: dev/mfi/mfi_disk.c
> > ===
> > --- dev/mfi/mfi_disk.c (revision 257407)
> > +++ dev/mfi/mfi_disk.c (working copy)
> > @@ -162,6 +162,7 @@
> > sc->ld_disk->d_unit = sc->ld_unit;
> > sc->ld_disk->d_sectorsize = secsize;
> > sc->ld_disk->d_mediasize = sectors * secsize;
> > + sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION;
> > if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) {
> > sc->ld_disk->d_fwheads = 255;
> > sc->ld_disk->d_fwsectors = 63;
> >
>
> Thank you for the feedback. But looking on mfi driver sources I would
> say that it calls biodone() from mfi_disk_complete() from cm_complete()
> method, which is called while holding mfi_io_lock mutex. I guess that if
> on top of mfi device would be some GEOM class, supporting direct
> dispatch and sending new requests down on previous request completion
> (or retrying requests), that could cause recursive mfi_io_lock
> acquisition. That is exactly the cause why I've added this flag. May be
> it is a bit paranoid, but it is better to be safe then sorry.
>
> Another good reason to drop the lock before calling biodone() would be
> reducing the lock hold time. Otherwise it may just increase lock
> congestion there and destroy all benefits of the direct dispatch.
Ah, interesting. What is your policy for such drivers? Should they be
left using g_up, should they drop the lock around biodone when completeing
multiple requests in an interrupt? Should they try to batch them by
waiting and doing biodone at the end after dropping the lock?
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 02.10.2013 20:30, John Baldwin wrote:
On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote:
On 07.09.2013 02:02, Jeremie Le Hen wrote:
On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote:
On 06.09.2013 11:06, Jeremie Le Hen wrote:
On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote:
On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin
wrote:
I've found and fixed possible double request completion, that could
cause
such symptoms if happened. Updated patch located as usual:
http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch
With this new one I cannot boot any more (I also updated the source
tree). This is a hand transcripted version:
Trying to mount root from zfs:zroot/root []...
panic: Batch flag already set
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper()
kdb_backtrace()
vpanic()
kassert_panic()
xpt_batch_start()
ata_interrupt()
softclock_call_cc()
softclock()
ithread_loop()
fork_exit()
fork_trampoline()
Thank you for the report. I see my fault. It is probably specific to
ata(4) driver only. I've workarounded that in new patch version, but
probably that area needs some rethinking.
http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch
I'm not sure you needed a confirmation, but it boots. Thanks :).
I didn't quite understand the thread; is direct dispatch enabled for
amd64? ISTR you said only i386 but someone else posted the macro for
amd64.
Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64.
FYI, I tested mfi with this patch set and mfid worked fine for handling g_up
directly:
Index: dev/mfi/mfi_disk.c
===
--- dev/mfi/mfi_disk.c (revision 257407)
+++ dev/mfi/mfi_disk.c (working copy)
@@ -162,6 +162,7 @@
sc->ld_disk->d_unit = sc->ld_unit;
sc->ld_disk->d_sectorsize = secsize;
sc->ld_disk->d_mediasize = sectors * secsize;
+ sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION;
if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) {
sc->ld_disk->d_fwheads = 255;
sc->ld_disk->d_fwsectors = 63;
Thank you for the feedback. But looking on mfi driver sources I would
say that it calls biodone() from mfi_disk_complete() from cm_complete()
method, which is called while holding mfi_io_lock mutex. I guess that if
on top of mfi device would be some GEOM class, supporting direct
dispatch and sending new requests down on previous request completion
(or retrying requests), that could cause recursive mfi_io_lock
acquisition. That is exactly the cause why I've added this flag. May be
it is a bit paranoid, but it is better to be safe then sorry.
Another good reason to drop the lock before calling biodone() would be
reducing the lock hold time. Otherwise it may just increase lock
congestion there and destroy all benefits of the direct dispatch.
--
Alexander Motin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote:
> On 07.09.2013 02:02, Jeremie Le Hen wrote:
> > On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote:
> >> On 06.09.2013 11:06, Jeremie Le Hen wrote:
> >>> On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote:
> On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin
wrote:
> > I've found and fixed possible double request completion, that could
cause
> > such symptoms if happened. Updated patch located as usual:
> > http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch
> >
> >>> With this new one I cannot boot any more (I also updated the source
> >>> tree). This is a hand transcripted version:
> >>>
> >>> Trying to mount root from zfs:zroot/root []...
> >>> panic: Batch flag already set
> >>> cpuid = 1
> >>> KDB: stack backtrace:
> >>> db_trace_self_wrapper()
> >>> kdb_backtrace()
> >>> vpanic()
> >>> kassert_panic()
> >>> xpt_batch_start()
> >>> ata_interrupt()
> >>> softclock_call_cc()
> >>> softclock()
> >>> ithread_loop()
> >>> fork_exit()
> >>> fork_trampoline()
> >>
> >> Thank you for the report. I see my fault. It is probably specific to
> >> ata(4) driver only. I've workarounded that in new patch version, but
> >> probably that area needs some rethinking.
> >>
> >> http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch
> >
> > I'm not sure you needed a confirmation, but it boots. Thanks :).
> >
> > I didn't quite understand the thread; is direct dispatch enabled for
> > amd64? ISTR you said only i386 but someone else posted the macro for
> > amd64.
>
> Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64.
FYI, I tested mfi with this patch set and mfid worked fine for handling g_up
directly:
Index: dev/mfi/mfi_disk.c
===
--- dev/mfi/mfi_disk.c (revision 257407)
+++ dev/mfi/mfi_disk.c (working copy)
@@ -162,6 +162,7 @@
sc->ld_disk->d_unit = sc->ld_unit;
sc->ld_disk->d_sectorsize = secsize;
sc->ld_disk->d_mediasize = sectors * secsize;
+ sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION;
if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) {
sc->ld_disk->d_fwheads = 255;
sc->ld_disk->d_fwsectors = 63;
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 03, 2013 at 11:48:38PM +0200, Olivier Cochard-Labbé wrote: > On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo wrote: > > Can anyone confirm how well tested/stable this patch set might be?? if > > theres positive input i have a zoo of dev machines i could load it on, to > > help further it. > > Just checking to see how widely its been tested, > > I've installed this patch on 3 differents machines there status after > about 12hours: > - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big > zraid (LSI MegaSAS Gen2 controller): Used for generating package with > poudriere? no probleme since; > - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in > gmirror: Used for generating package with poudriere too? no probleme > since; For testing GEOM direct dispatch on sparc64, please additionally use the following patch: http://people.freebsd.org/~marius/sparc64_GET_STACK_USAGE.diff Marius ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
Alexander Motin wrote: > I would like to invite more people to review and test my patches for > improving CAM and GEOM scalability, that for last six months you could > see developing in project/camlock SVN branch. Full diff of that branch > against present head (r255131) can be found here: > http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch So far I haven't noticed any issues with this patch (or later on with camlock_20130906.patch) using glabel, ggatec, geli and ZFS on amd64. cdda2wav continued to work as expected as well. Fabian signature.asc Description: PGP signature
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 07.09.2013 02:02, Jeremie Le Hen wrote: On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote: On 06.09.2013 11:06, Jeremie Le Hen wrote: On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote: On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin wrote: I've found and fixed possible double request completion, that could cause such symptoms if happened. Updated patch located as usual: http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch With this new one I cannot boot any more (I also updated the source tree). This is a hand transcripted version: Trying to mount root from zfs:zroot/root []... panic: Batch flag already set cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() kdb_backtrace() vpanic() kassert_panic() xpt_batch_start() ata_interrupt() softclock_call_cc() softclock() ithread_loop() fork_exit() fork_trampoline() Thank you for the report. I see my fault. It is probably specific to ata(4) driver only. I've workarounded that in new patch version, but probably that area needs some rethinking. http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch I'm not sure you needed a confirmation, but it boots. Thanks :). I didn't quite understand the thread; is direct dispatch enabled for amd64? ISTR you said only i386 but someone else posted the macro for amd64. Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64. -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote: > On 06.09.2013 11:06, Jeremie Le Hen wrote: > > On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote: > >> On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin wrote: > >>> I've found and fixed possible double request completion, that could cause > >>> such symptoms if happened. Updated patch located as usual: > >>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch > >>> > > With this new one I cannot boot any more (I also updated the source > > tree). This is a hand transcripted version: > > > > Trying to mount root from zfs:zroot/root []... > > panic: Batch flag already set > > cpuid = 1 > > KDB: stack backtrace: > > db_trace_self_wrapper() > > kdb_backtrace() > > vpanic() > > kassert_panic() > > xpt_batch_start() > > ata_interrupt() > > softclock_call_cc() > > softclock() > > ithread_loop() > > fork_exit() > > fork_trampoline() > > Thank you for the report. I see my fault. It is probably specific to > ata(4) driver only. I've workarounded that in new patch version, but > probably that area needs some rethinking. > > http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch I'm not sure you needed a confirmation, but it boots. Thanks :). I didn't quite understand the thread; is direct dispatch enabled for amd64? ISTR you said only i386 but someone else posted the macro for amd64. -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 06.09.2013 11:06, Jeremie Le Hen wrote: On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote: On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin wrote: I've found and fixed possible double request completion, that could cause such symptoms if happened. Updated patch located as usual: http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch With this new one I cannot boot any more (I also updated the source tree). This is a hand transcripted version: Trying to mount root from zfs:zroot/root []... panic: Batch flag already set cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() kdb_backtrace() vpanic() kassert_panic() xpt_batch_start() ata_interrupt() softclock_call_cc() softclock() ithread_loop() fork_exit() fork_trampoline() Thank you for the report. I see my fault. It is probably specific to ata(4) driver only. I've workarounded that in new patch version, but probably that area needs some rethinking. http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote: > On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin wrote: > > I've found and fixed possible double request completion, that could cause > > such symptoms if happened. Updated patch located as usual: > > http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch > > > > Good catch! > this new patch (applied to r255188) fix the problem on my laptop. With this new one I cannot boot any more (I also updated the source tree). This is a hand transcripted version: Trying to mount root from zfs:zroot/root []... panic: Batch flag already set cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() kdb_backtrace() vpanic() kassert_panic() xpt_batch_start() ata_interrupt() softclock_call_cc() softclock() ithread_loop() fork_exit() fork_trampoline() -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin wrote: > I've found and fixed possible double request completion, that could cause > such symptoms if happened. Updated patch located as usual: > http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch > Good catch! this new patch (applied to r255188) fix the problem on my laptop. Thanks, Olivier ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 04.09.2013 19:31, Olivier Cochard-Labbé wrote: On Wed, Sep 4, 2013 at 9:01 AM, Alexander Motin wrote: - HP EliteBook 8460p (amd64: r255188) with DVD replaced by a second hardrive (where fbsd is installed): It crash just after the message "GEOM: new disk ada1" during boot screenshot of the crash screen: http://goo.gl/tW1VIx A little more information: addr2line -e /boot/kernel/kernel.symbols 0x8083abd3 /usr/src/sys/geom/geom_io.c:129 Unfortunately I can't reproduce that and have not enough clues. It may be specific to some GEOM class. Could you describe/show all GEOM topology, file systems, etc. you have there? gpart show sysctl kern.geom.confxml I've found and fixed possible double request completion, that could cause such symptoms if happened. Updated patch located as usual: http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 09/04/13 11:00, John Baldwin wrote:
On Wednesday, September 04, 2013 10:11:28 am Nathan Whitehorn wrote:
On 09/04/13 08:20, Ryan Stone wrote:
On Wed, Sep 4, 2013 at 8:45 AM, Nathan Whitehorn
wrote:
Could you describe what this macro is supposed to do so that we can do
the
porting work?
-Nathan
#define GET_STACK_USAGE(total, used)
GET_STACK_USAGE sets the variable passed in total to the total amount
of stack space available to the current thread. used is set to the
amount of stack space currently used (this does not have to have
byte-precision). Netgraph uses this to decide when to stop recursing
and instead defer to a work queue (to prevent stack overflow). I
presume that Alexander is using it in a similar way. It looks like
the amd64 version could be ported to other architectures quite easily
if you were to account for stacks that grow up and stacks that grow
down:
http://svnweb.freebsd.org/base/head/sys/amd64/include/proc.h?revision=233291&view=markup
/* Get the current kernel thread stack usage. */
#define GET_STACK_USAGE(total, used) do {\
struct thread*td = curthread;\
(total) = td->td_kstack_pages * PAGE_SIZE;\
(used) = (char *)td->td_kstack +\
td->td_kstack_pages * PAGE_SIZE -\
(char *)&td;\
} while (0)
___
freebsd-hack...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
I think that should be MI for us anyway. I'm not aware of any
architectures FreeBSD supports with stacks that grow up. I'll give it a
test on PPC.
ia64 has the double stack thingie where the register stack spills into a stack
that grows up rather than down. Not sure how sparc64 window spills are
handled either.
Ah, very well. That's weird. Should be fine for PPC, however.
-Nathan
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Wed, Sep 4, 2013 at 9:01 AM, Alexander Motin wrote: > >> - HP EliteBook 8460p (amd64: r255188) with DVD replaced by a second >> hardrive (where fbsd is installed): It crash just after the message >> "GEOM: new disk ada1" during boot >> >> screenshot of the crash screen: >> http://goo.gl/tW1VIx >> >> A little more information: >> addr2line -e /boot/kernel/kernel.symbols 0x8083abd3 >> /usr/src/sys/geom/geom_io.c:129 > > > Unfortunately I can't reproduce that and have not enough clues. It may be > specific to some GEOM class. Could you describe/show all GEOM topology, file > systems, etc. you have there? > gpart show > sysctl kern.geom.confxml > ... Here are: - ada0: internal laptop hard-drive - ada1: the hard-drive that replace my DVD reader (partition 2: /boot in clear, partition 3: geli encrypted partition) => There is no special boot-loader, I manually select the hard-drive during Hardware: ahci0: port 0x4088-0x408f,0x4094-0x4097,0x4080-0x4087,0x4090-0x4093,0x4040-0x405f mem 0xd4827000-0xd48277ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 305245MB (625142448 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada1: quirks=0x1<4K> ada1: Previously was known as ad6 Regarding partition: root@laptop:/root # gpart show => 63 625142385 ada0 MBR (298G) 63 1985- free - (992k) 2048 31457280 1 ntfs (15G) 314593284194304 2 ntfs [active] (2.0G) 35653632 589486768 3 ntfs (281G) 625140400 2048- free - (1.0M) => 34 976773101 ada1 GPT (465G) 34 6- free - (3.0k) 40128 1 freebsd-boot (64k) 1684194304 2 freebsd-ufs (2.0G) 4194472 972578656 3 freebsd-ufs (463G) 976773128 7- free - (3.5k) And here is the long output of geom configuration: root@laptop:/root # sysctl kern.geom.confxml kern.geom.confxml: FD RAID DEV ada1p3.eli 4 r0w0e0 gpt/boot 4 r0w0e0 gptid/1c841adf-3cde-11e2-8cfc-a0b3cc295ab2 4 r0w0e0 ada1p3 3 r0w0e0 ada1p2 3 r0w0e0 ada1p1 3 r0w0e0 ntfs/System 4 r1w1e0 ntfs/BDEDrive 4 r0w0e0 ntfs/Recovery 4 r0w0e0 ada1 2 r0w0e0 ada0s3 3 r0w0e0 ada0s2 3 r0w0e0 ada0s1 3 r0w0e0 ada0 2 r0w0e0 ELI ada1p3.eli 3 928 928 BOOT 0 6 hardware 256 AES-XTS ACTIVE r1w1e1 r1w1e1 ada1p3.eli 497960271360 512 0 0 PART ada1 2 GPT 128 34 976773134 63 16 OK false r2w2e5 r1w1e1 ada1p3 497960271872 512 4096 0 4194472 976773127 3 freebsd-ufs 2147569664 497960271872 root 516e7cb6-6ecf-11d6-8ff8-00022d09712b c8e5c3ff-3cde-11e2-8cfc-a0b3cc295ab2 r1w1e2 ada1p2 2147483648 512 4096 0 168 4194471 2 freebsd-ufs 86016 2147483648 boot 516e7cb6-6ecf-11d6-8ff8-00022d09712b 874bc9a6-3cde-11e2-8cfc-a0b3cc295ab2 r0w0e0 ada1p1 65536 512 4096 0 40 167 1 freebsd-boot 20480 65536 83bd6b9d-7f41-11dc-be0b-001560b84f0f 1c841adf-3cde-11e2-8cfc-a0b3cc295ab2 ada0 2 MBR 4 63 625142447 63 16 OK false r1w1e2 r1w1e1 ada0s3 301817225216 512 0 1074790400 35653632 625140399 3 ntfs 18254659584 301817225216 7 r0w0e0 ada0s2 2147483648 512 0 374048 31459328 35653631 2 ntfs 16107175936 2147483648 7 active r0w0e0 ada0s1 16106127360 512 0 1048576 2048 31459327 1 ntfs 1048576 16106127360 7 LABEL ada1p2 3 r1w1e2 r1w1e1 gpt/boot 2147483648 512 4096 0 0 2147483648 4194304 0 0 ada1p1 3 r0w0e0 r0w0e0 gptid/1c841adf-3cde-11e2-8cfc-a0b3cc295ab2 65536 512 4096 0 0 65536 128 0 0 ada0s3
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Wednesday, September 04, 2013 10:11:28 am Nathan Whitehorn wrote:
> On 09/04/13 08:20, Ryan Stone wrote:
> > On Wed, Sep 4, 2013 at 8:45 AM, Nathan Whitehorn
wrote:
> >> Could you describe what this macro is supposed to do so that we can do
the
> >> porting work?
> >> -Nathan
> > #define GET_STACK_USAGE(total, used)
> >
> > GET_STACK_USAGE sets the variable passed in total to the total amount
> > of stack space available to the current thread. used is set to the
> > amount of stack space currently used (this does not have to have
> > byte-precision). Netgraph uses this to decide when to stop recursing
> > and instead defer to a work queue (to prevent stack overflow). I
> > presume that Alexander is using it in a similar way. It looks like
> > the amd64 version could be ported to other architectures quite easily
> > if you were to account for stacks that grow up and stacks that grow
> > down:
> >
> >
http://svnweb.freebsd.org/base/head/sys/amd64/include/proc.h?revision=233291&view=markup
> >
> > /* Get the current kernel thread stack usage. */
> > #define GET_STACK_USAGE(total, used) do {\
> > struct thread*td = curthread;\
> > (total) = td->td_kstack_pages * PAGE_SIZE;\
> > (used) = (char *)td->td_kstack +\
> > td->td_kstack_pages * PAGE_SIZE -\
> > (char *)&td;\
> > } while (0)
> > ___
> > freebsd-hack...@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>
> I think that should be MI for us anyway. I'm not aware of any
> architectures FreeBSD supports with stacks that grow up. I'll give it a
> test on PPC.
ia64 has the double stack thingie where the register stack spills into a stack
that grows up rather than down. Not sure how sparc64 window spills are
handled either.
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 09/04/13 08:20, Ryan Stone wrote:
On Wed, Sep 4, 2013 at 8:45 AM, Nathan Whitehorn wrote:
Could you describe what this macro is supposed to do so that we can do the
porting work?
-Nathan
#define GET_STACK_USAGE(total, used)
GET_STACK_USAGE sets the variable passed in total to the total amount
of stack space available to the current thread. used is set to the
amount of stack space currently used (this does not have to have
byte-precision). Netgraph uses this to decide when to stop recursing
and instead defer to a work queue (to prevent stack overflow). I
presume that Alexander is using it in a similar way. It looks like
the amd64 version could be ported to other architectures quite easily
if you were to account for stacks that grow up and stacks that grow
down:
http://svnweb.freebsd.org/base/head/sys/amd64/include/proc.h?revision=233291&view=markup
/* Get the current kernel thread stack usage. */
#define GET_STACK_USAGE(total, used) do {\
struct thread*td = curthread;\
(total) = td->td_kstack_pages * PAGE_SIZE;\
(used) = (char *)td->td_kstack +\
td->td_kstack_pages * PAGE_SIZE -\
(char *)&td;\
} while (0)
___
freebsd-hack...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
I think that should be MI for us anyway. I'm not aware of any
architectures FreeBSD supports with stacks that grow up. I'll give it a
test on PPC.
-Nathan
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 04.09.2013 15:45, Nathan Whitehorn wrote: On 09/04/13 02:01, Alexander Motin wrote: On 04.09.2013 00:48, Olivier Cochard-Labbé wrote: On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo wrote: Can anyone confirm how well tested/stable this patch set might be?? if theres positive input i have a zoo of dev machines i could load it on, to help further it. Just checking to see how widely its been tested, I've installed this patch on 3 differents machines there status after about 12hours: - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big zraid (LSI MegaSAS Gen2 controller): Used for generating package with poudriere… no probleme since; - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in gmirror: Used for generating package with poudriere too… no probleme since; I've forgot to mention, but GEOM direct dispatch is now active only on x86 because GET_STACK_USAGE macro now defined only there and I wanted to stay on a safe side. On other archs GEOM works in old queued way. Somebody should port that small macro to other archs. But that is still interesting data point. Thanks. Could you describe what this macro is supposed to do so that we can do the porting work? It supposed to report total and used stack sizes for current thread. I suppose that it will be equal for the most archs, but better somebody familiar with each one looked on it. The macro itself is not new. It is for years used in Netgraph for equivalent purpose. -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Wed, Sep 4, 2013 at 8:45 AM, Nathan Whitehorn wrote:
> Could you describe what this macro is supposed to do so that we can do the
> porting work?
> -Nathan
#define GET_STACK_USAGE(total, used)
GET_STACK_USAGE sets the variable passed in total to the total amount
of stack space available to the current thread. used is set to the
amount of stack space currently used (this does not have to have
byte-precision). Netgraph uses this to decide when to stop recursing
and instead defer to a work queue (to prevent stack overflow). I
presume that Alexander is using it in a similar way. It looks like
the amd64 version could be ported to other architectures quite easily
if you were to account for stacks that grow up and stacks that grow
down:
http://svnweb.freebsd.org/base/head/sys/amd64/include/proc.h?revision=233291&view=markup
/* Get the current kernel thread stack usage. */
#define GET_STACK_USAGE(total, used) do {\
struct thread*td = curthread;\
(total) = td->td_kstack_pages * PAGE_SIZE;\
(used) = (char *)td->td_kstack +\
td->td_kstack_pages * PAGE_SIZE -\
(char *)&td;\
} while (0)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 09/04/13 02:01, Alexander Motin wrote: On 04.09.2013 00:48, Olivier Cochard-Labbé wrote: On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo wrote: Can anyone confirm how well tested/stable this patch set might be?? if theres positive input i have a zoo of dev machines i could load it on, to help further it. Just checking to see how widely its been tested, I've installed this patch on 3 differents machines there status after about 12hours: - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big zraid (LSI MegaSAS Gen2 controller): Used for generating package with poudriere… no probleme since; - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in gmirror: Used for generating package with poudriere too… no probleme since; I've forgot to mention, but GEOM direct dispatch is now active only on x86 because GET_STACK_USAGE macro now defined only there and I wanted to stay on a safe side. On other archs GEOM works in old queued way. Somebody should port that small macro to other archs. But that is still interesting data point. Thanks. Could you describe what this macro is supposed to do so that we can do the porting work? -Nathan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 04.09.2013 11:20, Johan Hendriks wrote: Alexander Motin wrote: I would like to invite more people to review and test my patches for improving CAM and GEOM scalability, that for last six months you could see developing in project/camlock SVN branch. Full diff of that branch against present head (r255131) can be found here: http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch Heavy CAM changes there were focused on reducing scope of SIM lock to only protecting SIM internals, but not CAM core. That allows many times reduce lock congestion, especially on heavily parallel request submission with GEOM changes below. More detailed description of changes you could see here earlier: http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209 GEOM changes were focused on avoiding switching to GEOM up/down threads in relatively simple setups where respective classes don't require it (and were explicitly marked so). That allows save on context switches and on systems with several HBAs and disks talk to them concurrently (that is where CAM locking changes are handy). Such classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP, PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and some others. Requests to/from other classes will be queued to GEOM threads same as before. Together that allows to double block subsystem performance on high (at least 100-200K) IOPS benchmarks, allowing to reach up to a million total IOPS, while keeping full compatibility with all major ABIs/KBIs. Since we are already in 10.0 release process and changes are quite big, my plan is to wait and commit them to head branch after the freeze end, and then merge to stable/10. I hope the release process will go on schedule to not delay this work for another six months. This work is sponsored by iXsystems, Inc. Hello i would like to test this patch set. But how can i stress the machine, do you have a script or something i can use to make the system do heavy I/O on the disks! For testing IOPS over RAW disks or different GEOM providers (to exclude FS influence) I am using such a trivial synthetic test: #!/bin/sh for disk in da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 do for i in `jot 16` do dd if=/dev/$disk of=/dev/null bs=512 & done done iostat -zxw3 -c12 | grep total |tail -n 10 |cut -c 6-18 killall dd For total IOPS measurement in above script I am using dirtily hacked iostat tool that prints summary values in addition to per-disk ones: http://people.freebsd.org/~mav/camlock_patches/iostat.patch BTW I've uploaded new patch with some more minor CAM changes: http://people.freebsd.org/~mav/camlock_patches/camlock_20130904.patch -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
Alexander Motin wrote: Hi. I would like to invite more people to review and test my patches for improving CAM and GEOM scalability, that for last six months you could see developing in project/camlock SVN branch. Full diff of that branch against present head (r255131) can be found here: http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch Heavy CAM changes there were focused on reducing scope of SIM lock to only protecting SIM internals, but not CAM core. That allows many times reduce lock congestion, especially on heavily parallel request submission with GEOM changes below. More detailed description of changes you could see here earlier: http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209 GEOM changes were focused on avoiding switching to GEOM up/down threads in relatively simple setups where respective classes don't require it (and were explicitly marked so). That allows save on context switches and on systems with several HBAs and disks talk to them concurrently (that is where CAM locking changes are handy). Such classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP, PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and some others. Requests to/from other classes will be queued to GEOM threads same as before. Together that allows to double block subsystem performance on high (at least 100-200K) IOPS benchmarks, allowing to reach up to a million total IOPS, while keeping full compatibility with all major ABIs/KBIs. Since we are already in 10.0 release process and changes are quite big, my plan is to wait and commit them to head branch after the freeze end, and then merge to stable/10. I hope the release process will go on schedule to not delay this work for another six months. This work is sponsored by iXsystems, Inc. Hello i would like to test this patch set. But how can i stress the machine, do you have a script or something i can use to make the system do heavy I/O on the disks! regards Johan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On 04.09.2013 00:48, Olivier Cochard-Labbé wrote: On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo wrote: Can anyone confirm how well tested/stable this patch set might be?? if theres positive input i have a zoo of dev machines i could load it on, to help further it. Just checking to see how widely its been tested, I've installed this patch on 3 differents machines there status after about 12hours: - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big zraid (LSI MegaSAS Gen2 controller): Used for generating package with poudriere… no probleme since; - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in gmirror: Used for generating package with poudriere too… no probleme since; I've forgot to mention, but GEOM direct dispatch is now active only on x86 because GET_STACK_USAGE macro now defined only there and I wanted to stay on a safe side. On other archs GEOM works in old queued way. Somebody should port that small macro to other archs. But that is still interesting data point. Thanks. - HP EliteBook 8460p (amd64: r255188) with DVD replaced by a second hardrive (where fbsd is installed): It crash just after the message "GEOM: new disk ada1" during boot screenshot of the crash screen: http://goo.gl/tW1VIx A little more information: addr2line -e /boot/kernel/kernel.symbols 0x8083abd3 /usr/src/sys/geom/geom_io.c:129 Unfortunately I can't reproduce that and have not enough clues. It may be specific to some GEOM class. Could you describe/show all GEOM topology, file systems, etc. you have there? gpart show sysctl kern.geom.confxml ... Thank you! -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 03, 2013 at 11:24:26PM +0200, Jeremie Le Hen wrote: > On Tue, Sep 03, 2013 at 02:10:32PM -0400, Outback Dingo wrote: > > On Tue, Sep 3, 2013 at 9:42 AM, Jeremie Le Hen wrote: > > > > > On Mon, Sep 02, 2013 at 11:49:33AM +0300, Alexander Motin wrote: > > > > Hi. > > > > > > > > I would like to invite more people to review and test my patches for > > > > improving CAM and GEOM scalability, that for last six months you could > > > > see developing in project/camlock SVN branch. Full diff of that branch > > > > against present head (r255131) can be found here: > > > > http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch > > > > > > I'm building my kernel right now. > > > > > > > > Can anyone confirm how well tested/stable this patch set might be?? if > > theres positive input i have a zoo of dev machines i could load it on, to > > help further it. > > Just checking to see how widely its been tested, > > Very stable so far. I'm doing a make -j 4 buildworld in parallel of a > periodic security run. This has been running for hours without failure. FWIW, I have 4 drives total, distributed in 2 zpool containing 2 mirrored zdev each. -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 3, 2013 at 5:48 PM, Olivier Cochard-Labbé wrote: > On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo > wrote: > > Can anyone confirm how well tested/stable this patch set might be?? if > > theres positive input i have a zoo of dev machines i could load it on, to > > help further it. > > Just checking to see how widely its been tested, > > I've installed this patch on 3 differents machines there status after > about 12hours: > - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big > zraid (LSI MegaSAS Gen2 controller): Used for generating package with > poudriere… no probleme since; > - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in > gmirror: Used for generating package with poudriere too… no probleme > since; > - HP EliteBook 8460p (amd64: r255188) with DVD replaced by a second > hardrive (where fbsd is installed): It crash just after the message > "GEOM: new disk ada1" during boot > > screenshot of the crash screen: > http://goo.gl/tW1VIx > > A little more information: > addr2line -e /boot/kernel/kernel.symbols 0x8083abd3 > /usr/src/sys/geom/geom_io.c:129 > > Regards, > > Olivier > Be nice if it was backported to 9/stable. not sure how feasible it is though... patch fails in a few places. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 3, 2013 at 8:10 PM, Outback Dingo wrote: > Can anyone confirm how well tested/stable this patch set might be?? if > theres positive input i have a zoo of dev machines i could load it on, to > help further it. > Just checking to see how widely its been tested, I've installed this patch on 3 differents machines there status after about 12hours: - SUN FIRE X4170 M2 (amd64: r255178) with 6 SAS harddrives in one big zraid (LSI MegaSAS Gen2 controller): Used for generating package with poudriere… no probleme since; - HAL/Fujitsu SPARC64-V (sparc64: r255178) with two SCSI-3 disks in gmirror: Used for generating package with poudriere too… no probleme since; - HP EliteBook 8460p (amd64: r255188) with DVD replaced by a second hardrive (where fbsd is installed): It crash just after the message "GEOM: new disk ada1" during boot screenshot of the crash screen: http://goo.gl/tW1VIx A little more information: addr2line -e /boot/kernel/kernel.symbols 0x8083abd3 /usr/src/sys/geom/geom_io.c:129 Regards, Olivier ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 03, 2013 at 02:10:32PM -0400, Outback Dingo wrote: > On Tue, Sep 3, 2013 at 9:42 AM, Jeremie Le Hen wrote: > > > On Mon, Sep 02, 2013 at 11:49:33AM +0300, Alexander Motin wrote: > > > Hi. > > > > > > I would like to invite more people to review and test my patches for > > > improving CAM and GEOM scalability, that for last six months you could > > > see developing in project/camlock SVN branch. Full diff of that branch > > > against present head (r255131) can be found here: > > > http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch > > > > I'm building my kernel right now. > > > > > Can anyone confirm how well tested/stable this patch set might be?? if > theres positive input i have a zoo of dev machines i could load it on, to > help further it. > Just checking to see how widely its been tested, Very stable so far. I'm doing a make -j 4 buildworld in parallel of a periodic security run. This has been running for hours without failure. -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Tue, Sep 3, 2013 at 9:42 AM, Jeremie Le Hen wrote: > On Mon, Sep 02, 2013 at 11:49:33AM +0300, Alexander Motin wrote: > > Hi. > > > > I would like to invite more people to review and test my patches for > > improving CAM and GEOM scalability, that for last six months you could > > see developing in project/camlock SVN branch. Full diff of that branch > > against present head (r255131) can be found here: > > http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch > > I'm building my kernel right now. > > Can anyone confirm how well tested/stable this patch set might be?? if theres positive input i have a zoo of dev machines i could load it on, to help further it. Just checking to see how widely its been tested, > -- > Jeremie Le Hen > > Scientists say the world is made up of Protons, Neutrons and Electrons. > They forgot to mention Morons. > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
On Mon, Sep 02, 2013 at 11:49:33AM +0300, Alexander Motin wrote: > Hi. > > I would like to invite more people to review and test my patches for > improving CAM and GEOM scalability, that for last six months you could > see developing in project/camlock SVN branch. Full diff of that branch > against present head (r255131) can be found here: > http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch I'm building my kernel right now. -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
[RFC][CFT] GEOM direct dispatch and fine-grained CAM locking
Hi. I would like to invite more people to review and test my patches for improving CAM and GEOM scalability, that for last six months you could see developing in project/camlock SVN branch. Full diff of that branch against present head (r255131) can be found here: http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch Heavy CAM changes there were focused on reducing scope of SIM lock to only protecting SIM internals, but not CAM core. That allows many times reduce lock congestion, especially on heavily parallel request submission with GEOM changes below. More detailed description of changes you could see here earlier: http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209 GEOM changes were focused on avoiding switching to GEOM up/down threads in relatively simple setups where respective classes don't require it (and were explicitly marked so). That allows save on context switches and on systems with several HBAs and disks talk to them concurrently (that is where CAM locking changes are handy). Such classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP, PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and some others. Requests to/from other classes will be queued to GEOM threads same as before. Together that allows to double block subsystem performance on high (at least 100-200K) IOPS benchmarks, allowing to reach up to a million total IOPS, while keeping full compatibility with all major ABIs/KBIs. Since we are already in 10.0 release process and changes are quite big, my plan is to wait and commit them to head branch after the freeze end, and then merge to stable/10. I hope the release process will go on schedule to not delay this work for another six months. This work is sponsored by iXsystems, Inc. -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
