Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/3/2012 5:18 PM, John Baldwin wrote:
Seems to apply to RELENG_9 just fine. Are there any stress tests you
suggest I run that might expose some bugs ? The machine is not
production yet, so its ok to crash it.
Probably pho's stress2 stuff. Thinks like dbench might be a good start as
well for initial testing.
dbench runs just fine with 20 clients. I am letting stress2's disk
stress test now. The tw_cli seems to run the same when the controller
is super busy with pho's stress
I havent looked at performance differences, but a quick eyeball shows
about the same.
0{offsite2}# tw_cli /c0 show
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache
AVrfy
--
u0RAID-0OK - - 64K 931.521 ON
-
Port Status Unit SizeBlocksSerial
---
p0 OK u0 465.76 GB 976773168 WD-WCAYUEY18298
p1 OK u0 465.76 GB 976773168 WD-WMAYUL256317
0{offsite2}#
Operation CountAvgLatMaxLat
NTCreateX1523290 1.527 2151.921
Close1119090 0.681 2001.144
Rename 64489 3.669 748.957
Unlink307507 3.305 2075.871
Deltree 4035.922 194.337
Mkdir 20 0.014 0.113
Qpathinfo1380911 0.292 637.855
Qfileinfo 242016 0.001 0.201
Qfsinfo 253036 0.006 2.063
Sfileinfo 124125 3.539 1479.315
Find 533771 0.417 1501.775
WriteX759745 0.195 403.113
ReadX2386679 0.033 322.923
LockX 4952 0.004 0.018
UnlockX 4952 0.003 0.240
Flush 10677559.541 2081.524
Throughput 79.6165 MB/sec 20 clients 20 procs max_latency=2151.929 ms
Mon Aug 13 16:38:18 EDT 2012
run: run time 3+00:00:00, incarnations 1, load 100, verbose 1
16:38:18 Loop #1
rw: run time 0+00:02:00, incarnations 17, load 100, verbose 1
creat: run time 0+00:02:00, incarnations 64, load 80, verbose 1
mkdir: run time 0+00:02:00, incarnations 52, load 80, verbose 1
16:40:56 Loop #2
rw: run time 0+00:02:00, incarnations 98, load 100, verbose 1
creat: run time 0+00:02:00, incarnations 28, load 80, verbose 1
mkdir: run time 0+00:02:00, incarnations 34, load 80, verbose 1
16:48:39 Loop #3
rw: run time 0+00:02:00, incarnations 63, load 100, verbose 1
creat: run time 0+00:02:00, incarnations 80, load 80, verbose 1
mkdir: run time 0+00:02:00, incarnations 19, load 80, verbose 1
16:53:59 Loop #4
rw: run time 0+00:02:00, incarnations 11, load 100, verbose 1
creat: run time 0+00:02:00, incarnations 46, load 80, verbose 1
mkdir: run time 0+00:02:00, incarnations 21, load 80, verbose 1
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/12 5:04 PM, Mike Tancsa wrote:
Start up the tw_cli client
0{offsite2}# tw_cli
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c09650SE-2LP 2 21 0 1 1 -
//offsite2 exit
0{offsite2}# ls -l /dev/tw*
crw--- 1 root operator - 0, 37 Aug 9 16:58 /dev/twa0
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
It then disappears ?!
Bizarre, it seems to disappear while tw_cli is running? I'm curious if
'rm -W /dev/twe0' brings it back?
If so, it might be interesting to see a ktrace of tw_cli.
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/10/2012 9:31 AM, John Baldwin wrote:
On 8/9/12 5:04 PM, Mike Tancsa wrote:
Start up the tw_cli client
0{offsite2}# tw_cli
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c09650SE-2LP 2 21 0 1 1 -
//offsite2 exit
0{offsite2}# ls -l /dev/tw*
crw--- 1 root operator - 0, 37 Aug 9 16:58 /dev/twa0
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
It then disappears ?!
Bizarre, it seems to disappear while tw_cli is running? I'm curious if
'rm -W /dev/twe0' brings it back?
nada
0{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# ls -l /dev/twe*
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
If so, it might be interesting to see a ktrace of tw_cli.
I just ran ktrace tw_cli
File is available at
http://www.tancsa.com/misc/ktrace.out
and
http://www.tancsa.com/misc/ktrace.txt
---Mike
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/10/12 9:52 AM, Mike Tancsa wrote:
On 8/10/2012 9:31 AM, John Baldwin wrote:
On 8/9/12 5:04 PM, Mike Tancsa wrote:
Start up the tw_cli client
0{offsite2}# tw_cli
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c09650SE-2LP 2 21 0 1 1 -
//offsite2 exit
0{offsite2}# ls -l /dev/tw*
crw--- 1 root operator - 0, 37 Aug 9 16:58 /dev/twa0
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
It then disappears ?!
Bizarre, it seems to disappear while tw_cli is running? I'm curious if
'rm -W /dev/twe0' brings it back?
nada
0{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# rm -W /dev/twe0
rm: /dev/twe0: No such file or directory
1{offsite2}# ls -l /dev/twe*
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
If so, it might be interesting to see a ktrace of tw_cli.
I just ran ktrace tw_cli
File is available at
http://www.tancsa.com/misc/ktrace.out
and
http://www.tancsa.com/misc/ktrace.txt
5972 tw_cli CALL
__sysctl(0x7fffd798,0x2,0x7fffd7a0,0x7fffd790,0x7fffd960,0x16)
5972 tw_cli SCTL sysctl.name2oid
5972 tw_cli RET __sysctl -1 errno 2 No such file or directory
5972 tw_cli CALL unlink(0x7fffd9d0)
5972 tw_cli NAMI /dev/twe0
5972 tw_cli RET unlink 0
...
Oh! I moved the sysctl's for twe out of hw and under the device node.
That seem to have broken tw_cli. I will revert that and generate an
updated patch.
Try http://www.FreeBSD.org/~jhb/patches/twe_locking2.patch
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/10/2012 10:05 AM, John Baldwin wrote:
5972 tw_cli CALL
__sysctl(0x7fffd798,0x2,0x7fffd7a0,0x7fffd790,0x7fffd960,0x16)
5972 tw_cli SCTL sysctl.name2oid
5972 tw_cli RET __sysctl -1 errno 2 No such file or directory
5972 tw_cli CALL unlink(0x7fffd9d0)
5972 tw_cli NAMI /dev/twe0
5972 tw_cli RET unlink 0
...
Oh! I moved the sysctl's for twe out of hw and under the device node.
That seem to have broken tw_cli. I will revert that and generate an
updated patch.
Try http://www.FreeBSD.org/~jhb/patches/twe_locking2.patch
Looks good! Both the tw_cli and 3dm2 web interface work now.
Doing some simple disk tests show everything to be pretty well the same.
new driver
0{offsite2}# dd if=/dev/zero of=/mnt/test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 14.677410 secs (71441488 bytes/sec)
0{offsite2}# dd if=/mnt/test of=/dev/null bs=1024
1024000+0 records in
1024000+0 records out
1048576000 bytes transferred in 10.325622 secs (101550879 bytes/sec)
0{offsite2}# umount /mnt ; mount /dev/twed0 /mnt
0{offsite2}# dd if=/mnt/test of=/dev/null bs=2048
512000+0 records in
512000+0 records out
1048576000 bytes transferred in 10.347670 secs (101334503 bytes/sec)
0{offsite2}#
old driver
0{offsite2}# dd if=/dev/zero of=/mnt/test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 14.695589 secs (71353111 bytes/sec)
0{offsite2}# umount /mnt ; mount /dev/twed0 /mnt
0{offsite2}# dd if=/mnt/test of=/dev/null bs=2048
512000+0 records in
512000+0 records out
1048576000 bytes transferred in 10.305835 secs (101745856 bytes/sec)
0{offsite2}#
I will do some more stress tests through the day
0{offsite2}# patch -p9 twe_locking2.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe.c 2009-12-25
17:35:14.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe.c 2012-08-03
04:00:49.0
--
Patching file twe.c using Plan A...
Hunk #1 succeeded at 151.
Hunk #2 succeeded at 167.
Hunk #3 succeeded at 189.
Hunk #4 succeeded at 208.
Hunk #5 succeeded at 226.
Hunk #6 succeeded at 234.
Hunk #7 succeeded at 271.
Hunk #8 succeeded at 288.
Hunk #9 succeeded at 310.
Hunk #10 succeeded at 337.
Hunk #11 succeeded at 349.
Hunk #12 succeeded at 405.
Hunk #13 succeeded at 521.
Hunk #14 succeeded at 557.
Hunk #15 succeeded at 580.
Hunk #16 succeeded at 595.
Hunk #17 succeeded at 608.
Hunk #18 succeeded at 646.
Hunk #19 succeeded at 769.
Hunk #20 succeeded at 863.
Hunk #21 succeeded at 921.
Hunk #22 succeeded at 952.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1051.
Hunk #25 succeeded at 1082.
Hunk #26 succeeded at 1104.
Hunk #27 succeeded at 1140.
Hunk #28 succeeded at 1151.
Hunk #29 succeeded at 1177.
Hunk #30 succeeded at 1206.
Hunk #31 succeeded at 1309.
Hunk #32 succeeded at 1447.
Hunk #33 succeeded at 1469.
Hunk #34 succeeded at 1499.
Hunk #35 succeeded at 1513.
Hunk #36 succeeded at 1531.
Hunk #37 succeeded at 1554.
Hunk #38 succeeded at 1589.
Hunk #39 succeeded at 1618.
Hunk #40 succeeded at 1644.
Hunk #41 succeeded at 1696.
Hunk #42 succeeded at 1778.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_compat.h
2005-05-29 04:45:51.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_compat.h 2012-08-03
03:58:12.0
--
Patching file twe_compat.h using Plan A...
Hunk #1 succeeded at 43.
Hunk #2 succeeded at 68.
Hunk #3 succeeded at 82.
Hunk #4 succeeded at 92.
Hunk #5 succeeded at 108.
Hunk #6 succeeded at 128.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_freebsd.c
2012-03-12 08:05:24.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-10
14:04:10.0
--
Patching file twe_freebsd.c using Plan A...
Hunk #1 succeeded at 69.
Hunk #2 succeeded at 83.
Hunk #3 succeeded at 101.
Hunk #4 succeeded at 179.
Hunk #5 succeeded at 189.
Hunk #6 succeeded at 218.
Hunk #7 succeeded at 248.
Hunk #8 succeeded at 299.
Hunk #9 succeeded at 421.
Hunk #10 succeeded at 432.
Hunk #11 succeeded at 468.
Hunk #12 succeeded at 503.
Hunk #13 succeeded at 525.
Hunk #14 succeeded at 540.
Hunk #15 succeeded at 573.
Hunk #16 succeeded at 592.
Hunk #17 succeeded at 611.
Hunk #18 succeeded at 687.
Hunk #19 succeeded at 736.
Hunk #20 succeeded at 841.
Hunk #21 succeeded at 864.
Hunk #22 succeeded at 883.
Hunk #23 succeeded at 1045.
Hunk #24 succeeded at 1104.
Hunk #25 succeeded at 1158.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|---
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/8/2012 2:39 PM, Mike Tancsa wrote: On 8/8/2012 7:27 AM, John Baldwin wrote: Looks like it breaks 3dm2 and the tw_cli. With the patch, I am not able to see the 8006 controller I added. Ugh, ok. A few questions: 1) Does the driver see any attached drives/volumes? Yes 2) If it does, does basic I/O to the drives work? yes 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything, fails in that routine when tw_cli makes a request? Yes, for sure. Let me know what you would like me to add. One more data point, /dev/twe0 does not exist with the patch and I think thats why the utils fail outright. ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/12 8:22 AM, Mike Tancsa wrote: On 8/8/2012 2:39 PM, Mike Tancsa wrote: On 8/8/2012 7:27 AM, John Baldwin wrote: Looks like it breaks 3dm2 and the tw_cli. With the patch, I am not able to see the 8006 controller I added. Ugh, ok. A few questions: 1) Does the driver see any attached drives/volumes? Yes 2) If it does, does basic I/O to the drives work? yes Ok, so that's good. I didn't break anything fundamental with driver commands. 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything, fails in that routine when tw_cli makes a request? Yes, for sure. Let me know what you would like me to add. One more data point, /dev/twe0 does not exist with the patch and I think thats why the utils fail outright. Oh, hmm. That's odd. Do you get any error messages on the console when twe0 attaches? Also, you have INVARIANTS enabled, yes? (make_dev() panics when it fails if INVARIANTS is enabled). Maybe try something like this (relative to the patched driver): --- //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03 18:10:04.0 +++ /Users/jhb/work/p4/cleanup/sys/dev/twe/twe_freebsd.c2012-08-03 18:10:04.0 @@ -342,9 +342,12 @@ /* * Create the control device. */ +device_printf(sc-twe_dev, Calling make_dev()\n); sc-twe_dev_t = make_dev(twe_cdevsw, device_get_unit(sc-twe_dev), UID_ROOT, GID_OPERATOR, S_IRUSR | S_IWUSR, twe%d, device_get_unit(sc-twe_dev)); sc-twe_dev_t-si_drv1 = sc; +device_printf(sc-twe_dev, make_dev() returned %p (%s)\n, sc-twe_dev_t, + sc-twe_dev_t-si_name); /* * Schedule ourselves to bring the controller up once interrupts are available. * This isn't strictly necessary, since we disable interrupts while probing the -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/2012 9:16 AM, John Baldwin wrote:
One more data point, /dev/twe0 does not exist with the patch and I think
thats why the utils fail outright.
Oh, hmm. That's odd. Do you get any error messages on the console
when twe0 attaches? Also, you have INVARIANTS enabled, yes?
(make_dev() panics when it fails if INVARIANTS is enabled).
I have INVARIANTS in the kernel, sorry, do I need to do something to
make it active ?
0{offsite2}# sysctl -a | grep -i invar
kern.features.invariant_support: 1
options INVARIANT_SUPPORT
options INVARIANTS
kern.timecounter.invariant_tsc: 1
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
0{offsite2}#
Nothing odd in dmesg
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
pci6: ACPI PCI bus on pcib6
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe
27f at device 0.0 on pci6
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
isab0: PCI-ISA bridge at device 31.0 on pci0
Patch below is causing a panic now on boot. Just going to add debugging
to the serial console to see what it is and
0{offsite2}# kldload twe
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe27f at device
0.0 on pci6
twe0: [GIANT-LOCKED]
Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address = 0x3
fault code = supervisor read data, page not present
instruction pointer = 0x20:0x81813eb3
stack pointer = 0x28:0xff84529d33f0
frame pointer = 0x28:0xff84529d3430
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 1324 (kldload)
[ thread pid 1324 tid 100146 ]
Stopped at twe_setup+0x153:movzbl 0x3(%rdx),%eax
db
Maybe try something like this (relative to the patched driver):
--- //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c2012-08-03
18:10:04.0
+++ /Users/jhb/work/p4/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
18:10:04.0
@@ -342,9 +342,12 @@
/*
* Create the control device.
*/
+device_printf(sc-twe_dev, Calling make_dev()\n);
sc-twe_dev_t = make_dev(twe_cdevsw, device_get_unit(sc-twe_dev),
UID_ROOT, GID_OPERATOR,
S_IRUSR | S_IWUSR, twe%d,
device_get_unit(sc-twe_dev));
sc-twe_dev_t-si_drv1 = sc;
+device_printf(sc-twe_dev, make_dev() returned %p (%s)\n,
sc-twe_dev_t,
+ sc-twe_dev_t-si_name);
/*
* Schedule ourselves to bring the controller up once interrupts
are available.
* This isn't strictly necessary, since we disable interrupts while
probing the
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/12 10:31 AM, Mike Tancsa wrote:
On 8/9/2012 9:16 AM, John Baldwin wrote:
One more data point, /dev/twe0 does not exist with the patch and I think
thats why the utils fail outright.
Oh, hmm. That's odd. Do you get any error messages on the console
when twe0 attaches? Also, you have INVARIANTS enabled, yes?
(make_dev() panics when it fails if INVARIANTS is enabled).
I have INVARIANTS in the kernel, sorry, do I need to do something to
make it active ?
0{offsite2}# sysctl -a | grep -i invar
kern.features.invariant_support: 1
options INVARIANT_SUPPORT
options INVARIANTS
kern.timecounter.invariant_tsc: 1
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
TSC: P-state invariant, performance statistics
0{offsite2}#
Nothing odd in dmesg
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
pci6: ACPI PCI bus on pcib6
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe
27f at device 0.0 on pci6
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
isab0: PCI-ISA bridge at device 31.0 on pci0
Patch below is causing a panic now on boot. Just going to add debugging
to the serial console to see what it is and
0{offsite2}# kldload twe
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe27f at device
0.0 on pci6
twe0: [GIANT-LOCKED]
Odd, the patched driver shouldn't be triggering this message.
Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address = 0x3
fault code = supervisor read data, page not present
instruction pointer = 0x20:0x81813eb3
stack pointer = 0x28:0xff84529d33f0
frame pointer = 0x28:0xff84529d3430
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 1324 (kldload)
[ thread pid 1324 tid 100146 ]
Stopped at twe_setup+0x153:movzbl 0x3(%rdx),%eax
db
Humm, I wonder if this module has a mix of old and new .o files somehow?
Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?
--
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/2012 1:21 PM, John Baldwin wrote:
Humm, I wonder if this module has a mix of old and new .o files somehow?
Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?
I think you are right. I nuked all the kernel files and recompiled
again, and it no longer panics and I see the entry in /dev now!?
0{offsite2}# kldload twe
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe27f at device
0.0 on pci6
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
twe0: Calling make_dev()
twe0: make_dev() returned 0xfe0122105200 (twe0)
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
0{offsite2}#
---Mike
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/9/2012 3:28 PM, Mike Tancsa wrote:
On 8/9/2012 1:21 PM, John Baldwin wrote:
Humm, I wonder if this module has a mix of old and new .o files somehow?
Perhaps try rebuilding twe.ko from scratch after doing a 'make clean'?
I think you are right. I nuked all the kernel files and recompiled
again, and it no longer panics and I see the entry in /dev now!?
0{offsite2}# kldload twe
twe0: 3ware Storage Controller. Driver version 1.50.01.002 port
0x2000-0x200f mem 0xe281-0xe281000f,0xe200-0xe27f at device
0.0 on pci6
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
twe0: Calling make_dev()
twe0: make_dev() returned 0xfe0122105200 (twe0)
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
0{offsite2}#
OK, some more mystery.
If I load it as a kld,
I see the device. (Note, I added MDT into the version string)
0{offsite2}# twe0: 3ware Storage Controller. Driver version
1.50.01.002MDT port 0x2000-0x200f mem
0xe281-0xe281000f,0xe200-0xe27f at device 0.0 on pci6
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
twe0: Calling make_dev()
twe0: make_dev() returned 0xfe0026581a00 (twe0)
twed0 on twe0
twed0: 953878MB (1953542144 sectors)
0{offsite2}# ls -l /dev/tw*
crw--- 1 root operator - 0, 37 Aug 9 16:58 /dev/twa0
crw--- 1 root operator - 0, 173 Aug 9 17:00 /dev/twe0
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
Start up the tw_cli client
0{offsite2}# tw_cli
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c09650SE-2LP 2 21 0 1 1 -
//offsite2 exit
0{offsite2}# ls -l /dev/tw*
crw--- 1 root operator - 0, 37 Aug 9 16:58 /dev/twa0
crw-r- 1 root operator - 0, 174 Aug 9 17:00 /dev/twed0
0{offsite2}#
It then disappears ?!
---Mike
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On Tuesday, August 07, 2012 10:11:01 AM Mike Tancsa wrote: On 8/3/2012 5:26 PM, John Baldwin wrote: If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload (iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives (power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see that all/some of the failure paths are properly stimulated). 3dm2 testing would be good for the ioctl handling, but the most critical tests are basic I/O. Looks like it breaks 3dm2 and the tw_cli. With the patch, I am not able to see the 8006 controller I added. Ugh, ok. A few questions: 1) Does the driver see any attached drives/volumes? 2) If it does, does basic I/O to the drives work? 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything, fails in that routine when tw_cli makes a request? Thanks. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/8/2012 7:27 AM, John Baldwin wrote: Looks like it breaks 3dm2 and the tw_cli. With the patch, I am not able to see the 8006 controller I added. Ugh, ok. A few questions: 1) Does the driver see any attached drives/volumes? Yes 2) If it does, does basic I/O to the drives work? yes 3) Can you add some debugging printfs to twe_ioctl() to see what, if anything, fails in that routine when tw_cli makes a request? Yes, for sure. Let me know what you would like me to add. ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/3/2012 5:26 PM, John Baldwin wrote:
If there's a tool for poking at the drives/controller, I would use
that, plus camcontrol. Of course you want a data intensive workload
(iometer/iozone/xdd with async and sync mode, random reads and sequential
reads, etc), and maybe resort to manual testing like pulling drives
(power, data) if you don't mind creating failures. If you have some
failed/failing drives kicking around, that would be a good test as well (see
that all/some of the failure paths are properly stimulated).
3dm2 testing would be good for the ioctl handling, but the most critical
tests are basic I/O.
Looks like it breaks 3dm2 and the tw_cli. With the patch, I am not able
to see the 8006 controller I added.
tw_cli without the patch
0{offsite2}# tw_cli
//offsite2 /c0 show
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache
AVrfy
--
u0RAID-0OK - - 64K 931.521 ON
-
Port Status Unit SizeBlocksSerial
---
p0 OK u0 465.76 GB 976773168 WD-WCAYUEY18298
p1 OK u0 465.76 GB 976773168 WD-WMAYUL256317
//offsite2
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c08006-2LP 2 21 0 3 - -
c19650SE-2LP 2 21 0 1 1 -
The boot array is c1 (the off twa controller)
with the patch (invariants and invariants_support in the kernel)
0{offsite2}# tw_cli
//offsite2 show
Ctl Model(V)Ports Drives Units NotOpt RRate VRate BBU
c09650SE-2LP 2 21 0 1 1 -
//offsite2
---Mike
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[PATCH] Add locking to twe(4) so it no longer uses Giant
We only have a few storage drivers left that use Giant and twe(4) is one of them. I don't have any hardware to test with, however. I have verified the patch compiles, but I'd appreciate it if some folks with twe(4) hardware could test it. The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
I'll try to find time to try it later today, but I'm in the middle of budget wrangling and I can't make any promises. Before I try, will these patches apply to the twe driver in 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18. I really don't have time to install current right now. -- R. Kevin Oberman, Network Engineer E-mail: kob6...@gmail.com On Fri, Aug 3, 2012 at 11:18 AM, John Baldwin j...@freebsd.org wrote: We only have a few storage drivers left that use Giant and twe(4) is one of them. I don't have any hardware to test with, however. I have verified the patch compiles, but I'd appreciate it if some folks with twe(4) hardware could test it. The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote: I'll try to find time to try it later today, but I'm in the middle of budget wrangling and I can't make any promises. Before I try, will these patches apply to the twe driver in 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18. I really don't have time to install current right now. I believe the patches should apply fine on 8 and 9. If you use it on 8 or 9 please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for initial testing. Thanks! -- R. Kevin Oberman, Network Engineer E-mail: kob6...@gmail.com On Fri, Aug 3, 2012 at 11:18 AM, John Baldwin j...@freebsd.org wrote: We only have a few storage drivers left that use Giant and twe(4) is one of them. I don't have any hardware to test with, however. I have verified the patch compiles, but I'd appreciate it if some folks with twe(4) hardware could test it. The patch is at http://www.FreeBSD.org/~jhb/patches/twe_locking.patch -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On 8/3/2012 3:39 PM, John Baldwin wrote:
On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote:
I'll try to find time to try it later today, but I'm in the middle of
budget wrangling and I can't make any promises.
Before I try, will these patches apply to the twe driver in
9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18.
I really don't have time to install current right now.
I believe the patches should apply fine on 8 and 9. If you use it on 8 or 9
please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for
initial testing. Thanks!
Seems to apply to RELENG_9 just fine. Are there any stress tests you
suggest I run that might expose some bugs ? The machine is not
production yet, so its ok to crash it.
{offsite2}# patch -p9 twe_locking.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe.c 2009-12-25
17:35:14.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe.c 2012-08-03
04:00:49.0
--
Patching file twe.c using Plan A...
Hunk #1 succeeded at 151.
Hunk #2 succeeded at 167.
Hunk #3 succeeded at 189.
Hunk #4 succeeded at 208.
Hunk #5 succeeded at 226.
Hunk #6 succeeded at 234.
Hunk #7 succeeded at 271.
Hunk #8 succeeded at 288.
Hunk #9 succeeded at 310.
Hunk #10 succeeded at 337.
Hunk #11 succeeded at 349.
Hunk #12 succeeded at 405.
Hunk #13 succeeded at 521.
Hunk #14 succeeded at 557.
Hunk #15 succeeded at 580.
Hunk #16 succeeded at 595.
Hunk #17 succeeded at 608.
Hunk #18 succeeded at 646.
Hunk #19 succeeded at 769.
Hunk #20 succeeded at 863.
Hunk #21 succeeded at 921.
Hunk #22 succeeded at 952.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1051.
Hunk #25 succeeded at 1082.
Hunk #26 succeeded at 1104.
Hunk #27 succeeded at 1140.
Hunk #28 succeeded at 1151.
Hunk #29 succeeded at 1177.
Hunk #30 succeeded at 1206.
Hunk #31 succeeded at 1309.
Hunk #32 succeeded at 1447.
Hunk #33 succeeded at 1469.
Hunk #34 succeeded at 1499.
Hunk #35 succeeded at 1513.
Hunk #36 succeeded at 1531.
Hunk #37 succeeded at 1554.
Hunk #38 succeeded at 1589.
Hunk #39 succeeded at 1618.
Hunk #40 succeeded at 1644.
Hunk #41 succeeded at 1696.
Hunk #42 succeeded at 1778.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_compat.h
2005-05-29 04:45:51.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_compat.h 2012-08-03
03:58:12.0
--
Patching file twe_compat.h using Plan A...
Hunk #1 succeeded at 43.
Hunk #2 succeeded at 68.
Hunk #3 succeeded at 82.
Hunk #4 succeeded at 92.
Hunk #5 succeeded at 108.
Hunk #6 succeeded at 128.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twe_freebsd.c
2012-03-12 08:05:24.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twe_freebsd.c 2012-08-03
18:10:04.0
--
Patching file twe_freebsd.c using Plan A...
Hunk #1 succeeded at 69.
Hunk #2 succeeded at 83.
Hunk #3 succeeded at 101.
Hunk #4 succeeded at 180.
Hunk #5 succeeded at 188.
Hunk #6 succeeded at 211.
Hunk #7 succeeded at 241.
Hunk #8 succeeded at 292.
Hunk #9 succeeded at 414.
Hunk #10 succeeded at 425.
Hunk #11 succeeded at 461.
Hunk #12 succeeded at 496.
Hunk #13 succeeded at 518.
Hunk #14 succeeded at 533.
Hunk #15 succeeded at 566.
Hunk #16 succeeded at 585.
Hunk #17 succeeded at 604.
Hunk #18 succeeded at 680.
Hunk #19 succeeded at 729.
Hunk #20 succeeded at 834.
Hunk #21 succeeded at 857.
Hunk #22 succeeded at 876.
Hunk #23 succeeded at 1038.
Hunk #24 succeeded at 1097.
Hunk #25 succeeded at 1151.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--
|--- //depot/vendor/freebsd/src/sys/dev/twe/twevar.h2009-12-25
17:35:14.0
|+++ //depot/user/jhb/cleanup/sys/dev/twe/twevar.h 2012-08-03
04:00:49.0
--
Patching file twevar.h using Plan A...
Hunk #1 succeeded at 134.
Hunk #2 succeeded at 210.
Hunk #3 succeeded at 255.
done
0{offsite2}#
--
---
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, m...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On Aug 3, 2012, at 1:42 PM, Mike Tancsa wrote: On 8/3/2012 3:39 PM, John Baldwin wrote: On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote: I'll try to find time to try it later today, but I'm in the middle of budget wrangling and I can't make any promises. Before I try, will these patches apply to the twe driver in 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18. I really don't have time to install current right now. I believe the patches should apply fine on 8 and 9. If you use it on 8 or 9 please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for initial testing. Thanks! Seems to apply to RELENG_9 just fine. Are there any stress tests you suggest I run that might expose some bugs ? The machine is not production yet, so its ok to crash it. Looking really quickly at the patch, it looks like most of the locking changes are made around management pieces, and not data handling pieces (but I might have missed something). If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload (iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives (power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see that all/some of the failure paths are properly stimulated). Any variance or combining of these tests will help build confidence in the change being proposed. HTH, -Garrett___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On Friday, August 03, 2012 4:42:59 pm Mike Tancsa wrote: On 8/3/2012 3:39 PM, John Baldwin wrote: On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote: I'll try to find time to try it later today, but I'm in the middle of budget wrangling and I can't make any promises. Before I try, will these patches apply to the twe driver in 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18. I really don't have time to install current right now. I believe the patches should apply fine on 8 and 9. If you use it on 8 or 9 please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for initial testing. Thanks! Seems to apply to RELENG_9 just fine. Are there any stress tests you suggest I run that might expose some bugs ? The machine is not production yet, so its ok to crash it. Probably pho's stress2 stuff. Thinks like dbench might be a good start as well for initial testing. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add locking to twe(4) so it no longer uses Giant
On Friday, August 03, 2012 5:17:09 pm Garrett Cooper wrote: On Aug 3, 2012, at 1:42 PM, Mike Tancsa wrote: On 8/3/2012 3:39 PM, John Baldwin wrote: On Friday, August 03, 2012 2:56:00 pm Kevin Oberman wrote: I'll try to find time to try it later today, but I'm in the middle of budget wrangling and I can't make any promises. Before I try, will these patches apply to the twe driver in 9.1-PRERELEASE? My system with twe was updated to RELENG_9 on July 18. I really don't have time to install current right now. I believe the patches should apply fine on 8 and 9. If you use it on 8 or 9 please make sure to enable INVARIANTS and INVARIANT_SUPPORT at least for initial testing. Thanks! Seems to apply to RELENG_9 just fine. Are there any stress tests you suggest I run that might expose some bugs ? The machine is not production yet, so its ok to crash it. Looking really quickly at the patch, it looks like most of the locking changes are made around management pieces, and not data handling pieces (but I might have missed something). No, it changes those paths, too. There just aren't many of them. All the data enters through twed_strategy() and leaves via twed_intr() (invoked from twe_intr()). If there's a tool for poking at the drives/controller, I would use that, plus camcontrol. Of course you want a data intensive workload (iometer/iozone/xdd with async and sync mode, random reads and sequential reads, etc), and maybe resort to manual testing like pulling drives (power, data) if you don't mind creating failures. If you have some failed/failing drives kicking around, that would be a good test as well (see that all/some of the failure paths are properly stimulated). 3dm2 testing would be good for the ioctl handling, but the most critical tests are basic I/O. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
