The iostat I posted previously was from a system we had already tuned the
zfs:zfs_vdev_max_pending depth down to 10 (as visible by the max of about 10 in
actv per disk).
I reset this value in /etc/system to 7, rebooted, and started a scrub. iostat
output showed busier disks (%b is higher, which seemed odd) but a cap of about
7 queue items per disk, proving the tuning was effective. iostat at a
high-water mark during the test looked like this:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c8t3d0
8344.5 0.0 359640.4 0.0 0.1 300.5 0.0 36.0 0 4362 c9
190.0 0.0 6800.4 0.0 0.0 6.6 0.0 34.8 0 99 c9t8d0
185.0 0.0 6917.1 0.0 0.0 6.1 0.0 32.9 0 94 c9t9d0
187.0 0.0 6640.9 0.0 0.0 6.5 0.0 34.6 0 98 c9t10d0
186.5 0.0 6543.4 0.0 0.0 7.0 0.0 37.5 0 100 c9t11d0
180.5 0.0 7203.1 0.0 0.0 6.7 0.0 37.2 0 100 c9t12d0
195.5 0.0 7352.4 0.0 0.0 7.0 0.0 35.8 0 100 c9t13d0
188.0 0.0 6884.9 0.0 0.0 6.6 0.0 35.2 0 99 c9t14d0
204.0 0.0 6990.1 0.0 0.0 7.0 0.0 34.3 0 100 c9t15d0
199.0 0.0 7336.7 0.0 0.0 7.0 0.0 35.2 0 100 c9t16d0
180.5 0.0 6837.9 0.0 0.0 7.0 0.0 38.8 0 100 c9t17d0
198.0 0.0 7668.9 0.0 0.0 7.0 0.0 35.3 0 100 c9t18d0
203.0 0.0 7983.2 0.0 0.0 7.0 0.0 34.5 0 100 c9t19d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t20d0
195.5 0.0 7096.4 0.0 0.0 6.7 0.0 34.1 0 98 c9t21d0
189.5 0.0 7757.2 0.0 0.0 6.4 0.0 33.9 0 97 c9t22d0
195.5 0.0 7645.9 0.0 0.0 6.6 0.0 33.8 0 99 c9t23d0
194.5 0.0 7925.9 0.0 0.0 7.0 0.0 36.0 0 100 c9t24d0
188.5 0.0 6725.6 0.0 0.0 6.2 0.0 32.8 0 94 c9t25d0
188.5 0.0 7199.6 0.0 0.0 6.5 0.0 34.6 0 98 c9t26d0
196.0 0.0 6666.9 0.0 0.0 6.3 0.0 32.1 0 95 c9t27d0
193.5 0.0 7455.4 0.0 0.0 6.2 0.0 32.0 0 95 c9t28d0
189.0 0.0 7400.9 0.0 0.0 6.3 0.0 33.2 0 96 c9t29d0
182.5 0.0 9397.0 0.0 0.0 7.0 0.0 38.3 0 100 c9t30d0
192.5 0.0 9179.5 0.0 0.0 7.0 0.0 36.3 0 100 c9t31d0
189.5 0.0 9431.8 0.0 0.0 7.0 0.0 36.9 0 100 c9t32d0
187.5 0.0 9082.0 0.0 0.0 7.0 0.0 37.3 0 100 c9t33d0
188.5 0.0 9368.8 0.0 0.0 7.0 0.0 37.1 0 100 c9t34d0
180.5 0.0 9332.8 0.0 0.0 7.0 0.0 38.8 0 100 c9t35d0
183.0 0.0 9690.3 0.0 0.0 7.0 0.0 38.2 0 100 c9t36d0
186.0 0.0 9193.8 0.0 0.0 7.0 0.0 37.6 0 100 c9t37d0
180.5 0.0 8233.4 0.0 0.0 7.0 0.0 38.8 0 100 c9t38d0
175.5 0.0 9085.2 0.0 0.0 7.0 0.0 39.9 0 100 c9t39d0
177.0 0.0 9340.0 0.0 0.0 7.0 0.0 39.5 0 100 c9t40d0
175.5 0.0 8831.0 0.0 0.0 7.0 0.0 39.9 0 100 c9t41d0
190.5 0.0 9177.8 0.0 0.0 7.0 0.0 36.7 0 100 c9t42d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c9t43d0
196.0 0.0 9180.5 0.0 0.0 7.0 0.0 35.7 0 100 c9t44d0
193.5 0.0 9496.8 0.0 0.0 7.0 0.0 36.2 0 100 c9t45d0
187.0 0.0 8699.5 0.0 0.0 7.0 0.0 37.4 0 100 c9t46d0
198.5 0.0 9277.0 0.0 0.0 7.0 0.0 35.2 0 100 c9t47d0
185.5 0.0 9778.3 0.0 0.0 7.0 0.0 37.7 0 100 c9t48d0
192.0 0.0 8384.2 0.0 0.0 7.0 0.0 36.4 0 100 c9t49d0
198.5 0.0 8864.7 0.0 0.0 7.0 0.0 35.2 0 100 c9t50d0
192.0 0.0 9369.8 0.0 0.0 7.0 0.0 36.4 0 100 c9t51d0
182.5 0.0 8825.7 0.0 0.0 7.0 0.0 38.3 0 100 c9t52d0
202.0 0.0 7387.9 0.0 0.0 7.0 0.0 34.6 0 100 c9t55d0
...and sure enough about 20 minutes into it I get this (bus reset?):
scsi: [ID 107833 kern.warning] WARNING:
/p...@0,0/pci8086,6...@4/pci1000,3...@0/s...@34,0 (sd49):
incomplete read- retrying
scsi: [ID 107833 kern.warning] WARNING:
/p...@0,0/pci8086,6...@4/pci1000,3...@0/s...@21,0 (sd30):
incomplete read- retrying
scsi: [ID 107833 kern.warning] WARNING:
/p...@0,0/pci8086,6...@4/pci1000,3...@0/s...@1e,0 (sd27):
incomplete read- retrying
scsi: [ID 365881 kern.info] /p...@0,0/pci8086,6...@4/pci1000,3...@0 (mpt0):
Rev. 8 LSI, Inc. 1068E found.
scsi: [ID 365881 kern.info] /p...@0,0/pci8086,6...@4/pci1000,3...@0 (mpt0):
mpt0 supports power management.
scsi: [ID 365881 kern.info] /p...@0,0/pci8086,6...@4/pci1000,3...@0 (mpt0):
mpt0: IOC Operational.
During the "bus reset", iostat output looked like this:
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot
device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c8
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c8t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c8t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c8t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c8t3d0
0.0 0.0 0.0 0.0 0.0 88.0 0.0 0.0 0 2200 0 3 0 3 c9
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t8d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t9d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t10d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t11d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t12d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t13d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t14d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t15d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t16d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t17d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t18d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t19d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t20d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t21d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t22d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t23d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t24d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t25d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t26d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t27d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t28d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t29d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 1 0 1
c9t30d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t31d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t32d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 1 0 1
c9t33d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t34d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t35d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t36d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t37d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t38d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t39d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t40d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t41d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t42d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t43d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t44d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t45d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t46d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t47d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t48d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t49d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t50d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0 0 0
c9t51d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 1 0 1
c9t52d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0
c9t55d0
During our previous testing, we had tried even setting this max_pending value
down to 1, but we still hit the problem (albeit it took a little longer to hit
it) and I couldn't find anything else I could set to throttle IO to the disk,
hence the frustration.
If you hadn't seen this output, would you say that 7 was a "reasonable" value
for that max_pending queue for our architecture and should give the LSI
controller in this situation enough breathing room to operate? If so, I
*should* be able to scrub the disks successfully (ZFS isn't to blame) and
therefore have to point the finger at the mpt-driver/LSI-firmware/disk-firmware
instead.
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
