Re: [zfs-discuss] zfs sata mirror slower than single disk
sorry to insist, but still no real answer... On Mon, 16 Jul 2012, Bob Friesenhahn wrote: On Tue, 17 Jul 2012, Michael Hase wrote: So only one thing left: mirror should read 2x I don't think that mirror should necessarily read 2x faster even though the potential is there to do so. Last I heard, zfs did not include a special read scheduler for sequential reads from a mirrored pair. As a result, 50% of the time, a read will be scheduled for a device which already has a read scheduled. If this is indeed true, the typical performance would be 150%. There may be some other scheduling factor (e.g. estimate of busyness) which might still allow zfs to select the right side and do better than that. If you were to add a second vdev (i.e. stripe) then you should see very close to 200% due to the default round-robin scheduling of the writes. My expectation would be 200%, as 4 disks are involved. It may not be the perfect 4x scaling, but imho it should be (and is for a scsi system) more than half of the theoretical throughput. This is solaris or a solaris derivative, not linux ;-) It is really difficult to measure zfs read performance due to caching effects. One way to do it is to write a large file (containing random data such as returned from /dev/urandom) to a zfs filesystem, unmount the filesystem, remount the filesystem, and then time how long it takes to read the file once. The reason why this works is because remounting the filesystem restarts the filesystem cache. Ok, did a zpool export/import cycle between the dd read and write test. This really empties the arc, checked this with arc_summary.pl. the test even uses two processes in parallel (doesn't make a difference). Result is still the same: dd write: 2x 58 MB/sec -- perfect, each disk does 110 MB/sec dd read: 2x 68 MB/sec -- imho too slow, about 68 MB/sec per disk For writes each disk gets 900 128k io requests/sec with asvc_t in the 8-9 msec range. For reads each disk only gets 500 io requests/sec, asvc_t 18-20 msec with the default zfs_vdev_maxpending=10. When reducing zfs_vdev_maxpending the asvc_t drops accordingly, the i/o rate remains at 500/sec per disk, throughput also the same. I think iostat values should be reliable here. These high iops numbers make sense as we work on empty pools so there aren't very high seek times. All benchmarks (dd, bonnie, will try iozone) lead to the same result: on the sata mirror pair read performance is in the range of a single disk. For the sas disks (only two available for testing) and for the scsi system there is quite good throughput scaling. Here for comparison a table for 1-4 36gb 15k u320 scsi disks on an old sxde box (nevada b130): seq write factor seq read factor MB/sec MB/sec single821 78 1 mirror791137 1.75 2x mirror1201.5 251 3.2 This is exactly what's imho to be expected from mirrors and striped mirrors. It just doesn't happen for my sata pool. Still have no reference numbers for other sata pools, just one with the 4k/512bytes sector problem which is even slower than mine. It seems the zfs performance people just use sas disks and be done. Michael Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ old ibm dual opteron intellistation with external hp msa30, 36gb 15k u320 scsi disks pool: scsi1 state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM scsi1 ONLINE 0 0 0 c3t4d0ONLINE 0 0 0 errors: No known data errors Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP zfssingle 16G 137 99 82739 20 39453 9 314 99 78251 7 856.9 8 Latency 160ms4799ms5292ms 43210us3274ms2069ms Version 1.96 --Sequential Create-- Random Create zfssingle -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8819 34 + +++ 26318 68 20390 73 + +++ 26846 72 Latency 16413us 108us 231us 12206us 46us 124us 1.96,1.96,zfssingle,1,1342514790,16G,,137,99,82739,20,39453,9,314,99,78251,7,856.9,8,16,8819,34,+,+++,26318,68,20390,73,+,+++,26846,72,160ms,4799ms,5292ms,43210us,3274ms,2069ms,16413us,108us,231us,12206us,46us,124us ## pool: scsi1 state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM scsi1 ONLINE 0
Re: [zfs-discuss] zfs sata mirror slower than single disk
On Tue, 17 Jul 2012, Michael Hase wrote: If you were to add a second vdev (i.e. stripe) then you should see very close to 200% due to the default round-robin scheduling of the writes. My expectation would be 200%, as 4 disks are involved. It may not be the perfect 4x scaling, but imho it should be (and is for a scsi system) more than half of the theoretical throughput. This is solaris or a solaris derivative, not linux ;-) Here are some results from my own machine based on the 'virgin mount' test approach. The results show less boost than is reported by a benchmark tool like 'iozone' which sees benefits from caching. I get an initial sequential read speed of 657 MB/s on my new pool which has 1200 MB/s of raw bandwidth (if mirrors could produce 100% boost). Reading the file a second time reports 6.9 GB/s. The below is with a 2.6 GB test file but with a 26 GB test file (just add another zero to 'count' and wait longer) I see an initial read rate of 618 MB/s and a re-read rate of 8.2 GB/s. The raw disk can transfer 150 MB/s. % zpool status pool: tank state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scan: scrub repaired 0 in 0h10m with 0 errors on Mon Jul 16 04:30:48 2012 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0ONLINE 0 0 0 c7t5393E8CA21FAd0p0 ONLINE 0 0 0 c11t5393D8CA34B2d0p0 ONLINE 0 0 0 mirror-1ONLINE 0 0 0 c8t5393E8CA2066d0p0 ONLINE 0 0 0 c12t5393E8CA2196d0p0 ONLINE 0 0 0 mirror-2ONLINE 0 0 0 c9t5393D8CA82A2d0p0 ONLINE 0 0 0 c13t5393E8CA2116d0p0 ONLINE 0 0 0 mirror-3ONLINE 0 0 0 c10t5393D8CA59C2d0p0 ONLINE 0 0 0 c14t5393D8CA828Ed0p0 ONLINE 0 0 0 errors: No known data errors % pfexec zfs create tank/zfstest % pfexec zfs create tank/zfstest/defaults % cd /tank/zfstest/defaults % pfexec dd if=/dev/urandom of=random.dat bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 36.8133 s, 71.2 MB/s % cd .. % pfexec zfs umount tank/zfstest/defaults % pfexec zfs mount tank/zfstest/defaults % cd defaults % dd if=random.dat of=/dev/null bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 3.99229 s, 657 MB/s % pfexec dd if=/dev/rdsk/c7t5393E8CA21FAd0p0 of=/dev/null bs=128k count=2000 2000+0 records in 2000+0 records out 262144000 bytes (262 MB) copied, 1.74532 s, 150 MB/s % bc scale=8 657/150 4.3800 It is very difficult to benchmark with a cache which works so well: % dd if=random.dat of=/dev/null bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 0.379147 s, 6.9 GB/s Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Problem: Disconnected command timeout for Target X
Hi all, I'm using Opensolaris snv_134 with LSI Controllers and a motherboard supermicro, with 20 sata disks, zfs in raid-10 conf. I mounted this zfs_storage with NFS. I'm not opensolaris specialist. What're the commands to show hardware information? Like 'lshw' in linux but for opensolaris. The storage stopped working, but ping responds. SSH and NFS is out. When I open the console showing this messages: Jul 2 13:00:27 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:00:27 storageDisconnected command timeout for Target 4 Jul 2 13:01:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:01:28 storageDisconnected command timeout for Target 3 Jul 2 13:02:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:02:28 storageDisconnected command timeout for Target 2 Jul 2 13:03:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:03:29 storageDisconnected command timeout for Target 1 Jul 2 13:04:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:04:29 storageDisconnected command timeout for Target 0 Jul 2 13:05:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:05:40 storageDisconnected command timeout for Target 6 Jul 2 13:06:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:06:40 storageDisconnected command timeout for Target 5 Any ideas? Could help me? -- Roberto Scudeller ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Problem: Disconnected command timeout for Target X
On Tue, 17 Jul 2012, Roberto Scudeller wrote: Hi all, I'm using Opensolaris snv_134 with LSI Controllers and a motherboard supermicro, with 20 sata disks, zfs in raid-10 conf. I mounted this zfs_storage with NFS. I'm not opensolaris specialist. What're the commands to show hardware information? Like 'lshw' in linux but for opensolaris. cfgadm, prtconf, prtpicl, prtdiag zpool status fmadm faulty It sounds like you may have a broken cable or power supply failure to some disks. Bob The storage stopped working, but ping responds. SSH and NFS is out. When I open the console showing this messages: Jul 2 13:00:27 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:00:27 storage Disconnected command timeout for Target 4 Jul 2 13:01:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:01:28 storage Disconnected command timeout for Target 3 Jul 2 13:02:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:02:28 storage Disconnected command timeout for Target 2 Jul 2 13:03:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:03:29 storage Disconnected command timeout for Target 1 Jul 2 13:04:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:04:29 storage Disconnected command timeout for Target 0 Jul 2 13:05:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:05:40 storage Disconnected command timeout for Target 6 Jul 2 13:06:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,340a@3/pci1000,3140@0 (mpt2): Jul 2 13:06:40 storage Disconnected command timeout for Target 5 Any ideas? Could help me? -- Roberto Scudeller -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs sata mirror slower than single disk
On Tue, 17 Jul 2012, Bob Friesenhahn wrote: On Tue, 17 Jul 2012, Michael Hase wrote: If you were to add a second vdev (i.e. stripe) then you should see very close to 200% due to the default round-robin scheduling of the writes. My expectation would be 200%, as 4 disks are involved. It may not be the perfect 4x scaling, but imho it should be (and is for a scsi system) more than half of the theoretical throughput. This is solaris or a solaris derivative, not linux ;-) Here are some results from my own machine based on the 'virgin mount' test approach. The results show less boost than is reported by a benchmark tool like 'iozone' which sees benefits from caching. I get an initial sequential read speed of 657 MB/s on my new pool which has 1200 MB/s of raw bandwidth (if mirrors could produce 100% boost). Reading the file a second time reports 6.9 GB/s. The below is with a 2.6 GB test file but with a 26 GB test file (just add another zero to 'count' and wait longer) I see an initial read rate of 618 MB/s and a re-read rate of 8.2 GB/s. The raw disk can transfer 150 MB/s. To work around these caching effects just use a file 2 times the size of ram, iostat then shows the numbers really coming from disk. I always test like this. a re-read rate of 8.2 GB/s is really just memory bandwidth, but quite impressive ;-) % pfexec zfs create tank/zfstest/defaults % cd /tank/zfstest/defaults % pfexec dd if=/dev/urandom of=random.dat bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 36.8133 s, 71.2 MB/s % cd .. % pfexec zfs umount tank/zfstest/defaults % pfexec zfs mount tank/zfstest/defaults % cd defaults % dd if=random.dat of=/dev/null bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 3.99229 s, 657 MB/s % pfexec dd if=/dev/rdsk/c7t5393E8CA21FAd0p0 of=/dev/null bs=128k count=2000 2000+0 records in 2000+0 records out 262144000 bytes (262 MB) copied, 1.74532 s, 150 MB/s % bc scale=8 657/150 4.3800 It is very difficult to benchmark with a cache which works so well: % dd if=random.dat of=/dev/null bs=128k count=2 2+0 records in 2+0 records out 262144 bytes (2.6 GB) copied, 0.379147 s, 6.9 GB/s This is not my point, I'm pretty sure I did not measure any arc effects - maybe with the one exception of the raid0 test on the scsi array. Don't know why the arc had this effect, filesize was 2x of ram. The point is: I'm searching for an explanation for the relative slowness of a mirror pair of sata disks, or some tuning knobs, or something like the disks are plain crap, or maybe: zfs throttles sata disks in general (don't know the internals). In the range of 600 MB/s other issues may show up (pcie bus contention, hba contention, cpu load). And performance at this level could be just good enough, not requiring any further tuning. Could you recheck with only 4 disks (2 mirror pairs)? If you just get some 350 MB/s it could be the same problem as with my boxes. All sata disks? Michael Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs sata mirror slower than single disk
On Tue, 17 Jul 2012, Michael Hase wrote: The below is with a 2.6 GB test file but with a 26 GB test file (just add another zero to 'count' and wait longer) I see an initial read rate of 618 MB/s and a re-read rate of 8.2 GB/s. The raw disk can transfer 150 MB/s. To work around these caching effects just use a file 2 times the size of ram, iostat then shows the numbers really coming from disk. I always test like this. a re-read rate of 8.2 GB/s is really just memory bandwidth, but quite impressive ;-) Yes, in the past I have done benchmarking with file size 2X the size of memory. This does not necessary erase all caching because the ARC is smart enough not to toss everything. At the moment I have an iozone benchark run up from 8 GB to 256 GB file size. I see that it has started the 256 GB size now. It may be a while. Maybe a day. In the range of 600 MB/s other issues may show up (pcie bus contention, hba contention, cpu load). And performance at this level could be just good enough, not requiring any further tuning. Could you recheck with only 4 disks (2 mirror pairs)? If you just get some 350 MB/s it could be the same problem as with my boxes. All sata disks? Unfortunately, I already put my pool into use and can not conveniently destroy it now. The disks I am using are SAS (7200 RPM, 1 GB) but return similar per-disk data rates as the SATA disks I use for the boot pool. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Largest zpool / Most Vdevs
While I'm aware of the published maximums of zfs, I was wondering if anyone could share information on the largest zpool (both total usable capacity and vdevs) they've seen/deployed? I know that generally speaking more vdevs = more iops, but I would think that there is a point of diminishing returns. Any opinions on this? Cheers, Matt Hardy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs sata mirror slower than single disk
On Tue, 17 Jul 2012, Michael Hase wrote: To work around these caching effects just use a file 2 times the size of ram, iostat then shows the numbers really coming from disk. I always test like this. a re-read rate of 8.2 GB/s is really just memory bandwidth, but quite impressive ;-) Ok, the iozone benchmark finally completed. The results do suggest that reading from mirrors substantially improves the throughput. This is interesting since the results differ (better than) from my 'virgin mount' test approach: Command line used: iozone -a -i 0 -i 1 -y 64 -q 512 -n 8G -g 256G KB reclen write rewritereadreread 8388608 64 572933 1008668 6945355 7509762 8388608 128 2753805 2388803 6482464 7041942 8388608 256 2508358 2331419 2969764 3045430 8388608 512 2407497 2131829 3021579 3086763 16777216 64 671365 879080 6323844 6608806 16777216 128 1279401 2286287 6409733 6739226 16777216 256 2382223 2211097 2957624 3021704 16777216 512 2237742 2179611 3048039 3085978 33554432 64 933712 699966 6418428 6604694 33554432 128 459896 431640 6443848 6546043 33554432 256 90 430989 2997615 3026246 33554432 512 427158 430891 3042620 3100287 67108864 64 426720 427167 6628750 6738623 67108864 128 419328 422581 153 6743711 67108864 256 419441 419129 3044352 3056615 67108864 512 431053 417203 3090652 3112296 134217728 64 417668 55434 759351 760994 134217728 128 409383 400433 759161 765120 134217728 256 408193 405868 763892 766184 134217728 512 408114 403473 761683 766615 268435456 64 418910 55239 768042 768498 268435456 128 408990 399732 763279 766882 268435456 256 413919 399386 760800 764468 268435456 512 410246 403019 766627 768739 Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Problem: Disconnected command timeout for Target X
Hi Bob, Thanks for the answers. How do I test your theory? In this case, I use common disks SATA 2, not Nearline SAS (NL SATA) or SAS. Do you think the disks SATA are the problem? Cheers, 2012/7/17 Bob Friesenhahn bfrie...@simple.dallas.tx.us On Tue, 17 Jul 2012, Roberto Scudeller wrote: Hi all, I'm using Opensolaris snv_134 with LSI Controllers and a motherboard supermicro, with 20 sata disks, zfs in raid-10 conf. I mounted this zfs_storage with NFS. I'm not opensolaris specialist. What're the commands to show hardware information? Like 'lshw' in linux but for opensolaris. cfgadm, prtconf, prtpicl, prtdiag zpool status fmadm faulty It sounds like you may have a broken cable or power supply failure to some disks. Bob The storage stopped working, but ping responds. SSH and NFS is out. When I open the console showing this messages: Jul 2 13:00:27 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:00:27 storageDisconnected command timeout for Target 4 Jul 2 13:01:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:01:28 storageDisconnected command timeout for Target 3 Jul 2 13:02:28 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:02:28 storageDisconnected command timeout for Target 2 Jul 2 13:03:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:03:29 storageDisconnected command timeout for Target 1 Jul 2 13:04:29 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:04:29 storageDisconnected command timeout for Target 0 Jul 2 13:05:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:05:40 storageDisconnected command timeout for Target 6 Jul 2 13:06:40 storage scsi: [ID 107833 kern.warning] WARNING: /pci@0 ,0/pci8086,340a@3/**pci1000,3140@0 (mpt2): Jul 2 13:06:40 storageDisconnected command timeout for Target 5 Any ideas? Could help me? -- Roberto Scudeller -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/** users/bfriesen/ http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ -- Roberto Scudeller ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Problem: Disconnected command timeout for Target X
On Tue, 17 Jul 2012, Roberto Scudeller wrote: Hi Bob, Thanks for the answers. How do I test your theory? I would use 'dd' to see if it is possible to transfer data from one of the problem devices. Gain physical access to the system and check the signal and power cables to these devices closely. Use 'iostat -xe' to see what error counts have accumulated. Also 'iostat -E'. In this case, I use common disks SATA 2, not Nearline SAS (NL SATA) or SAS. Do you think the disks SATA are the problem? There have been reports of congestion leading to timeouts and resets when SATA disks are on expanders. There have also been reports that one failing disk can cause problems when on expanders. Regardless, if this system has been previously operating fine for some time, these errors would indicate a change in the hardware shared by all these devices. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Has anyone switched from IR - IT firmware on the fly ? (existing zpool on LSI 9211-8i)
We have a running zpool with a 12 disk raidz3 vdev in it ... we gave ZFS the
full, raw disks ... all is well.
However, we built it on two LSI 9211-8i cards and we forgot to change from IR
firmware to IT firmware.
Is there any danger in shutting down the OS, flashing the cards to IT firmware,
and then booting back up ?
We did not create any raid configuration - as far as we know, the LSI cards are
just passing through the disks to ZFS ... but maybe not ?
I'd like to hear of someone else doing this successfully before we try it ...
We created the zpool with raw disks:
zpool create -m /mount/point MYPOOL raidz3 da{0,1,2,3,4,5,6,7,8,9,10,11}
and diskinfo tells us that each disk is:
da1 512 3000592982016 5860533168
The physical label (the sticker) on the disk also says 5860533168 sectors ...
so that seems to line up ...
Someone else in the world has made this change while inflight and can confirm
?
Thanks.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Has anyone switched from IR - IT firmware on the fly ? (existing zpool on LSI 9211-8i)
Hi Jason,
I have done this in the past. (3x LSI 1068E - IBM BR10i).
Your pool has no tie with the hardware used to host it (including your
HBA). You could change all your hardware, and still import your pool
correctly.
If you really want to be on the safe side; you can export your pool before
the firmware change and then import when your satisfied the firmware
change is complete.
Export: http://docs.oracle.com/cd/E19082-01/817-2271/gazqr/index.html
Import: http://docs.oracle.com/cd/E19082-01/817-2271/gazuf/index.html
Damon Pollard
On Wed, Jul 18, 2012 at 6:14 AM, Jason Usher jushe...@yahoo.com wrote:
We have a running zpool with a 12 disk raidz3 vdev in it ... we gave ZFS
the full, raw disks ... all is well.
However, we built it on two LSI 9211-8i cards and we forgot to change from
IR firmware to IT firmware.
Is there any danger in shutting down the OS, flashing the cards to IT
firmware, and then booting back up ?
We did not create any raid configuration - as far as we know, the LSI
cards are just passing through the disks to ZFS ... but maybe not ?
I'd like to hear of someone else doing this successfully before we try it
...
We created the zpool with raw disks:
zpool create -m /mount/point MYPOOL raidz3 da{0,1,2,3,4,5,6,7,8,9,10,11}
and diskinfo tells us that each disk is:
da1 512 3000592982016 5860533168
The physical label (the sticker) on the disk also says 5860533168 sectors
... so that seems to line up ...
Someone else in the world has made this change while inflight and can
confirm ?
Thanks.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Has anyone switched from IR - IT firmware on the fly ? (existing zpool on LSI 9211-8i)
Ok, and your LSI 1068E also had alternate IR and IT firmwares, and you went
from IR - IT ?
Is that correct ?
Thanks.
--- On Tue, 7/17/12, Damon Pollard damon.poll...@birchmangroup.com wrote:
From: Damon Pollard damon.poll...@birchmangroup.com
Subject: Re: [zfs-discuss] Has anyone switched from IR - IT firmware on the
fly ? (existing zpool on LSI 9211-8i)
To: Jason Usher jushe...@yahoo.com
Cc: zfs-discuss@opensolaris.org
Date: Tuesday, July 17, 2012, 5:05 PM
Hi Jason,
I have done this in the past. (3x LSI 1068E - IBM BR10i).
Your pool has no tie with the hardware used to host it (including your HBA).
You could change all your hardware, and still import your pool correctly.
If you really want to be on the safe side; you can export your pool before the
firmware change and then import when your satisfied the firmware change is
complete.
Export: http://docs.oracle.com/cd/E19082-01/817-2271/gazqr/index.html
Import: http://docs.oracle.com/cd/E19082-01/817-2271/gazuf/index.html
Damon Pollard
On Wed, Jul 18, 2012 at 6:14 AM, Jason Usher jushe...@yahoo.com wrote:
We have a running zpool with a 12 disk raidz3 vdev in it ... we gave ZFS the
full, raw disks ... all is well.
However, we built it on two LSI 9211-8i cards and we forgot to change from IR
firmware to IT firmware.
Is there any danger in shutting down the OS, flashing the cards to IT firmware,
and then booting back up ?
We did not create any raid configuration - as far as we know, the LSI cards are
just passing through the disks to ZFS ... but maybe not ?
I'd like to hear of someone else doing this successfully before we try it ...
We created the zpool with raw disks:
zpool create -m /mount/point MYPOOL raidz3 da{0,1,2,3,4,5,6,7,8,9,10,11}
and diskinfo tells us that each disk is:
da1 512 3000592982016 5860533168
The physical label (the sticker) on the disk also says 5860533168 sectors ...
so that seems to line up ...
Someone else in the world has made this change while inflight and can confirm
?
Thanks.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Has anyone switched from IR - IT firmware on the fly ? (existing zpool on LSI 9211-8i)
Correct.
LSI 1068E has IR and IT firmwares + I have gone from IR - IT and IT - IR
without hassle.
Damon Pollard
On Wed, Jul 18, 2012 at 8:13 AM, Jason Usher jushe...@yahoo.com wrote:
Ok, and your LSI 1068E also had alternate IR and IT firmwares, and you
went from IR - IT ?
Is that correct ?
Thanks.
--- On Tue, 7/17/12, Damon Pollard damon.poll...@birchmangroup.com
wrote:
From: Damon Pollard damon.poll...@birchmangroup.com
Subject: Re: [zfs-discuss] Has anyone switched from IR - IT firmware on
the fly ? (existing zpool on LSI 9211-8i)
To: Jason Usher jushe...@yahoo.com
Cc: zfs-discuss@opensolaris.org
Date: Tuesday, July 17, 2012, 5:05 PM
Hi Jason,
I have done this in the past. (3x LSI 1068E - IBM BR10i).
Your pool has no tie with the hardware used to host it (including your
HBA). You could change all your hardware, and still import your pool
correctly.
If you really want to be on the safe side; you can export your pool before
the firmware change and then import when your satisfied the firmware
change is complete.
Export: http://docs.oracle.com/cd/E19082-01/817-2271/gazqr/index.html
Import: http://docs.oracle.com/cd/E19082-01/817-2271/gazuf/index.html
Damon Pollard
On Wed, Jul 18, 2012 at 6:14 AM, Jason Usher jushe...@yahoo.com wrote:
We have a running zpool with a 12 disk raidz3 vdev in it ... we gave ZFS
the full, raw disks ... all is well.
However, we built it on two LSI 9211-8i cards and we forgot to change from
IR firmware to IT firmware.
Is there any danger in shutting down the OS, flashing the cards to IT
firmware, and then booting back up ?
We did not create any raid configuration - as far as we know, the LSI
cards are just passing through the disks to ZFS ... but maybe not ?
I'd like to hear of someone else doing this successfully before we try it
...
We created the zpool with raw disks:
zpool create -m /mount/point MYPOOL raidz3 da{0,1,2,3,4,5,6,7,8,9,10,11}
and diskinfo tells us that each disk is:
da1 512 3000592982016 5860533168
The physical label (the sticker) on the disk also says 5860533168 sectors
... so that seems to line up ...
Someone else in the world has made this change while inflight and can
confirm ?
Thanks.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
