Re: [zfs-discuss] Nexsan SATABeast and ZFS
I would like to go back to my question for a second: I checked with my Nexsan supplier and they confirmed that access to every single disk in SATABeast is not possible. The smallest entities I can create on the SATABeast are RAID 0 or 1 arrays. With RAID 1 I'll loose too much disk space and I believe that leaves me with RAID 0 as the only reasonable option. But with this unsecure RAID format I'll need higher redundancy in the ZFS configuration. I think I'll go with the following configuration: On the Nexsan SATABeast: * 14 disks configured in 7 RAID arrays with RAID level 0 (each disk is 1 TB which gives me a total of 14 TB raw disk space). * Each RAID 0 array configured as one volume. On the Sun Fire X4100 M2 with Solaris 10: * Add all 7 volumes to one zpool configured in on raidz2 (gives me approx. 8,8 TB available disk space) Any comments or suggestions? Best regards, Lars-Gunnar Persson On 11. mars. 2009, at 02.39, Bob Friesenhahn wrote: On Tue, 10 Mar 2009, A Darren Dunham wrote: What part isn't true? ZFS has a independent checksum for the data block. But if the data block is spread over multiple disks, then each of the disks have to be read to verify the checksum. I interpreted what you said to imply that RAID6 type algorithms were being used to validate the data, rather than to correct wrong data. I agree that it is necessary to read a full ZFS block in order to use the ZFS block checksum. I also agree that a raidz2 vdev has IOPS behavior which is similar to a single disk. From what I understand, a raidz2 with a very large number of disks won't use all of the disks to store one ZFS block. There is a maximum number of disks in a stripe which can be supported by the ZFS block size. -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ext4 bug zfs handling of the very same situation
Hi, there was recently a bug reported against EXT4 that gets triggered by KDE: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781 Now I'd like to verify that my understanding of ZFS behavior and implementations is correct, and ZFS is unaffected from this kind of issue. Maybe somebody would like to comment on this. The underlying problem with ext4 is that some kde executables do something like this: 1a) open and read data from file x, close file x 1b) open and truncate file x 1c) write data to file x 1d) close file x or 2a) open and read data from file x, close file x 2b) open and truncate file x.new 2c) write data to file x.new 2d) close file x.new 2e) rename file x.new to file x Concerning case 1) I think ZFS may lose data if power is lost right after 1b) and open(xxx,O_WRONLY|O_TRUNC|O_CREAT) is issued in a transaction group separately from the one containing 1c/1d. Concerning case 2) I cannot see ZFS losing any data, because of copy-on-write and transaction grouping. Theodore Ts'o (ext4 developer) commented that both cases are flawed and cannot be supported correctly, because of a lacking fsync() before close. Is this correct? His comment is over here: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54 Any thoughts or comments? TIA, Thomas ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ext4 bug zfs handling of the very same situation
The underlying problem with ext4 is that some kde executables do something like this: 1a) open and read data from file x, close file x 1b) open and truncate file x 1c) write data to file x 1d) close file x or 2a) open and read data from file x, close file x 2b) open and truncate file x.new 2c) write data to file x.new 2d) close file x.new 2e) rename file x.new to file x Concerning case 1) I think ZFS may lose data if power is lost right after 1b) and open(xxx,O_WRONLY|O_TRUNC|O_CREAT) is issued in a transaction group separately from the one containing 1c/1d. Yes, I would assume that is possible but the change for it happening is small. Other filesystems prefer to write the meta data prompt, but ZFS will easily wait until the file is completely written. And UFS has the extra problem that it can change the file size and reading will show garbage in the file. (It changes the inode, possibly because it's in the log, but it hasn't written the data). We've seen that problem with UFS and the /etc/*_* driver files. Precisely because we didn't flush/fsync. (And in some cases we used fsync(fileno(file)), but the new content was still in the stdio buffer) Only versioned filesystems can make the first sequence work. Concerning case 2) I cannot see ZFS losing any data, because of copy-on-write and transaction grouping. Theodore Ts'o (ext4 developer) commented that both cases are flawed and cannot be supported correctly, because of a lacking fsync() before close. Is this correct? His comment is over here: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54 Perhaps we should Theodore Ts'o that ZFS gets this right :-) I'm assuming that all transactions in group N all happened before group N+1 at least when it comes to the partial order in which the transactions happen. Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] [osol-discuss] ext4 bug zfs handling of the very same situation
Thomas Maier-Komor tho...@maier-komor.de wrote: there was recently a bug reported against EXT4 that gets triggered by KDE: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781 Now I'd like to verify that my understanding of ZFS behavior and implementations is correct, and ZFS is unaffected from this kind of issue. Maybe somebody would like to comment on this. The underlying problem with ext4 is that some kde executables do something like this: 1a) open and read data from file x, close file x 1b) open and truncate file x 1c) write data to file x 1d) close file x or 2a) open and read data from file x, close file x 2b) open and truncate file x.new 2c) write data to file x.new 2d) close file x.new 2e) rename file x.new to file x Concerning case 1) I think ZFS may lose data if power is lost right after 1b) and open(xxx,O_WRONLY|O_TRUNC|O_CREAT) is issued in a transaction group separately from the one containing 1c/1d. It depends on what you call loses data. If you truncate an existing file, the content of the file is expected to be lost if the file system behaves correctly. Concerning case 2) I cannot see ZFS losing any data, because of copy-on-write and transaction grouping. Theodore Ts'o (ext4 developer) commented that both cases are flawed and cannot be supported correctly, because of a lacking fsync() before close. Is this correct? His comment is over here: https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54 An application cannot be sure on whether the data has been written to disk if there was no fsync() call. Star for this reason by default calls fsync() before closing a file in order to be able to include information about problems in the star exit code. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] reboot when copying large amounts of data
I have a H8DM8-2 motherboard with a pair of AOC-SAT2-MV8 SATA controller cards in a 16-disk Supermicro chassis. I'm running OpenSolaris 2008.11, and the machine performs very well unless I start to copy a large amount of data to the ZFS (software raid) array that's on the Supermicro SATA controllers. If I do this, the machine inevitably reboots. What can I do to troubleshoot? The BIOS of the motherboard and the SATA card firmware are fully updated. I'm running the latest stable OpenSolaris, and see nothing amiss in the system logs when this happens. I've enabled savecore and debug-level syslog, but am getting no indicators from Solaris as to what's wrong. Interestingly, I can push the same amount of data to the mirror boot disks, which are on the board's built-in nVidia SATA controller without issue. The vdev I'm pushing to is a 5-disk raidz2 with 2 hot spares. Help! :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Export ZFS via ISCSI to Linux - Is it stable for production use now?
Hello, I want to setup an opensolaris for centralized storage server, using ZFS as the underlying FS, on a RAID 10 SATA disks. I will export the storage blocks using ISCSI to RHEL 5 (less than 10 clients, and I will format the partition as EXT3) I want to ask... 1. Is this setup suitable for mission critical use now? 2. Can I use LVM with this setup? Currently we are using NFS as the centralized storage solutions. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Export ZFS via ISCSI to Linux - Is it stable for production use now?
howard chen wrote: Hello, I want to setup an opensolaris for centralized storage server, using ZFS as the underlying FS, on a RAID 10 SATA disks. I will export the storage blocks using ISCSI to RHEL 5 (less than 10 clients, and I will format the partition as EXT3) I want to ask... 1. Is this setup suitable for mission critical use now? Yes, why wouldn't it be ? It is even officially supported as part of the Sun Unified Storage 7000 series appliances built from OpenSolaris and ZFS technology. 2. Can I use LVM with this setup? Yes you can. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Import Error - Please Help
Okay so for some reason my Opensolaris 106 installation quit booting properly (completely seperate problem). At this point I was still able to boot into safe mode, but not into a normal desktop session. First a disk failed, I replaced this disk with another. The array completely resilvered. I was able to scrub the array, etc. Second, I added 2 more disks, issuing the replace command to use these 2 new disks instead of 2 of the old disks (so I replaced 2 good IDE disks with 2 good SATA disks, all 4 disks were attached at the same time, thus the reason for multiple replace's at one time). The resilver completed successfully. I was able to scrub the array. I exported the array, rebooted, imported the array, and scrubbed it again - no problems. Now I removed the 2 IDE drives, rebooted again, was able to import the array, scrub it, etc no problems. (Until this point everything has been done in safe mode, and the replacing stuff never went away although the scrub's completed fully, and all the origional drives (that had been replaced) have been removed). SO I then proceed to install opensolaris 109 (in order to get a fully booting/working machine again). Now when I zpool import I get the following: # zpool import pool: tank id: 15417328101188042571 state: UNAVAIL action: The pool cannot be imported due to damaged devices or data. config: tankUNAVAIL insufficient replicas raidz1ONLINE c1t1d0 ONLINE c3t0d0 ONLINE c2t0d0 ONLINE c4t1d0 ONLINE c1t0d0p2ONLINE raidz1UNAVAIL corrupted data c4t7d0 ONLINE c4t6d0 ONLINE replacing ONLINE 13069118156582972341 ONLINE c2t1d0ONLINE replacing DEGRADED 11144084490198638692 UNAVAIL cannot open c5t0d0ONLINE replacing ONLINE 3239645952837551668 ONLINE c5t1d0ONLINE c4t0d0 ONLINE c4t5d0 ONLINE c4t2d0 ONLINE c4t4d0 ONLINE c4t3d0 ONLINE # Everything shows as online except for one vdev (that is currently replacing), so I don't understand why it says insufficient replicas instead of DEGRADED Any help appreciated. Thanks in advance, -mike -- Michael Dupree jr. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Nexsan SATABeast and ZFS
Lars-Gunnar Persson wrote: I would like to go back to my question for a second: I checked with my Nexsan supplier and they confirmed that access to every single disk in SATABeast is not possible. The smallest entities I can create on the SATABeast are RAID 0 or 1 arrays. With RAID 1 I'll loose too much disk space and I believe that leaves me with RAID 0 as the only reasonable option. But with this unsecure RAID format I'll need higher redundancy in the ZFS configuration. I think I'll go with the following configuration: On the Nexsan SATABeast: * 14 disks configured in 7 RAID arrays with RAID level 0 (each disk is 1 TB which gives me a total of 14 TB raw disk space). * Each RAID 0 array configured as one volume. So what the front end will see is 7 disks, 2TB each disk. On the Sun Fire X4100 M2 with Solaris 10: * Add all 7 volumes to one zpool configured in on raidz2 (gives me approx. 8,8 TB available disk space) You'll get 5 LUNs worth of space in this config, or 10TB of usable space. Any comments or suggestions? Given the hardware constraints (no single-disk volumes allowed) this is a good configuration for most purposes. The advantages/disadvantages are: . 10TB of usable disk space, out of 14TB purchased. . At least three hard disk failures are required to lose the ZFS pool. . Random non-cached read performance will be about 300 IO/sec. . Sequential reads and writes of the whole ZFS blocksize will be fast (up to 2000 IO/sec). . One hard drive failure will cause the used blocks of the 2TB LUN (raid0 pair) to be resilvered, even though the other half of the pair is not damaged. The other half of the pair is more likely to fail during the ZFS resilvering operation because of increased load. You'll want to pay special attention to the cache settings on the Nexsan. You earlier showed that the write cache is enabled, but IIRC the array doesn't have a nonvolatile (battery-backed) cache. If that's the case, MAKE SURE it's hooked up to a UPS that can support it for the 30 second cache flush timeout on the array. And make sure you don't power it down hard. I think you want to uncheck the ignore FUA setting, so that FUA requests are respected. My guess is that this will cause the array to properly handle the cache_flush requests that ZFS uses to ensure data consistancy. --Joe ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Export ZFS via ISCSI to Linux - Is it stable for production use now?
Hello, On Wed, Mar 11, 2009 at 10:20 PM, Darren J Moffat darr...@opensolaris.org wrote: 1. Is this setup suitable for mission critical use now? Yes, why wouldn't it be ? Because I just wonder why some other people are using zfs/fuse on Linux, e.g. http://www.drwetter.org/blog/zfs_under_linux.en.html http://www.wizy.org/wiki/ZFS_on_FUSE Also seems hard to find a complete tutorial on accessing ZFS from Linux using ISCSI. This should be attractive, isn't? So I don't know if it is experimental. Howard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Export ZFS via ISCSI to Linux - Is it stable for production use now?
howard chen wrote: Hello, On Wed, Mar 11, 2009 at 10:20 PM, Darren J Moffat darr...@opensolaris.org wrote: 1. Is this setup suitable for mission critical use now? Yes, why wouldn't it be ? Because I just wonder why some other people are using zfs/fuse on Linux, e.g. http://www.drwetter.org/blog/zfs_under_linux.en.html http://www.wizy.org/wiki/ZFS_on_FUSE ZFS/fuse on Linux gives you something very different to exporting ZFS ZVOLs over iSCSI. They solve completely different problems. As for why people play with zfs/fuse on Linux rather than running ZFS on OpenSolaris - well because they can, or because their religion means they want to use Linux or just to test zfs/fuse. Same as why people play with ZFS on FreeBSD or MacOS X. Because they can and for what ever reasons they have they want to use ZFS and an OS other than OpenSolaris. None of that implies that sharing ZVOLs over iSCSI isn't stable. Also seems hard to find a complete tutorial on accessing ZFS from Linux using ISCSI. This should be attractive, isn't? On the OpenSolaris side: Create your pool: # zpool create tank your list of disks and raid config goes here Turn on the iSCSI service # svcadm enable iscsitgtd [ or if you are on a release with COMSTART make that iscsi/target ] Create the ZVOL for the iSCSI lun and share it out: # zfs create -V 10g -o shareiscsi=on tank/lun1 If you want more control over the sharing options see iscsitadm(1M) That is all that has to be done on the OpenSolaris side to make a 10g lun available over iSCSI. The rest of it is all how Linux sets up its iSCSI client side which I don't know but I know on Solaris it is very easy using iscsiadm(1M). -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Export ZFS via ISCSI to Linux - Is it stable for production use now?
I blogged this a while ago: http://blog.clockworm.com/2007/10/connecting-linux-centos-5-to-solaris.html On Wed, Mar 11, 2009 at 1:02 PM, howard chen howac...@gmail.com wrote: Hello, On Wed, Mar 11, 2009 at 10:20 PM, Darren J Moffat darr...@opensolaris.org wrote: 1. Is this setup suitable for mission critical use now? Yes, why wouldn't it be ? Because I just wonder why some other people are using zfs/fuse on Linux, e.g. http://www.drwetter.org/blog/zfs_under_linux.en.html http://www.wizy.org/wiki/ZFS_on_FUSE Also seems hard to find a complete tutorial on accessing ZFS from Linux using ISCSI. This should be attractive, isn't? So I don't know if it is experimental. Howard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
Something is not right in the IO space. The messages talk about vendor ID = 11AB 0x11AB Marvell Semiconductor TMC Research Vendor Id: 0x1030 Short Name: TMC Does fmdump -eV give any clue when the box comes back up? ..Remco Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
fmdump is not helping much: r...@host:~# fmdump -eV TIME CLASS fmdump: /var/fm/fmd/errlog is empty comparing that screenshot to the output of cfgadm is interesting - looks like the controller(s): r...@host:~# cfgadm -v Ap_Id Receptacle Occupant Condition Information When Type Busy Phys_Id sata4/0::dsk/c4t0d0connectedconfigured ok Mod: ST3250310NS FRev: SN06 SN: 9SF06CZZ unavailable disk n/devices/p...@0,0/pci15d9,1...@5:0 sata4/1::dsk/c4t1d0connectedconfigured ok Mod: ST3250310NS FRev: SN06 SN: 9SF06BC8 unavailable disk n/devices/p...@0,0/pci15d9,1...@5:1 sata5/0emptyunconfigured ok unavailable sata-portn/devices/p...@0,0/pci15d9,1...@5,1:0 sata5/1::dsk/c7t1d0connectedconfigured ok Mod: WDC WD10EACS-00D6B0 FRev: 01.01A01 SN: WD-WCAU40244615 unavailable disk n/devices/p...@0,0/pci15d9,1...@5,1:1 sata6/0emptyunconfigured ok unavailable sata-portn/devices/p...@0,0/pci15d9,1...@5,2:0 sata6/1emptyunconfigured ok unavailable sata-portn/devices/p...@0,0/pci15d9,1...@5,2:1 sata7/0emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:0 sata7/1emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:1 sata7/2::dsk/c5t2d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0376631 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:2 sata7/3::dsk/c5t3d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0350798 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:3 sata7/4::dsk/c5t4d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0403574 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:4 sata7/5::dsk/c5t5d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0312592 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:5 sata7/6::dsk/c5t6d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0399779 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:6 sata7/7::dsk/c5t7d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT0441660 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@4:7 sata8/0::dsk/c6t0d0connectedconfigured ok Mod: WDC WD7500AYYS-01RCA0 FRev: 30.04G30 SN: WD-WCAPT1000344 unavailable disk n /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:0 sata8/1emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:1 sata8/2emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:2 sata8/3emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:3 sata8/4emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:4 sata8/5emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:5 sata8/6emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:6 sata8/7emptyunconfigured ok unavailable sata-portn /devices/p...@0,0/pci10de,3...@a/pci1033,1...@0/pci11ab,1...@6:7 On Wed, Mar 11, 2009 at 2:40 PM, Blake blake.ir...@gmail.com wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks
Re: [zfs-discuss] reboot when copying large amounts of data
I think that TMC Research is the company that designed the Supermicro-branded controller card that has the Marvell SATA controller chip on it. Googling around I see connections between Supermicro and TMC. This is the card: http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm On Wed, Mar 11, 2009 at 3:08 PM, Remco Lengers re...@lengers.com wrote: Something is not right in the IO space. The messages talk about vendor ID = 11AB 0x11AB Marvell Semiconductor TMC Research Vendor Id: 0x1030 Short Name: TMC Does fmdump -eV give any clue when the box comes back up? ..Remco Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
Could the problem be related to this bug: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6793353 I'm testing setting the maximum payload size as a workaround, as noted in the bug notes. On Wed, Mar 11, 2009 at 3:14 PM, Blake blake.ir...@gmail.com wrote: I think that TMC Research is the company that designed the Supermicro-branded controller card that has the Marvell SATA controller chip on it. Googling around I see connections between Supermicro and TMC. This is the card: http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm On Wed, Mar 11, 2009 at 3:08 PM, Remco Lengers re...@lengers.com wrote: Something is not right in the IO space. The messages talk about vendor ID = 11AB 0x11AB Marvell Semiconductor TMC Research Vendor Id: 0x1030 Short Name: TMC Does fmdump -eV give any clue when the box comes back up? ..Remco Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
looks worth a go otherwise: if the boot disk is also off that controller it may be too hosed to write anything to the boot disk hence FMA doesn't see any issue when it comes up. Possible further actions: - Upgrade FW of controller to highest or known working level - Upgrade driver or OS level. - Try another controller (may be its broken and barfs under stress ?) - Analyze the crash dump (if any is saved) - It may be its a know Solaris or driver bug and somebody has heard of it before. hth, ..Remco Blake wrote: Could the problem be related to this bug: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6793353 I'm testing setting the maximum payload size as a workaround, as noted in the bug notes. On Wed, Mar 11, 2009 at 3:14 PM, Blake blake.ir...@gmail.com wrote: I think that TMC Research is the company that designed the Supermicro-branded controller card that has the Marvell SATA controller chip on it. Googling around I see connections between Supermicro and TMC. This is the card: http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm On Wed, Mar 11, 2009 at 3:08 PM, Remco Lengers re...@lengers.com wrote: Something is not right in the IO space. The messages talk about vendor ID = 11AB 0x11AB Marvell Semiconductor TMC Research Vendor Id: 0x1030 Short Name: TMC Does fmdump -eV give any clue when the box comes back up? ..Remco Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
Any chance this could be the motherboard? I suspect the controller. The boot disks are on the built-in nVidia controller. On Wed, Mar 11, 2009 at 3:41 PM, Remco Lengers re...@lengers.com wrote: - Upgrade FW of controller to highest or known working level I think I have the latest controller firmware. - Upgrade driver or OS level. I'm going to try to go from 101b to 108 or whatever the current dev release is. - Try another controller (may be its broken and barfs under stress ?) In the works. - Analyze the crash dump (if any is saved) Crash dump is not saving properly. - It may be its a know Solaris or driver bug and somebody has heard of it before. Any takers on this? :) hth, Thanks! ..Remco Blake wrote: Could the problem be related to this bug: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6793353 I'm testing setting the maximum payload size as a workaround, as noted in the bug notes. On Wed, Mar 11, 2009 at 3:14 PM, Blake blake.ir...@gmail.com wrote: I think that TMC Research is the company that designed the Supermicro-branded controller card that has the Marvell SATA controller chip on it. Googling around I see connections between Supermicro and TMC. This is the card: http://www.supermicro.com/products/accessories/addon/AOC-SAT2-MV8.cfm On Wed, Mar 11, 2009 at 3:08 PM, Remco Lengers re...@lengers.com wrote: Something is not right in the IO space. The messages talk about vendor ID = 11AB 0x11AB Marvell Semiconductor TMC Research Vendor Id: 0x1030 Short Name: TMC Does fmdump -eV give any clue when the box comes back up? ..Remco Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. BTW, I'm copying to a raidz2 of only 5 disks, not 16 (the chassis supports 16, but isn't fully populated). So far as I know, there is no spinup happening - these are not RAID controllers, just dumb SATA JBOD controllers, so I don't think they control drive spin in any particular way. Correct me if I'm wrong, of course. On Wed, Mar 11, 2009 at 11:23 AM, Marc Bevand m.bev...@gmail.com wrote: The copy operation will make all the disks start seeking at the same time and will make your CPU activity jump to a significant percentage to compute the ZFS checksum and RAIDZ parity. I think you could be overloading your PSU because of the sudden increase in power consumption... However if you are *not* using SATA staggered spin-up, then the above theory is unlikely because spinning up consumes much more power than when seeking. So, in a sense, a successful boot proves your PSU is powerful enough. Trying reproducing the problem by copying data on a smaller number of disks. You tried 2 and 16. Try 8. -marc ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS on a SAN
Hi All, I'm new on ZFS, so I hope this isn't too basic a question. I have a host where I setup ZFS. The Oracle DBAs did their thing and I know have a number of ZFS datasets with their respective clones and snapshots on serverA. I want to export some of the clones to serverB. Do I need to zone serverB to see the same LUNs as serverA? Or does it have to have preexisting, empty LUNs to import the clones? Please help. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. -Erik On Wed, 2009-03-11 at 13:18 -0700, Grant Lowe wrote: Hi All, I'm new on ZFS, so I hope this isn't too basic a question. I have a host where I setup ZFS. The Oracle DBAs did their thing and I know have a number of ZFS datasets with their respective clones and snapshots on serverA. I want to export some of the clones to serverB. Do I need to zone serverB to see the same LUNs as serverA? Or does it have to have preexisting, empty LUNs to import the clones? Please help. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
Hi Eric, Thanks for the quick response. Then on hostB, the new LUN will need the same amount of disk space for the pool, as on hostA, if I'm understanding you correctly. Correct? Thanks! - Original Message From: Erik Trimble erik.trim...@sun.com To: Grant Lowe gl...@sbcglobal.net Cc: zfs-discuss@opensolaris.org Sent: Wednesday, March 11, 2009 1:42:06 PM Subject: Re: [zfs-discuss] ZFS on a SAN I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. -Erik On Wed, 2009-03-11 at 13:18 -0700, Grant Lowe wrote: Hi All, I'm new on ZFS, so I hope this isn't too basic a question. I have a host where I setup ZFS. The Oracle DBAs did their thing and I know have a number of ZFS datasets with their respective clones and snapshots on serverA. I want to export some of the clones to serverB. Do I need to zone serverB to see the same LUNs as serverA? Or does it have to have preexisting, empty LUNs to import the clones? Please help. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
Erik Trimble wrote: I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. Need to be clear on the terminology here. Yes, it is possible for two systems to have access to a single LUN and have ZFS file systems. The ZFS limitation is at the vdev level (partition or slice), which is below the LUN level. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' LUN masking or reservations occur at the LUN level, which is why it is often better (safer) to design with the expectation that a LUN will be only available to one host at a time. Or, to say it differently, if you think of one vdev/LUN, then you can innoculate yourself from later grief :-) ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. +1 -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. savecore -L is not what you want. By default, for OpenSolaris, savecore on boot is disabled. But the core will have been dumped into the dump slice, which is not used for swap. So you should be able to run savecore at a later time to collect the core from the last dump. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
On Wed, 2009-03-11 at 13:50 -0700, Grant Lowe wrote: Hi Eric, Thanks for the quick response. Then on hostB, the new LUN will need the same amount of disk space for the pool, as on hostA, if I'm understanding you correctly. Correct? Thanks! I'm assuming you're referring to my second case, where you don't bother to 'import'/'export' and single LUN, and just use the 'send'/'receive' option. HostB's zpool only needs to have enough space to handle the snapshot size. You can of course delete a snapshot on either host after it's taken, so you may not need to keep all snapshots around on both hosts. So, let's say this happens: pool on hostA is 2TB, with 1TB of active data, and a typical snapshot of 100GB. Which means you can keep around 10 snapshots on hostA. On hostB, you need a zpool of 1TB for the 'live' data, plus 100GB extra to store an incoming snapshot, but once you've received the snapshot, you can delete any older snapshot, so you really only need 1.1TB on hostB. Unless, of course, you want to keep the older snapshots. For old timer's sake, compare 'zfs send' and 'zfs'receive' to 'ufsdump' and 'ufsrestore' (respectively) for their utility. -Erik - Original Message From: Erik Trimble erik.trim...@sun.com To: Grant Lowe gl...@sbcglobal.net Cc: zfs-discuss@opensolaris.org Sent: Wednesday, March 11, 2009 1:42:06 PM Subject: Re: [zfs-discuss] ZFS on a SAN I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. -Erik On Wed, 2009-03-11 at 13:18 -0700, Grant Lowe wrote: Hi All, I'm new on ZFS, so I hope this isn't too basic a question. I have a host where I setup ZFS. The Oracle DBAs did their thing and I know have a number of ZFS datasets with their respective clones and snapshots on serverA. I want to export some of the clones to serverB. Do I need to zone serverB to see the same LUNs as serverA? Or does it have to have preexisting, empty LUNs to import the clones? Please help. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
I guess I didn't make it clear that I had already tried using savecore to retrieve the core from the dump device. I added a larger zvol for dump, to make sure that I wasn't running out of space on the dump device: r...@host:~# dumpadm Dump content: kernel pages Dump device: /dev/zvol/dsk/rpool/bigdump (dedicated) Savecore directory: /var/crash/host Savecore enabled: yes I was using the -L option only to try to get some idea of why the system load was climbing to 1 during a simple file copy. On Wed, Mar 11, 2009 at 4:58 PM, Richard Elling richard.ell...@gmail.com wrote: Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. savecore -L is not what you want. By default, for OpenSolaris, savecore on boot is disabled. But the core will have been dumped into the dump slice, which is not used for swap. So you should be able to run savecore at a later time to collect the core from the last dump. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
http://www.intel.com/products/server/storage-systems/ssr212mc2/ssr212mc2-overview.htm http://www.intel.com/support/motherboards/server/ssr212mc2/index.htm It's hard to use the HAL sometimes. I am trying to locate chipset info but having a hard time... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Why did my zvol shrink ?
I'm doing a little testing and I hit a strange point. Here is a zvol (clone) pool1/volclone type volume - pool1/volclone origin pool1/v...@diff1 - pool1/volclone reservation none default pool1/volclone volsize 191G - pool1/volclone volblocksize 8K - The zvol has UFS on it. It has always been 191G and we've never attempted to resize it. However, if I just try to grow it, it gives me an error: -bash-3.00# growfs /dev/zvol/rdsk/pool1/volclone 400555998 sectors current size of 400556032 sectors Is the zvol is somehow smaller than it was originally? How/why? It fsck ok, so UFS doesn't seem to notice. This is solaris 10 u6 currently, the machine (and zpool) have gone through a few update releases since creation. Thanks for any input, -Brian -- --- Brian H. Nelson Youngstown State University System Administrator Media and Academic Computing bnelson[at]cis.ysu.edu --- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
Looks like I may have slightly answered my question. I found that it uses the server board S5000PSL, which is in the HCL: http://www.sun.com/bigadmin/hcl/data/systems/details/2944.html It's not listed in the OpenSolaris one, only the Solaris one. I would think something this old (dated 2007) would be forward compatible... On Wed, Mar 11, 2009 at 5:14 PM, mike mike...@gmail.com wrote: http://www.intel.com/products/server/storage-systems/ssr212mc2/ssr212mc2-overview.htm http://www.intel.com/support/motherboards/server/ssr212mc2/index.htm It's hard to use the HAL sometimes. I am trying to locate chipset info but having a hard time... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
Hi Eric, Thanks. That scenario makes sense. I have a better of how to set things up now. It's a three-step process, which I didn't realize. grant - Original Message From: Erik Trimble erik.trim...@sun.com To: Grant Lowe gl...@sbcglobal.net Cc: zfs-discuss@opensolaris.org Sent: Wednesday, March 11, 2009 1:42:06 PM Subject: Re: [zfs-discuss] ZFS on a SAN I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. -Erik On Wed, 2009-03-11 at 13:18 -0700, Grant Lowe wrote: Hi All, I'm new on ZFS, so I hope this isn't too basic a question. I have a host where I setup ZFS. The Oracle DBAs did their thing and I know have a number of ZFS datasets with their respective clones and snapshots on serverA. I want to export some of the clones to serverB. Do I need to zone serverB to see the same LUNs as serverA? Or does it have to have preexisting, empty LUNs to import the clones? Please help. Thanks. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on a SAN
Hey Richard, That explanation help clarify things. I have another question, but maybe it'll be a new topic. Basically I would like to export some stuff of vxfs file system on a different host, and import to zfs. Is there a way to do that? grant - Original Message From: Richard Elling richard.ell...@gmail.com To: Erik Trimble erik.trim...@sun.com Cc: Grant Lowe gl...@sbcglobal.net; zfs-discuss@opensolaris.org Sent: Wednesday, March 11, 2009 1:52:31 PM Subject: Re: [zfs-discuss] ZFS on a SAN Erik Trimble wrote: I'm not 100% sure what your question here is, but let me give you a (hopefully) complete answer: (1) ZFS is NOT a clustered file system, in the sense that it is NOT possible for two hosts to have the same LUN mounted at the same time, even if both are hooked to a SAN and can normally see that LUN. Need to be clear on the terminology here. Yes, it is possible for two systems to have access to a single LUN and have ZFS file systems. The ZFS limitation is at the vdev level (partition or slice), which is below the LUN level. (2) ZFS can do failover, however. If you have a LUN from a SAN on hostA, create a ZFS pool in it, and use as normal. Should you with to failover the LUN to hostB, you need to do a 'zpool export zpool' on hostA, then 'zpool import zpool' on hostB. If hostA has been lost completely (hung/died/etc) and you are unable to do an 'export' on it, you can force the import on hostB via 'zpool import -f zpool' LUN masking or reservations occur at the LUN level, which is why it is often better (safer) to design with the expectation that a LUN will be only available to one host at a time. Or, to say it differently, if you think of one vdev/LUN, then you can innoculate yourself from later grief :-) ZFS requires that you import/export entire POOLS, not just filesystems. So, given what you seem to want, I'd recommend this: On the SAN, create (2) LUNs - one for your primary data, and one for your snapshots/backups. On hostA, create a zpool on the primary data LUN (call it zpool A), and another zpool on the backup LUN (zpool B). Take snapshots on A, then use 'zfs send' and 'zfs receive' to copy the clone/snapshot over to zpool B. then 'zpool export B' On hostB, import the snapshot pool: 'zfs import B' It might just be as easy to have two independent zpools on each host, and just do a 'zfs send' on hostA, and 'zfs receive' on hostB to copy the snapshot/clone over the wire. +1 -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
On Wed, 11 Mar 2009 17:55:23 -0700 mike mike...@gmail.com wrote: Looks like I may have slightly answered my question. I found that it uses the server board S5000PSL, which is in the HCL: http://www.sun.com/bigadmin/hcl/data/systems/details/2944.html It's not listed in the OpenSolaris one, only the Solaris one. I would think something this old (dated 2007) would be forward compatible... Just because it hasn't been specifically mentioned in the OpenSolaris or SX hcl, doesn't mean that it isn't supported. Neither does it necessarily mean that support was dropped. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
On Mar 11, 2009, at 20:14, mike wrote: http://www.intel.com/products/server/storage-systems/ssr212mc2/ssr212mc2-overview.htm http://www.intel.com/support/motherboards/server/ssr212mc2/index.htm It's hard to use the HAL sometimes. I am trying to locate chipset info but having a hard time... If you know someone who already has the hardware, you can ask them to run the Sun Device Detection Tool: http://www.sun.com/bigadmin/hcl/hcts/device_detect.jsp It runs under other operating system (Windows, Linux, BSD) AFAIK, so a re-install or reboot isn't necessary to see what it comes up with. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
doesnt it require java and x11? On Wed, Mar 11, 2009 at 6:53 PM, David Magda dma...@ee.ryerson.ca wrote: On Mar 11, 2009, at 20:14, mike wrote: http://www.intel.com/products/server/storage-systems/ssr212mc2/ssr212mc2-overview.htm http://www.intel.com/support/motherboards/server/ssr212mc2/index.htm It's hard to use the HAL sometimes. I am trying to locate chipset info but having a hard time... If you know someone who already has the hardware, you can ask them to run the Sun Device Detection Tool: http://www.sun.com/bigadmin/hcl/hcts/device_detect.jsp It runs under other operating system (Windows, Linux, BSD) AFAIK, so a re-install or reboot isn't necessary to see what it comes up with. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Trying to determine if this box will be compatible with Opensolaris or Solaris
On Mar 11, 2009, at 21:59, mike wrote: On Wed, Mar 11, 2009 at 6:53 PM, David Magda dma...@ee.ryerson.ca wrote: If you know someone who already has the hardware, you can ask them to run the Sun Device Detection Tool: http://www.sun.com/bigadmin/hcl/hcts/device_detect.jsp It runs under other operating system (Windows, Linux, BSD) AFAIK, so a re-install or reboot isn't necessary to see what it comes up with. doesnt it require java and x11? Yes, it requires Java 1.5+; a GUI is needed, but I don't think X11 is specifically required (X is the GUI on Unix-y systems of course). Java doesn't specifically need X, it simply uses whatever the OS has. Looking at the page a bit more, you can run commands on the system and save the output to a file that can be processed by the tool on another system: Apart from testing the current system on which Sun Device Detection Tool is invoked, you can also test the device data files that are generated from the external systems. To test the external device data files, print the PCI configuration of the external systems to a text file by using the following commands: • prtconf -pv on Solaris OS. • lspci -vv -n on Linux OS. • reg query hklm\system\currentcontrolset\enum\pci /s on Windows OS. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] User quota design discussion..
In the style of a discussion over a beverage, and talking about user-quotas on ZFS, I recently pondered a design for implementing user quotas on ZFS after having far too little sleep. It is probably nothing new, but I would be curious what you experts think of the feasibility of implementing such a system and/or whether or not it would even realistically work. I'm not suggesting that someone should do the work, or even that I will, but rather in the interest of chatting about it. Feel free to ridicule me as required! :) Thoughts: Here at work we would like to have user quotas based on uid (and presumably gid) to be able to fully replace the NetApps we run. Current ZFS are not good enough for our situation. We simply can not mount 500,000 file-systems on all the NFS clients. Nor do all servers we run support mirror-mounts. Nor do auto-mount see newly created directories without a full remount. Current UFS-style-user-quotas are very exact. To the byte even. We do not need this precision. If a user has 50MB of quota, and they are able to reach 51MB usage, then that is acceptable to us. Especially since they have to go under 50MB to be able to write new data, anyway. Instead of having complicated code in the kernel layer, slowing down the file-system with locking and semaphores (and perhaps avoiding learning indepth ZFS code?), I was wondering if a more simplistic setup could be designed, that would still be acceptable. I will use the word 'acceptable' a lot. Sorry. My thoughts are that the ZFS file-system will simply write a 'transaction log' on a pipe. By transaction log I mean uid, gid and 'byte count changed'. And by pipe I don't necessarily mean pipe(2), but it could be a fifo, pipe or socket. But currently I'm thinking '/dev/quota' style. User-land will then have a daemon, whether or not it is one daemon per file-system or really just one daemon does not matter. This process will open '/dev/quota' and empty the transaction log entries constantly. Take the uid,gid entries and update the byte-count in its database. How we store this database is up to us, but since it is in user-land it should have more flexibility, and is not as critical to be fast as it would have to be in kernel. The daemon process can also grow in number of threads as demand increases. Once a user's quota reaches the limit (note here that /the/ call to write() that goes over the limit will succeed, and probably a couple more after. This is acceptable) the process will blacklist the uid in kernel. Future calls to creat/open(CREAT)/write/(insert list of calls) will be denied. Naturally calls to unlink/read etc should still succeed. If the uid goes under the limit, the uid black-listing will be removed. If the user-land process crashes or dies, for whatever reason, the buffer of the pipe will grow in the kernel. If the daemon is restarted sufficiently quickly, all is well, it merely needs to catch up. If the pipe does ever get full and items have to be discarded, a full-scan will be required of the file-system. Since even with UFS quotas we need to occasionally run 'quotacheck', it would seem this too, is acceptable (if undesirable). If you have no daemon process running at all, you have no quotas at all. But the same can be said about quite a few daemons. The administrators need to adjust their usage. I can see a complication with doing a rescan. How could this be done efficiently? I don't know if there is a neat way to make this happen internally to ZFS, but from a user-land only point of view, perhaps a snapshot could be created (synchronised with the /dev/quota pipe reading?) and start a scan on the snapshot, while still processing kernel log. Once the scan is complete, merge the two sets. Advantages are that only small hooks are required in ZFS. The byte updates, and the blacklist with checks for being blacklisted. Disadvantages are that it is loss of precision, and possibly slower rescans? Sanity? But I do not really know the internals of ZFS, so I might be completely wrong, and everyone is laughing already. Discuss? Lund -- Jorgen Lundman | lund...@lundman.net Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo| +81 (0)90-5578-8500 (cell) Japan| +81 (0)3 -3375-1767 (home) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
If you're having issues with a disk contoller or disk IO driver its highly likely that a savecore to disk after the panic will fail. I'm not sure how to work around this, maybe a dedicated dump device not on a controller that uses a different driver then the one that you're having issues with? -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Blake Sent: Wednesday, March 11, 2009 4:45 PM To: Richard Elling Cc: Marc Bevand; zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] reboot when copying large amounts of data I guess I didn't make it clear that I had already tried using savecore to retrieve the core from the dump device. I added a larger zvol for dump, to make sure that I wasn't running out of space on the dump device: r...@host:~# dumpadm Dump content: kernel pages Dump device: /dev/zvol/dsk/rpool/bigdump (dedicated) Savecore directory: /var/crash/host Savecore enabled: yes I was using the -L option only to try to get some idea of why the system load was climbing to 1 during a simple file copy. On Wed, Mar 11, 2009 at 4:58 PM, Richard Elling richard.ell...@gmail.com wrote: Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. savecore -L is not what you want. By default, for OpenSolaris, savecore on boot is disabled. But the core will have been dumped into the dump slice, which is not used for swap. So you should be able to run savecore at a later time to collect the core from the last dump. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Why did my zvol shrink ?
Brian H. Nelson wrote: I'm doing a little testing and I hit a strange point. Here is a zvol (clone) pool1/volclone type volume - pool1/volclone origin pool1/v...@diff1 - pool1/volclone reservation none default pool1/volclone volsize 191G - pool1/volclone volblocksize 8K - The zvol has UFS on it. It has always been 191G and we've never attempted to resize it. However, if I just try to grow it, it gives me an error: -bash-3.00# growfs /dev/zvol/rdsk/pool1/volclone400555998 sectors current size of 400556032 sectors Is the zvol is somehow smaller than it was originally? How/why? I think ufs requires an integer number of cylinder groups and I'm guessing the volume size you have presented it with is somewhere in between - so it has rounded down to the largest cylinder group boundary less than or equal to the device size. Gavin ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
My dump device is already on a different controller - the motherboards built-in nVidia SATA controller. The raidz2 vdev is the one I'm having trouble with (copying the same files to the mirrored rpool on the nVidia controller work nicely). I do notice that, when using cp to copy the files to the raidz2 pool, load on the machine climbs steadily until the crash, and one proc core pegs at 100%. Frustrating, yes. On Thu, Mar 12, 2009 at 12:31 AM, Maidak Alexander J maidakalexand...@johndeere.com wrote: If you're having issues with a disk contoller or disk IO driver its highly likely that a savecore to disk after the panic will fail. I'm not sure how to work around this, maybe a dedicated dump device not on a controller that uses a different driver then the one that you're having issues with? -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Blake Sent: Wednesday, March 11, 2009 4:45 PM To: Richard Elling Cc: Marc Bevand; zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] reboot when copying large amounts of data I guess I didn't make it clear that I had already tried using savecore to retrieve the core from the dump device. I added a larger zvol for dump, to make sure that I wasn't running out of space on the dump device: r...@host:~# dumpadm Dump content: kernel pages Dump device: /dev/zvol/dsk/rpool/bigdump (dedicated) Savecore directory: /var/crash/host Savecore enabled: yes I was using the -L option only to try to get some idea of why the system load was climbing to 1 during a simple file copy. On Wed, Mar 11, 2009 at 4:58 PM, Richard Elling richard.ell...@gmail.com wrote: Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. savecore -L is not what you want. By default, for OpenSolaris, savecore on boot is disabled. But the core will have been dumped into the dump slice, which is not used for swap. So you should be able to run savecore at a later time to collect the core from the last dump. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] reboot when copying large amounts of data
Hm - Crashes, or hangs? Moreover - how do you know a CPU is pegged? Seems like we could do a little more discovery on what the actual problem here is, as I can read it about 4 different ways. By this last piece of information, I'm guessing the system does not crash, but goes really really slow?? Crash == panic == we see stack dump on console and try to take a dump hang == nothing works == no response - might be worth looking at mdb -K or booting with a -k on the boot line. So - are we crashing, hanging, or something different? It might simply be that you are eating up all your memory, and your physical backing storage is taking a while to catch up? Nathan. Blake wrote: My dump device is already on a different controller - the motherboards built-in nVidia SATA controller. The raidz2 vdev is the one I'm having trouble with (copying the same files to the mirrored rpool on the nVidia controller work nicely). I do notice that, when using cp to copy the files to the raidz2 pool, load on the machine climbs steadily until the crash, and one proc core pegs at 100%. Frustrating, yes. On Thu, Mar 12, 2009 at 12:31 AM, Maidak Alexander J maidakalexand...@johndeere.com wrote: If you're having issues with a disk contoller or disk IO driver its highly likely that a savecore to disk after the panic will fail. I'm not sure how to work around this, maybe a dedicated dump device not on a controller that uses a different driver then the one that you're having issues with? -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Blake Sent: Wednesday, March 11, 2009 4:45 PM To: Richard Elling Cc: Marc Bevand; zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] reboot when copying large amounts of data I guess I didn't make it clear that I had already tried using savecore to retrieve the core from the dump device. I added a larger zvol for dump, to make sure that I wasn't running out of space on the dump device: r...@host:~# dumpadm Dump content: kernel pages Dump device: /dev/zvol/dsk/rpool/bigdump (dedicated) Savecore directory: /var/crash/host Savecore enabled: yes I was using the -L option only to try to get some idea of why the system load was climbing to 1 during a simple file copy. On Wed, Mar 11, 2009 at 4:58 PM, Richard Elling richard.ell...@gmail.com wrote: Blake wrote: I'm attaching a screenshot of the console just before reboot. The dump doesn't seem to be working, or savecore isn't working. On Wed, Mar 11, 2009 at 11:33 AM, Blake blake.ir...@gmail.com wrote: I'm working on testing this some more by doing a savecore -L right after I start the copy. savecore -L is not what you want. By default, for OpenSolaris, savecore on boot is disabled. But the core will have been dumped into the dump slice, which is not used for swap. So you should be able to run savecore at a later time to collect the core from the last dump. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- // // Nathan Kroenert nathan.kroen...@sun.com // // Systems Engineer Phone: +61 3 9869-6255 // // Sun Microsystems Fax:+61 3 9869-6288 // // Level 7, 476 St. Kilda Road Mobile: 0419 305 456// // Melbourne 3004 VictoriaAustralia // // ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
