Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Just FWIW, the proper way to do online expansion with ZFS is quite different than that of a RAID controller, as doing a regular OCE process is quite difficult in ZFS (due to the variable-size stripes that ZFS uses, which makes the simple expansion of regular RAID-5 or RAID-6 not workable - or so I've read on the ZFS mailing lists) The way that I believe you can add capacity to an existing RAIDZ / RAIDZ2 is as follows: (this is based directly off of their documentation: http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view) Your ZFS pool is made up of vdevs, which are in turn made up of raw disks / partitions / unicorns, other block devices. To increase the capacity of your existing pool, you simply add another vdev to it, and ZFS will intelligently start distributing the data between the two vdevs. The example, straight from their documentation (with some commentary from me) is an existing ZFS pool named rpool, which as 3 disks and is in a RAIDZ configuration # zpool status pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 errors: No known data errors Now, we want to add three *new* disks to the storage pool - to do this, we simply add a new RAIDZ device comprised of our three new disks into the existing pool: # zpool add rpool raidz c2t2d0 c2t3d0 c2t4d0 # zpool status pool: rpool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c2t2d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 c2t4d0 ONLINE 0 0 0 errors: No known data errors If you're looking for more info on ZFS, I suggest looking at the Sun ZFS administration guide: http://docs.sun.com/app/docs/doc/819-5461/gavwn?a=browse Particularly, the section #5, on ZFS Storage Pools: http://docs.sun.com/app/docs/doc/819-5461/gavwn?a=view Charles Richards richar...@gmail.com charlesrichards.net On Jan 3, 2009, at 5:19 AM, Frederique Rijsdijk wrote: After some reading, I come back from my original idea. Main reason is I'd like to be able to grow the fs as the need develops in time. One could create a raidz zpool with a couple of disks, but when adding a disk later on, it will not become part of the raidz (I tested this). It seems vdevs can not be nested (create raidz sets and join them as a whole), so I came up with the following: Start out with 4*1TB, and use geom_raid5 to create an independent redundant pool of storage: 'graid5 label -v graid5a da0 da1 da2 da3' (this is all tested in vmware, one of these 'da' drives is 8GB) Then I 'zpool create bigvol /dev/raid5/graid5a', and I have a / bigvol of 24G - sounds about right to me for a raid5 volume. Now lets say later in time I need more storage, I buy another 4 of these drives, and 'graid5 label -v graid5b da4 da5 da6 da7' and 'zpool add bigvol /dev/raid5/graid5b' Now my bigvol is 48G. Very cool! Now I have redundant storage that can grow and it's pretty easy too. Is this OK (besides from the fact that graid5 is not in production yet, nor is ZFS ;) or are there easier (or better) ways to do this? - So I want redundancy (I don't want one failing drive to cause me to loose all my data) - I want to be able to grow the filesystem if I need to, by adding a (set of) drive(s) later on. -- FR ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Charles Richards wrote: If his data (photo / video storage) is that important, then perhaps he wants to do soemthing else ... but for a cheap way to get tons of storage, ZFS can't be beat. Of course I'm backing up the very important data on a daily basis (de photo's mainly), but the videos (which will take up the biggest part of course) are not critical. I just want a huge volume to store it on. If some sort of redundancy can be built into that, that's very nice. I've been using GEOM (gconcat) so far, but I'd like to move to ZFS due to it's ease of use (apart from tuning your system to it) and also because GEOM is not able to provide any raid5-like setups. Just raid3 or mirror. ZFS has got it's caveats and gotchas - you *must* tune your FreeBSD installation to get stability. See here: http://wiki.freebsd.org/ZFSTuningGuide I was aware of that, I'll look in to it. I'm running a fileserver-in-a-closet (patent pending) on FreeBSD 7.0-amd64, with 10x250GB drives in a single RAIDZ2 for my home storage needs. I'm using old Maxtor SATA150 drives, which are desktop class. Several of them have had to remap sectors while being a part of the array, and I've never had ZFS complain, nor had the drive be dropped by the OS. Ok that's good information. I also read/noticed that ZFS seems to run best on AMD64 platforms. That's OK then. I'm running i386 now, but I'm happy to switch. I'd suggest that the OP and yourself do some in-depth reading about ZFS and how it works. The best documentation I've found as yet is here: http://opensolaris.org/os/community/zfs/docs/ondiskformat0822.pdf Will do, thanks again. -- FR ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
On Sat, 03 Jan 2009 10:00:09 +0100 Frederique Rijsdijk frederi...@isafeelin.org wrote: I also read/noticed that ZFS seems to run best on AMD64 platforms. That's not quite correct. ZFS runs best on 64bits platforms. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv104 ++ + All that's really worth doing is what we do for others (Lewis Carrol) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Hello About your zfs experience... please let me tell mine.. I was having toubles with ufs2 on some small servers (desktop grade) machines running postgresql (64 bits, or 32 bits) all running FreeBSD 7.X some still 6.X... the problem is that sometimes the database brokes because the ufs2 fsck wipes out the pg_log files (that holds metadata, and so the database is lost/compromiesed) it used to be aobout 1 or 2 times a week, well you willl say that it is very often... but there are more thatn 1200 servers... that is about a database problem in 16 years the machines are in remote zones and cannot stop.. So a month ago I decided to try zfs... first on onpensolaris (that, as expected, works very well...) than in a set of 4 machines running FreeBSD 7.0/7.1 (a small partition to boot, than filesystem / on zfs). 1 of them with 2gb of memory, the others with 1g, 512m, 256m (I know that zfs is unstable with less than 512mb...) but it is only for testing the machines varies from amd64 to celeron... (64 and 32 bits..) NO RAID on the small machine, a buildworld lasted 2 days... All running a database (test of course) without no break. with a custom application that updates tables (several thousand rows, with foreing keys and triggers...) and then rollback... the drives area always with access light on... Sometimes (several times a day, random..) the machines are switched off.. without shutdown... and than switched on again... about (5 -10 times) that is about 60 * 8 - 480 power on/power off cycles each machine Well. I can say that I still have not lost a database... it is incredible fast, reliable comes up without any fsck wait time. in less than a minute... Now I will start to put it on Dells and those PERC controllers... That is my experience... Sergio ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
After some reading, I come back from my original idea. Main reason is I'd like to be able to grow the fs as the need develops in time. One could create a raidz zpool with a couple of disks, but when adding a disk later on, it will not become part of the raidz (I tested this). It seems vdevs can not be nested (create raidz sets and join them as a whole), so I came up with the following: Start out with 4*1TB, and use geom_raid5 to create an independent redundant pool of storage: 'graid5 label -v graid5a da0 da1 da2 da3' (this is all tested in vmware, one of these 'da' drives is 8GB) Then I 'zpool create bigvol /dev/raid5/graid5a', and I have a /bigvol of 24G - sounds about right to me for a raid5 volume. Now lets say later in time I need more storage, I buy another 4 of these drives, and 'graid5 label -v graid5b da4 da5 da6 da7' and 'zpool add bigvol /dev/raid5/graid5b' Now my bigvol is 48G. Very cool! Now I have redundant storage that can grow and it's pretty easy too. Is this OK (besides from the fact that graid5 is not in production yet, nor is ZFS ;) or are there easier (or better) ways to do this? - So I want redundancy (I don't want one failing drive to cause me to loose all my data) - I want to be able to grow the filesystem if I need to, by adding a (set of) drive(s) later on. -- FR ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Hi freebsd-questions, For personal use (photo/video storage), I'm looking into creating a huge single ZFS (raidz) volume that will replace my current collection of drives used as storage. I'm thinking 4*1TB drives in RAID5(z). think twice before doing. My question is regarding the flavour of drivers that one can choose from: Desktop class drives, or the so called RAID/Enterprise class drives. but cheap drives they are OK. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Wojciech Puchar woj...@wojtek.tensor.gdynia.pl writes: Hi, think twice before doing. Could you elaborate please ? Regards Éric Masson -- [Linux] c'est une philosophie un art de vivre, un état intérieur, une sorte de fluide qui nous entoure et nous pénètre. Fais tourner stp ! -+- Guillaume in Guide du linuxien pervers - Tous drogués j'vous dis ! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
On Fri, 02 Jan 2009 16:40:38 +0100 Eric Masson e...@free.fr wrote: Wojciech Puchar woj...@wojtek.tensor.gdynia.pl writes: think twice before doing. Could you elaborate please ? Not again this anti-zfs story please.. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv104 ++ + All that's really worth doing is what we do for others (Lewis Carrol) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
think twice before doing. Could you elaborate please ? ZFS still doesn't work as described ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Wojciech Puchar woj...@wojtek.tensor.gdynia.pl writes: think twice before doing. Could you elaborate please ? Not again this anti-zfs story please.. already described it some time ago. it's not my data anyway ;) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
On Fri, Jan 02, 2009 at 05:48:27PM +0100, Wojciech Puchar wrote: think twice before doing. Could you elaborate please ? ZFS still doesn't work as described ... Is that comment FreeBSD specifc, or aimed at ZFS in general? -- One of the main causes of the fall of the roman empire was that, lacking zero, they had no way to indicate successful termination of their C programs. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
2009/1/3 Frederique Rijsdijk frederi...@isafeelin.org: Hi freebsd-questions, For personal use (photo/video storage), I'm looking into creating a huge single ZFS (raidz) volume that will replace my current collection of drives used as storage. I'm thinking 4*1TB drives in RAID5(z). My question is regarding the flavour of drivers that one can choose from: Desktop class drives, or the so called RAID/Enterprise class drives. The difference between the two being the way such a drive handles the bad-sector/block handling and remapping. I understand that Desktop class drives do all this internally, and this is a process that can take up to 60s (even minutes on some), and during this process the drive is unavailable to the controller. The RAID edition drives all appoach this differently and alot faster, typically before 8 seconds. How does ZFS handle this? Should I be looking for the RAID class drives or can Desktop class drives be used here? My worry is of course that such a drive (destkop class) will be marked defective and thrown out of the raid volume if a remapping of bad sectors occurs and the drive will be unresponsive to the controller/ZFS for 8 seconds. Some drives can be configured in this area, but not all, and there's quite a price difference in the two, the desktop class being up to 50% cheaper in some cases.. Anybody that can shed some light on this? Thanks, -- Frederique Hi, Before i knew the difference between the two, i got myself a bunch of desktop HDD. From what I've experience, freebsd just drops the drive. (Currently running in a gmirror config). I'm not sure about ZFS, but i would assume it would do the same. All you need to the do reattach the drive and it will sync back up again. I didn't know the reason why it dropped off, but when i checked the SMART, it showed 1 bad sector reallocation. If it happens to a disk with UFS, it crashes and restarts the machine, UFS doesn't like disappearing drives. Regards David N ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
On Fri, 2 Jan 2009 15:17:30 -0500 stan st...@panix.com wrote: On Fri, Jan 02, 2009 at 05:48:27PM +0100, Wojciech Puchar wrote: think twice before doing. Could you elaborate please ? ZFS still doesn't work as described ... Is that comment FreeBSD specifc, or aimed at ZFS in general? Mind you, ZFS on FreeBSD is not the same as on OpenSolaris-2008.11, Nevada or even Solaris 10. On those platforms ZFS generally does what it is supposed to do, other than it's still a developing FS. On *BSD related systems that is not always the case. Do a good readup. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv104 ++ + All that's really worth doing is what we do for others (Lewis Carrol) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
think twice before doing. Why? I've had better luck with ZFS than I've had with VINUM and GEOM in the past, and I've put my ZFS array through alot of stress. If his data (photo / video storage) is that important, then perhaps he wants to do soemthing else ... but for a cheap way to get tons of storage, ZFS can't be beat. ZFS has got it's caveats and gotchas - you *must* tune your FreeBSD installation to get stability. See here: http://wiki.freebsd.org/ZFSTuningGuide I'm running a fileserver-in-a-closet (patent pending) on FreeBSD 7.0- amd64, with 10x250GB drives in a single RAIDZ2 for my home storage needs. I'm using old Maxtor SATA150 drives, which are desktop class. Several of them have had to remap sectors while being a part of the array, and I've never had ZFS complain, nor had the drive be dropped by the OS. I'd suggest that the OP and yourself do some in-depth reading about ZFS and how it works. The best documentation I've found as yet is here: http://opensolaris.org/os/community/zfs/docs/ondiskformat0822.pdf Charles Richards richar...@gmail.com charlesrichards.net On Jan 2, 2009, at 8:07 AM, Wojciech Puchar wrote: Hi freebsd-questions, For personal use (photo/video storage), I'm looking into creating a huge single ZFS (raidz) volume that will replace my current collection of drives used as storage. I'm thinking 4*1TB drives in RAID5(z). My question is regarding the flavour of drivers that one can choose from: Desktop class drives, or the so called RAID/Enterprise class drives. but cheap drives they are OK. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
ZFS still doesn't work as described ... Is that comment FreeBSD specifc, or aimed at ZFS in general? general. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
dick hoogendijk wrote: On Fri, 2 Jan 2009 15:17:30 -0500 stan st...@panix.com wrote: On Fri, Jan 02, 2009 at 05:48:27PM +0100, Wojciech Puchar wrote: think twice before doing. Could you elaborate please ? ZFS still doesn't work as described ... Is that comment FreeBSD specifc, or aimed at ZFS in general? Mind you, ZFS on FreeBSD is not the same as on OpenSolaris-2008.11, Nevada or even Solaris 10. On those platforms ZFS generally does what it is supposed to do, other than it's still a developing FS. On *BSD related systems that is not always the case. Do a good readup. I had problems with ZFS about a year ago (or so). Since then, for me, ZFS has been quite reliable: amanda# zpool list NAMESIZEUSED AVAILCAP HEALTH ALTROOT storage1.82T 1.21T623G66% ONLINE - amanda# zpool status NAMESTATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz1ONLINE 0 0 0 ad0 ONLINE 0 0 0 ad2 ONLINE 0 0 0 ad4 ONLINE 2 0 0 ad6 ONLINE 0 1 0 ...with four drives as such (I'd call them 'resi' or 'home-user' quality: ad2: 476940MB WDC WD5000AAKS-00A7B0 01.03B01 at ata1-master SATA300 This machine, which runs AMANDA backup archiver, backing up ~8 FreeBSD servers at about 120Mbps network every night is: amanda# uname -a FreeBSD amanda.x 7.0-STABLE FreeBSD 7.0-STABLE #0: Thu Jul 17 15:24:40 UTC 2008 st...@x:/usr/obj/usr/src/sys/GENERIC i386 I've pushed the machine to 686Mbps network @225kpps, including FBSD SCP and Windows NetBIOS clients while running iperf on other boxen and was still able to write/read to the storage. Instead of this one-liner crap 'don't do it' information to the users of this list, lets begin explaining *why* its not working, and start providing coherent solutions as to how the OP can work around the issue, huh? Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Using HDD's for ZFS: 'desktop' vs 'raid / enterprise' -edition drives?
Steve Bertrand wrote: dick hoogendijk wrote: On Fri, 2 Jan 2009 15:17:30 -0500 stan st...@panix.com wrote: On Fri, Jan 02, 2009 at 05:48:27PM +0100, Wojciech Puchar wrote: think twice before doing. Could you elaborate please ? ZFS still doesn't work as described ... Is that comment FreeBSD specifc, or aimed at ZFS in general? Mind you, ZFS on FreeBSD is not the same as on OpenSolaris-2008.11, Nevada or even Solaris 10. On those platforms ZFS generally does what it is supposed to do, other than it's still a developing FS. On *BSD related systems that is not always the case. Do a good readup. I had problems with ZFS about a year ago (or so). Since then, for me, ZFS has been quite reliable: I forgot to mention... the Since then should have also included that at the time of unreliability, I was testing a new NVidia motherboard. Since then, I've swapped out the board to an Intel hardware platform. Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
