Re: [zfs-discuss] Zil on multiple usb keys
On 07/23/11 04:57, Michael DeMan wrote: Generally performance is going to pretty bad as well - USB sticks are not made to be written too rapidly. They are entirely different animals than SSDs. I would not be surprised (but would be curious to know if you still move forward on this) that you will find performance even worse trying to do this. Back in the snv_120 ish era I tried this experiement on both my pool and on a friends. In both cases we were serving NFS (he was also doing CIFS) which was mostly read but also had periods where 1-2 G of data was rapidly added (uploading photos or videos) over the network. In both the USB flash drive and in the case of a San Disk Extreme IV CF card in a CF-IDE enclosure the performance did not improve and in fact in the case of the CF card the enclosure was bugging such that the changes we had to make to the ata config did actually make it slower. I removed the separate log device from both of those pools (by manual hacking with specially build zfs kernel modules because slog removal didn't exist back then.). -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
+1 on the below, and in addition... ...compact flash, like off of USB sticks is not designed to deal with very many writes to it. Commonly it is used to store a bootable image that maybe once a year will have an upgrade on it. Basically, trying to use those devices for a ZIL, even they are mirrored - you should be prepared to having one die and be replaced very, very regularly. Generally performance is going to pretty bad as well - USB sticks are not made to be written too rapidly. They are entirely different animals than SSDs. I would not be surprised (but would be curious to know if you still move forward on this) that you will find performance even worse trying to do this. On Jul 18, 2011, at 1:54 AM, Fajar A. Nugraha wrote: First of all, using USB disks for permanent storage is a bad idea. Go for e-sata instead (http://en.wikipedia.org/wiki/Serial_ata#eSATA). It ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) The right way to do it, um, incorrectly is to create a striped device using SVM, and use that as a vdev for your pool. So yes, you could create two 800GB stripes, and use them to create a ZFS mirror. But it would be a really bad idea. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb mirrored striped thing is sounding like a bad idea... what about just creating a pool of all disks, without using mirrors? I seen something called copies, which if i am reading correctly, will make sure a number of copies of a file exist... Am i reading that correctly? If this does work the way i think it works, then taking all 4 disks, and making one large 1.6Tb pool, setting copies to 2, should, in theory, create a poor mans pool with striping, right? --Tiernan On Mon, Jul 18, 2011 at 7:20 AM, Brandon High bh...@freaks.com wrote: On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) The right way to do it, um, incorrectly is to create a striped device using SVM, and use that as a vdev for your pool. So yes, you could create two 800GB stripes, and use them to create a ZFS mirror. But it would be a really bad idea. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
On Mon, Jul 18, 2011 at 3:28 PM, Tiernan OToole lsmart...@gmail.com wrote: Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb mirrored striped thing is sounding like a bad idea... what about just creating a pool of all disks, without using mirrors? I seen something called copies, which if i am reading correctly, will make sure a number of copies of a file exist... Am i reading that correctly? If this does work the way i think it works, then taking all 4 disks, and making one large 1.6Tb pool, setting copies to 2, should, in theory, create a poor mans pool with striping, right? Step back a moment. What are your priorites? What is the most important thing for you? Is it space? Is it data protection? Is it something else? Once you determine that, it's easier to come up with a reasonable setup. Back to your original question, I'd like to note some things. First of all, using USB disks for permanent storage is a bad idea. Go for e-sata instead (http://en.wikipedia.org/wiki/Serial_ata#eSATA). It eliminates the overhead caused by USB-to-[P/S]ATA bridge. You can get something like a e-sata bracket (If your controller supports port multiplier) or e-sata PCI controller, plus e-sata enclosure with 2 or 4 drive bays (depending on your needs). Second, using copies=2 + stripe is, again, a bad idea. While copies=2 can protect you from something like bad sector, it will NOT protect you from drive failure. So when one drive broke your pool will still be unaccessible. Stick with strpe of mirrors instead. Go with what Edward suggested: rearrange the disk, and create stripe of (mirror of 500G internal + 500G external) + (mirror of 300G internal + 300G external). Another option, if you go with external enclosure route, is to just put all disks in the external enclosure, and go with the configuration Jim suggested (2 x 200GB mirror, plus 4 x 300GB mirror/raidz1) -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
On Mon, Jul 18, 2011 at 1:20 PM, Brandon High bh...@freaks.com wrote: On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) The right way to do it, um, incorrectly is to create a striped device using SVM, and use that as a vdev for your pool. So yes, you could create two 800GB stripes, and use them to create a ZFS mirror. But it would be a really bad idea. In freebsd you can use geom_stripe or geom_concat to create striped block device. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
2011/7/15 Eugen Leitl eu...@leitl.org: Speaking of which, is there a point in using an eSATA flash stick? If yes, which? It depends on the drive off course, you'll have to look up benchmark results - but there are eSata sticks out there that are more or less built to Perform (as opposed to providing cheap storage). -- Frank Van Damme No part of this copyright message may be reproduced, read or seen, dead or alive or by any means, including but not limited to telepathy without the benevolence of the author. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
If using two mirrors, you'll end up with a badly balanced pool. As it was, and possibly is, this will lead to a performance penalty when one VDEV is full (the 300GB VDEV). This write performance has reportedly been fixed in Illumos, but I don't know about S11ex. For OpenIndiana/Nexenta, the fix probably isn't in yet. roy - Original Message - Ok, so, taking 2 300Gb disks, and 2 500Gb disks, and creating an 800Gb mirrored striped thing is sounding like a bad idea... what about just creating a pool of all disks, without using mirrors? I seen something called copies, which if i am reading correctly, will make sure a number of copies of a file exist... Am i reading that correctly? If this does work the way i think it works, then taking all 4 disks, and making one large 1.6Tb pool, setting copies to 2, should, in theory, create a poor mans pool with striping, right? --Tiernan On Mon, Jul 18, 2011 at 7:20 AM, Brandon High bh...@freaks.com wrote: On Sun, Jul 17, 2011 at 12:13 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) The right way to do it, um, incorrectly is to create a striped device using SVM, and use that as a vdev for your pool. So yes, you could create two 800GB stripes, and use them to create a ZFS mirror. But it would be a really bad idea. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Jim Klimov if the OP were so inclined, he could craft a couple of striped pools (300+500) and then make a ZFS pool over these two. Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
2011-07-17 23:13, Edward Ned Harvey пишет: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Jim Klimov if the OP were so inclined, he could craft a couple of striped pools (300+500) and then make a ZFS pool over these two. Actually, you can't do that. You can't make a vdev from other vdev's, and when it comes to striping and mirroring your only choice is to do it the right way. Yup, that's what i said later in the post. If you were REALLY trying to go out of your way to do it wrong somehow, I suppose you could probably make a zvol from a stripe, and then export it to yourself via iscsi, repeat with another zvol, and then mirror the two iscsi targets. ;-) You might even be able to do the same crazy thing with simply zvol's and no iscsi... But either way you'd really be going out of your way to create a problem. ;-) Theoretically, at least, you could do that. AFAIK you can even do it without iSCSI by just addressing zvol device names with their full paths, an yes after crafting yourself an intricate problem you'd likely get one ;) Not that it's impossible or forbidden to shoot oneself in the foot, as your and my adventures demonstrate quite well ;) //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
Well, not knowing a lot about these, but if the flash stick is based on SSD, then it might work well, but if its just a standard USB key rebundled as a eSATA disk, maybe not... On Fri, Jul 15, 2011 at 5:54 PM, Eugen Leitl eu...@leitl.org wrote: On Fri, Jul 15, 2011 at 04:21:13PM +, Tiernan OToole wrote: This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Speaking of which, is there a point in using an eSATA flash stick? If yes, which? -- Eugen* Leitl a href=http://leitl.org;leitl/a http://leitl.org __ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
Thanks for the info. need to rebuild my machine and ZFS pool kind of new to this and realized i built it as a stripe, not a mirror... also, want to add extra disks... As a follow up question: I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? Thanks. --Tiernan On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it's still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That's the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn't make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set sync= command. It takes effect immediately. If you have an older system, you'll have to use a different command, and you'll probably have to remount your filesystem in order for the change to take effect. -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] (Zil on multiple usb keys) Mirroring the pool
Tiernan, Depending on how you have created your current pool, you *may* be able to add the mirroring without rebuilding it. Each disk in the stripe can have a second disk of equal size attached to it to form a mirrored component, or vdev. So if your pool has 2 500GB drives, attach another 500GB drive to each, forming a mirror of each stripe half. # zpool status pool: mypool state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM mypool ONLINE 0 0 0 c0t50d1 ONLINE 0 0 0 c0t50d2 ONLINE 0 0 0 errors: No known data errors # zpool attach mypool c0t50d1 c0t50d3 # zpool attach mypool c0t50d2 c0t50d4 # zpool status pool: mypool state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Sat Jul 16 07:38:07 2011 config: NAME STATE READ WRITE CKSUM mypool ONLINE 0 0 0 mirror ONLINE 0 0 0 c0t50d1 ONLINE 0 0 0 c0t50d3 ONLINE 0 0 0 mirror ONLINE 0 0 0 c0t50d2 ONLINE 0 0 0 c0t50d4 ONLINE 0 0 0 errors: No known data errors # Both single vdevs (c0t50d1 and c0t50d2) are now mirrored. If you don't have a second disk appropriately sized to match the current pool members, you can create one or two pools with your two 500GB and two 300GB disks,, depending on your needs. Either: # zpool create pool1 mirror 500GB-1 500GB-2 mirror 300GB-1 300GB-2 to make one ~800GB pool. Or # zpool create pool1 mirror 500GB-1 500GB-2 # zpool create pool2 mirror 300GB-1 300GB-2 to make two pools, one ~500GB and one ~300GB. As long as the mirrored pairs match they do not have to be all the same in the pool. Craig Tiernan OToole wrote: Thanks for the info. need to rebuild my machine and ZFS pool kind of new to this and realized i built it as a stripe, not a mirror... also, want to add extra disks... As a follow up question: I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? Thanks. --Tiernan On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it's still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That's the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn't make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set sync= command. It takes effect immediately. If you have an older system, you'll have to use a different command, and you'll probably have to remount your filesystem in order for the change to take effect. -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Craig Cory Senior Instructor :: ExitCertified : Oracle/Sun Certified System Administrator : Oracle/Sun Certified Network Administrator : Oracle/Sun Certified Security Administrator : Symantec/Veritas Certified Instructor : RedHat Certified Systems Administrator 8950 Cal Center Drive Bldg 1, Suite 110 Sacramento, California 95826 [e] craig.c...@exitcertified.com [p] 916.669.3970 [f] 916.669.3977 +-+ ExitCertified :: Excellence in IT Certified Education Certified training with Oracle, Sun Microsystems, Apple, Symantec, IBM, Red Hat, MySQL, Hitachi Storage, SpringSource and VMWare. 1.800.803.EXIT (3948) | www.ExitCertified.com
Re: [zfs-discuss] Zil on multiple usb keys
From: Tiernan OToole [mailto:lsmart...@gmail.com] Sent: Saturday, July 16, 2011 7:46 AM I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? I think the risk of accidental disconnection is higher on the USB drive. So I would recommend swapping the disks inside the enclosures... One 500 inside, one 500 outside, one 300 inside, one 300 outside. Mirror the 500G drives to each other, mirror the 300g drives to each other. That way, if you accidentally disconnect one or both of the external drives, you just plug it back in and everything moves forward without any problem. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
2011-07-16 15:46, Tiernan OToole ?: Thanks for the info. need to rebuild my machine and ZFS pool kind of new to this and realized i built it as a stripe, not a mirror... also, want to add extra disks... As a follow up question: I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? Is there a typo? It would rather be a 2*300Gb mirror and a 2*500Gb mirror, with a stripe above them as much as writes can get balanced. That would work (with forcing on command-line), is possible, moderately recommmended because unbalanced setups can have more issues than usual (hence you must use the force to enable such setup). And just in case, this pool can not be a bootable rpool. You might make a 2*200Gb slice mirror for an rpool and a more balanced 4*300Gb pool of any layout (raid10, raidz123)... As for using USB sticks, I started my unlucky setup with some sticks used as L2ARC, and about once a week the device got lost (possibly because a stick could slide a bit from its contact bay on the chassis - BIOS also did not see the stick until it was re-plugged). Loss of a device would also hang my pool for quite a long while... Thanks. --Tiernan On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com mailto:opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org mailto:zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- mailto:zfs-discuss- boun...@opensolaris.org mailto:boun...@opensolaris.org] On Behalf Of Tiernan OToole This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it's still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That's the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn't make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set sync= command. It takes effect immediately. If you have an older system, you'll have to use a different command, and you'll probably have to remount your filesystem in order for the change to take effect. -- Tiernan O'Toole blog.lotas-smartman.net http://blog.lotas-smartman.net www.tiernanotoolephotography.com http://www.tiernanotoolephotography.com www.the-hairy-one.com http://www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- ++ || | ?? ???, Jim Klimov | | ??? CTO | | ??? ??? ? ?? JSC COSHT | || | +7-903-7705859 (cellular) mailto:jimkli...@cos.ru | | CC:ad...@cos.ru,jimkli...@mail.ru | ++ | () ascii ribbon campaign - against html mail | | /\- against microsoft attachments | ++ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
So, i like the sound of that, but the box is a very frankinbox like... it has 2 SATA ports, one used for the boot drive, one for one of the 500s... the second 500Gb is IDE. The 2 USB drives both internally are SATA, so pulling one and plugging it internally wont work that well... but thats for the info. On Sat, Jul 16, 2011 at 3:09 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: Tiernan OToole [mailto:lsmart...@gmail.com] Sent: Saturday, July 16, 2011 7:46 AM I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? I think the risk of accidental disconnection is higher on the USB drive. So I would recommend swapping the disks inside the enclosures... One 500 inside, one 500 outside, one 300 inside, one 300 outside. Mirror the 500G drives to each other, mirror the 300g drives to each other. That way, if you accidentally disconnect one or both of the external drives, you just plug it back in and everything moves forward without any problem. -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
thats not a typo... I was thinking 2 pools, 800gb each, and mirrored... think i should mess around with this setup a bit more and see what i can get working... might work better if i just move them into a new enclosure... we see what happens... Thanks for the info on the USB drives... if the ZIL drive falls over, does ZFS not recover well? do i need to reboot fully? Thanks. --Tienan On Sat, Jul 16, 2011 at 3:23 PM, Jim Klimov jimkli...@cos.ru wrote: 2011-07-16 15:46, Tiernan OToole пишет: Thanks for the info. need to rebuild my machine and ZFS pool kind of new to this and realized i built it as a stripe, not a mirror... also, want to add extra disks... As a follow up question: I have 2 500Gb internal drives and 2 300Gb USB drives. If i where to create a 2 pools, a 300Gb and a 500Gb in each, and then mirror over them, would that work? is it even posible? or what would you recomend for that setup? Is there a typo? It would rather be a 2*300Gb mirror and a 2*500Gb mirror, with a stripe above them as much as writes can get balanced. That would work (with forcing on command-line), is possible, moderately recommmended because unbalanced setups can have more issues than usual (hence you must use the force to enable such setup). And just in case, this pool can not be a bootable rpool. You might make a 2*200Gb slice mirror for an rpool and a more balanced 4*300Gb pool of any layout (raid10, raidz123)... As for using USB sticks, I started my unlucky setup with some sticks used as L2ARC, and about once a week the device got lost (possibly because a stick could slide a bit from its contact bay on the chassis - BIOS also did not see the stick until it was re-plugged). Loss of a device would also hang my pool for quite a long while... Thanks. --Tiernan On Fri, Jul 15, 2011 at 5:39 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it's still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That's the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn't make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set sync= command. It takes effect immediately. If you have an older system, you'll have to use a different command, and you'll probably have to remount your filesystem in order for the change to take effect. -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing listzfs-discuss@opensolaris.orghttp://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- ++ || | Климов Евгений, Jim Klimov | | технический директор CTO | | ЗАО ЦОС и ВТ JSC COSHT | || | +7-903-7705859 (cellular) mailto:jimkli...@cos.ru jimkli...@cos.ru | | CC:ad...@cos.ru,jimkli...@mail.ru | ++ | () ascii ribbon campaign - against html mail | | /\- against microsoft attachments | ++ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Tiernan O'Toole blog.lotas-smartman.net www.tiernanotoolephotography.com www.the-hairy-one.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
2011-07-16 18:48, Tiernan OToole пишет: thats not a typo... I was thinking 2 pools, 800gb each, and mirrored... think i should mess around with this setup a bit more and see what i can get working... might work better if i just move them into a new enclosure... we see what happens... Well, in terms of mirroring over stripes, if any component of any stripe breaks, the whole half of the mirror is degraded. If another drive from another half also breaks, you're in trouble. Overall raid01 is assumed to be less reliable than raid10, especially if there are more than two couples of drives. Thanks for the info on the USB drives... if the ZIL drive falls over, does ZFS not recover well? do i need to reboot fully? In theory, you should have 2 drives (slices) mirrored for a ZIL. If one breaks, the other goes on. If all break, the pool reverts to on-disk ZIL areas in the course of several TXG syncs (a few seconds). So the big problem is constrained in poweroff/panic of the system in these few seconds when no ZIL is used. But in practice it may take ZFS a long time to detect the failure/death of the device, because it is not ZFS itself doing it - lower-level drivers like sd must detect and report the problem or the timeout... After all, I did not try a ZIL, or not for a long enough time to be certain of anything. From theory again, its critical property is fast writes with low latency, and reliability. Even in comparison with a dedicated HDD slice, my USB stick could not compete with either. But that was only a test setup, so the sticks were not optimized or fast (3-7Mb/s). You might want to look at CF-IDE adapters - new professional photo CF cards mention 600x (90Mb/s) speeds at reads and 30 to 90Mb/s writes, and some internal redundancy. Probably not cheap though... But many small mobos include a CF port, or a small adapter can be plugged into an IDE port. There were some adapters which naturally split one IDE port into master and slave CF ports. Or see a PCMCIA-CF port for a laptop... Or maybe even use a USB-CF card reader thingie. -- ++ || | Климов Евгений, Jim Klimov | | технический директор CTO | | ЗАО ЦОС и ВТ JSC COSHT | || | +7-903-7705859 (cellular) mailto:jimkli...@cos.ru | | CC:ad...@cos.ru,jimkli...@mail.ru | ++ | () ascii ribbon campaign - against html mail | | /\- against microsoft attachments | ++ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Jim Klimov Well, in terms of mirroring over stripes, if any component of any stripe breaks, the whole half of the mirror is degraded. If another drive from another half also breaks, you're in trouble. There's no such thing in ZFS. You have a series of mirrors in the pool, and if one side of one mirror breaks, no big deal. If another side of another mirror breaks, no big deal. The only problem is when you lose both sides of a single mirror. Now please, nobody say this is either raid10 or raid01. Because it's neither one. The definition of striping according to raid0 does not exist in ZFS - but the essence is preserved and improved upon. In a strictly defined raid0, you have a set number of devices which are all the same size. It benefits large sequential operations, but it hurts small operations. You cannot expand by simply adding more devices, and they all must be the same size. The ZFS concept of striping is more like a combination of raid0 striping and concatenation ... Preserve the best parts of each and throw away the bad parts of each. Optimize for both small operations and serial operations, expand with any size disk at any time. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Zil on multiple usb keys
This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Thanks. Sent from a fruity device ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Tiernan OToole This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Unfortunately, usb keys, even the fastest ones, are slower than physical hard drives. I even went out of my way to buy a super expensive super fast USB3 16G fob... And it's still slower than a super-cheap USB2 sata hard drive. There is a way you can evaluate the effect of adding a fast slog device without buying one. (It would have to be a fast device, certainly no USB fobs.) Just temporarily disable your ZIL. That's the fastest you can possibly go. If it makes a big difference, then getting a fast slog device will help you approach that theoretical limit. If it doesn't make a huge difference, then adding slog will not do you any good. To disable ZIL, if your pool is sufficiently recent, use the zfs set sync= command. It takes effect immediately. If you have an older system, you'll have to use a different command, and you'll probably have to remount your filesystem in order for the change to take effect. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Zil on multiple usb keys
On Fri, Jul 15, 2011 at 04:21:13PM +, Tiernan OToole wrote: This might be a stupid question, but here goes... Would adding, say, 4 4 or 8gb usb keys as a zil make enough of a difference for writes on an iscsi shared vol? I am finding reads are not too bad (40is mb/s over gige on 2 500gb drives stripped) but writes top out at about 10 and drop a lot lower... If I where to add a couple usb keys for zil, would it make a difference? Speaking of which, is there a point in using an eSATA flash stick? If yes, which? -- Eugen* Leitl a href=http://leitl.org;leitl/a http://leitl.org __ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
