Re: [zfs-discuss] ZFS on Linux vs FreeBSD
Hi Paul, I have been testing ZoL for a while now (somewhere around a year?) on two separate machines: 1) dual Socket 771 Xeon , 8GB ECC RAM, 12 Seagate 1TB ES.2 HD (2x6 disk raidz2), ubuntu oneiric, with the zfs-native/stable PPA 2) Intel Xeon CPU E31120, 8GB ECC RAM, 4 x 400GB WD RE2 ( 1 4 disk raidz1), ubuntu oneiric, zfs-native/daily PPA I would consider neither the daily nor the stable good enough for production use. I frequently get pools with all or nearly all of the disks marked as removed (a simple zfs export; zfs import -f fixes this). I also got kernel panics under heavy random IO (backuppc) on #1. I recently switched to openindiana on that machine, which is stable for the same workload. On machine #2 (my office workstation) I get routine crashes and slow performance (the raidz holds my home directory) That being said, I haven't lost any data yet, and bugs that have been affecting me are quickly fixed, so I think that at some point it will be stable, just not right now. Richard On 04/25/2012 05:48 AM, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? If not, then what kind of timeframe are we looking at to get past whatever is holding it back? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS on Linux vs FreeBSD
This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? If not, then what kind of timeframe are we looking at to get past whatever is holding it back? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
On Wed, Apr 25, 2012 at 05:48:57AM -0700, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? If not, then what kind of timeframe are we looking at to get past whatever is holding it back? I can't comment directly on experiences with ZoL as I haven't used it, but it does seem to be under active development. That can be a good thing or a bad thing. :) I for one would be hesitant to use it for anything production based solely on the youngness of the effort. That said, might be worthwhile to check out the ZoL mailing lists and bug reports to see what types of issues the early adopters are running into and whether or not they are showstoppers for you or you are willing to accept the risks. For your size requierements and your intent to use Gluster, it sounds like ext4 or xfs would be entirely suitable and are obviously more mature on Linux at this point. Regardless, curious to hear which way you end up going and how things work out. Ray ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
On Apr 25, 2012, at 5:48 AM, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB This is pretty small by today's standards. With 4TB disks, that is only 3-4 disks + redundancy. eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I wouldn't dream of building a clustered file system that small. Maybe when you get into the multiple-PB range, then it might make sense. I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? The computer science behind ZFS is sound. But it was also developed for Solaris which is quite different than Linux under the covers. So the Linux and other OS ports have issues around virtual memory system differences and fault management differences. This is the classic getting it to work is 20% of the effort, getting it to work when all else is failing is the other 80% case. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
9:59am, Richard Elling wrote: On Apr 25, 2012, at 5:48 AM, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB This is pretty small by today's standards. With 4TB disks, that is only 3-4 disks + redundancy. True. At my last job, we were used to researchers asking for individual 4-5TB filesystems, and 1-2TB increases in size. When I left, there was over a 100TB online (in '07). eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I wouldn't dream of building a clustered file system that small. Maybe when you get into the multiple-PB range, then it might make sense. The point of a clustered filesystem was to be able to spread our data out among all nodes and still have access from any node without having to run NFS. Size of the data set (once you get past the point where you can replicate it on each node) is irrelevant. I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? The computer science behind ZFS is sound. But it was also developed for Solaris which is quite different than Linux under the covers. So the Linux and other OS ports have issues around virtual memory system differences and fault management differences. This is the classic getting it to work is 20% of the effort, getting it to work when all else is failing is the other 80% case. -- richard I understand the 80/20 rule. But this doesn't really answer the question(s). If there weren't any major differences among operating systems, the project probably would have been done long ago. To put it slightly differently, if I used ZoL in production, would I be likely to experience performance or stability problems? Or would it be lacking in features that I would likely need?___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
On Apr 25, 2012, at 10:59 AM, Paul Archer wrote: 9:59am, Richard Elling wrote: On Apr 25, 2012, at 5:48 AM, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB This is pretty small by today's standards. With 4TB disks, that is only 3-4 disks + redundancy. True. At my last job, we were used to researchers asking for individual 4-5TB filesystems, and 1-2TB increases in size. When I left, there was over a 100TB online (in '07). 100TB is medium sized for today's systems, about 4RU or less :-) eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I wouldn't dream of building a clustered file system that small. Maybe when you get into the multiple-PB range, then it might make sense. The point of a clustered filesystem was to be able to spread our data out among all nodes and still have access from any node without having to run NFS. Size of the data set (once you get past the point where you can replicate it on each node) is irrelevant. Interesting, something more complex than NFS to avoid the complexities of NFS? ;-) I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? The computer science behind ZFS is sound. But it was also developed for Solaris which is quite different than Linux under the covers. So the Linux and other OS ports have issues around virtual memory system differences and fault management differences. This is the classic getting it to work is 20% of the effort, getting it to work when all else is failing is the other 80% case. -- richard I understand the 80/20 rule. But this doesn't really answer the question(s). If there weren't any major differences among operating systems, the project probably would have been done long ago. The issues are not only technical :-( To put it slightly differently, if I used ZoL in production, would I be likely to experience performance or stability problems? Or would it be lacking in features that I would likely need? It seems reasonably stable for the casual use cases. As for the features, that is a much more difficult question to answer. For example, if you use ACLs, you might find that some userland tools on some distros have full or no support for ACLs. Let us know how it works out for you. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
To put it slightly differently, if I used ZoL in production, would I be likely to experience performance or stability problems? I saw one team revert from ZoL (CentOS 6) back to ext on some backup servers for an application project, the killer was stat times (find running slow etc.), perhaps more layer 2 cache could have solved the problem, but it was easier to deploy ext/lvm2. The source filesystems were ext so zfs send/rcv was not an option. You may want to check with the ZoL project about where there development is with respect to performance, I heard that the focus was on stability. Jordan On Wed, Apr 25, 2012 at 10:59 AM, Paul Archer p...@paularcher.org wrote: 9:59am, Richard Elling wrote: On Apr 25, 2012, at 5:48 AM, Paul Archer wrote: This may fall into the realm of a religious war (I hope not!), but recently several people on this list have said/implied that ZFS was only acceptable for production use on FreeBSD (or Solaris, of course) rather than Linux with ZoL. I'm working on a project at work involving a large(-ish) amount of data, about 5TB, working its way up to 12-15TB This is pretty small by today's standards. With 4TB disks, that is only 3-4 disks + redundancy. True. At my last job, we were used to researchers asking for individual 4-5TB filesystems, and 1-2TB increases in size. When I left, there was over a 100TB online (in '07). eventually, spread among a dozen or so nodes. There may or may not be a clustered filesystem involved (probably gluster if we use anything). I wouldn't dream of building a clustered file system that small. Maybe when you get into the multiple-PB range, then it might make sense. The point of a clustered filesystem was to be able to spread our data out among all nodes and still have access from any node without having to run NFS. Size of the data set (once you get past the point where you can replicate it on each node) is irrelevant. I've been looking at ZoL as the primary filesystem for this data. We're a Linux shop, so I'd rather not switch to FreeBSD, or any of the Solaris-derived distros--although I have no problem with them, I just don't want to introduce another OS into the mix if I can avoid it. So, the actual questions are: Is ZoL really not ready for production use? If not, what is holding it back? Features? Performance? Stability? The computer science behind ZFS is sound. But it was also developed for Solaris which is quite different than Linux under the covers. So the Linux and other OS ports have issues around virtual memory system differences and fault management differences. This is the classic getting it to work is 20% of the effort, getting it to work when all else is failing is the other 80% case. -- richard I understand the 80/20 rule. But this doesn't really answer the question(s). If there weren't any major differences among operating systems, the project probably would have been done long ago. To put it slightly differently, if I used ZoL in production, would I be likely to experience performance or stability problems? Or would it be lacking in features that I would likely need? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
I saw one team revert from ZoL (CentOS 6) back to ext on some backup servers for an application project, the killer was stat times (find running slow etc.), perhaps more layer 2 cache could have solved the problem, but it was easier to deploy ext/lvm2. But stat times (think directory traversal) are horrible on ZFS/Solaris as well, at least on a workstation-class machine that doesn't run 24/7. Maybe on an always-on server with 256GB RAM or more, things would be different. For me, that's really the only pain point of using ZFS. Sorry for not being able to contribute any ZoL experience. I've been pondering whether it's worth trying for a few months myself already. Last time I checked, it didn't support the .zfs directory (for snapshot access), which you really don't want to miss after getting used to it. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
To put it slightly differently, if I used ZoL in production, would I be likely to experience performance or stability problems? I saw one team revert from ZoL (CentOS 6) back to ext on some backup servers for an application project, the killer was stat times (find running slow etc.), perhaps more layer 2 cache could have solved the problem, but it was easier to deploy ext/lvm2. Hmm... I've got 1.4TB in about 70K files in 2K directories, and a simple find on a cold FS took me about 6 seconds: root@hoard22:/hpool/12/db# time find . -type d | wc df -h 20822082 32912 real0m5.923s user0m0.052s sys 0m1.012s So I'd say I'm doing OK there. But I've got 10K disks and a fast SSD for caching.___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
9:08pm, Stefan Ring wrote: Sorry for not being able to contribute any ZoL experience. I've been pondering whether it's worth trying for a few months myself already. Last time I checked, it didn't support the .zfs directory (for snapshot access), which you really don't want to miss after getting used to it. Actually, rc8 (or was it rc7?) introduced/implemented the .zfs directory. If you're upgrading, you need to reboot, but other than that, it works perfectly. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS on Linux vs FreeBSD
As I understand it LLNL has very large datasets on ZFS on Linux. You could inquire with them, as well as http://groups.google.com/a/zfsonlinux.org/group/zfs-discuss/topics?pli=1 . My guess is that it's quite stable for at least some use cases (most likely: LLNL's!), but that may not be yours. You could always... test it, but if you do then please tell us how it went :) Nico -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
