Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-05-03 Thread Richard Jacobsen

Hi Paul,

I have been testing ZoL for a while now (somewhere around a year?) on 
two separate machines:


1) dual Socket 771 Xeon , 8GB ECC RAM, 12 Seagate 1TB ES.2 HD (2x6 disk 
raidz2), ubuntu oneiric, with the zfs-native/stable PPA
2) Intel Xeon CPU E31120, 8GB ECC RAM, 4 x 400GB WD RE2 ( 1 4 disk 
raidz1), ubuntu oneiric, zfs-native/daily PPA


I would consider neither the daily nor the stable good enough for 
production use.


I frequently get pools with all or nearly all of the disks marked as 
removed (a simple zfs export; zfs import -f fixes this).  I also got 
kernel panics under heavy random IO (backuppc) on #1.  I recently 
switched to openindiana on that machine, which is stable for the same 
workload.


On machine #2 (my office workstation) I get routine crashes and slow 
performance (the raidz holds my home directory)


That being said, I haven't lost any data yet, and bugs that have been 
affecting me are quickly fixed, so I think that at some point it will be 
stable, just not right now.


Richard

On 04/25/2012 05:48 AM, Paul Archer wrote:
This may fall into the realm of a religious war (I hope not!), but 
recently several people on this list have said/implied that ZFS was 
only acceptable for production use on FreeBSD (or Solaris, of course) 
rather than Linux with ZoL.


I'm working on a project at work involving a large(-ish) amount of 
data, about 5TB, working its way up to 12-15TB eventually, spread 
among a dozen or so nodes. There may or may not be a clustered 
filesystem involved (probably gluster if we use anything). I've been 
looking at ZoL as the primary filesystem for this data. We're a Linux 
shop, so I'd rather not switch to FreeBSD, or any of the 
Solaris-derived distros--although I have no problem with them, I just 
don't want to introduce another OS into the mix if I can avoid it.


So, the actual questions are:

Is ZoL really not ready for production use?

If not, what is holding it back? Features? Performance? Stability?

If not, then what kind of timeframe are we looking at to get past 
whatever is holding it back?

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Ray Van Dolson
On Wed, Apr 25, 2012 at 05:48:57AM -0700, Paul Archer wrote:
 This may fall into the realm of a religious war (I hope not!), but
 recently several people on this list have said/implied that ZFS was
 only acceptable for production use on FreeBSD (or Solaris, of course)
 rather than Linux with ZoL.
 
 I'm working on a project at work involving a large(-ish) amount of
 data, about 5TB, working its way up to 12-15TB eventually, spread
 among a dozen or so nodes. There may or may not be a clustered
 filesystem involved (probably gluster if we use anything). I've been
 looking at ZoL as the primary filesystem for this data. We're a Linux
 shop, so I'd rather not switch to FreeBSD, or any of the
 Solaris-derived distros--although I have no problem with them, I just
 don't want to introduce another OS into the mix if I can avoid it.
 
 So, the actual questions are:
 
 Is ZoL really not ready for production use?
 
 If not, what is holding it back? Features? Performance? Stability?
 
 If not, then what kind of timeframe are we looking at to get past
 whatever is holding it back?

I can't comment directly on experiences with ZoL as I haven't used it,
but it does seem to be under active development.  That can be a good
thing or a bad thing. :)

I for one would be hesitant to use it for anything production based
solely on the youngness of the effort.

That said, might be worthwhile to check out the ZoL mailing lists and
bug reports to see what types of issues the early adopters are running
into and whether or not they are showstoppers for you or you are
willing to accept the risks.

For your size requierements and your intent to use Gluster, it sounds
like ext4 or xfs would be entirely suitable and are obviously more
mature on Linux at this point.

Regardless, curious to hear which way you end up going and how things
work out.

Ray
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Richard Elling
On Apr 25, 2012, at 5:48 AM, Paul Archer wrote:

 This may fall into the realm of a religious war (I hope not!), but recently 
 several people on this list have said/implied that ZFS was only acceptable 
 for production use on FreeBSD (or Solaris, of course) rather than Linux with 
 ZoL.
 
 I'm working on a project at work involving a large(-ish) amount of data, 
 about 5TB, working its way up to 12-15TB

This is pretty small by today's standards.  With 4TB disks, that is only 3-4 
disks + redundancy.

 eventually, spread among a dozen or so nodes. There may or may not be a 
 clustered filesystem involved (probably gluster if we use anything).

I wouldn't dream of building a clustered file system that small. Maybe when you 
get into the
multiple-PB range, then it might make sense.

 I've been looking at ZoL as the primary filesystem for this data. We're a 
 Linux shop, so I'd rather not switch to FreeBSD, or any of the 
 Solaris-derived distros--although I have no problem with them, I just don't 
 want to introduce another OS into the mix if I can avoid it.
 
 So, the actual questions are:
 
 Is ZoL really not ready for production use?
 
 If not, what is holding it back? Features? Performance? Stability?

The computer science behind ZFS is sound. But it was also developed for Solaris 
which
is quite different than Linux under the covers. So the Linux and other OS ports 
have issues
around virtual memory system differences and fault management differences. This 
is the
classic getting it to work is 20% of the effort, getting it to work when all 
else is failing is
the other 80% case.
 -- richard


--
ZFS Performance and Training
richard.ell...@richardelling.com
+1-760-896-4422







___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Paul Archer

9:59am, Richard Elling wrote:


On Apr 25, 2012, at 5:48 AM, Paul Archer wrote:

  This may fall into the realm of a religious war (I hope not!), but 
recently several people on this list have
  said/implied that ZFS was only acceptable for production use on FreeBSD 
(or Solaris, of course) rather than Linux
  with ZoL.

  I'm working on a project at work involving a large(-ish) amount of data, 
about 5TB, working its way up to 12-15TB


This is pretty small by today's standards.  With 4TB disks, that is only 3-4 
disks + redundancy.

True. At my last job, we were used to researchers asking for individual 4-5TB 
filesystems, and 1-2TB increases in size. When I left, there was over a 100TB 
online (in '07).




  eventually, spread among a dozen or so nodes. There may or may not be a 
clustered filesystem involved (probably
  gluster if we use anything).


I wouldn't dream of building a clustered file system that small. Maybe when you 
get into the
multiple-PB range, then it might make sense.

The point of a clustered filesystem was to be able to spread our data out 
among all nodes and still have access from any node without having to run NFS. 
Size of the data set (once you get past the point where you can replicate it 
on each node) is irrelevant.





  I've been looking at ZoL as the primary filesystem for this data. We're a 
Linux shop, so I'd rather not switch to
  FreeBSD, or any of the Solaris-derived distros--although I have no 
problem with them, I just don't want to
  introduce another OS into the mix if I can avoid it.

  So, the actual questions are:

  Is ZoL really not ready for production use?

  If not, what is holding it back? Features? Performance? Stability?


The computer science behind ZFS is sound. But it was also developed for Solaris 
which
is quite different than Linux under the covers. So the Linux and other OS ports 
have issues
around virtual memory system differences and fault management differences. This 
is the
classic getting it to work is 20% of the effort, getting it to work when all 
else is failing is
the other 80% case.
 -- richard


I understand the 80/20 rule. But this doesn't really answer the question(s). 
If there weren't any major differences among operating systems, the project 
probably would have been done long ago.


To put it slightly differently, if I used ZoL in production, would I be likely 
to experience performance or stability problems? Or would it be lacking in 
features that I would likely need?___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Richard Elling
On Apr 25, 2012, at 10:59 AM, Paul Archer wrote:

 9:59am, Richard Elling wrote:
 
 On Apr 25, 2012, at 5:48 AM, Paul Archer wrote:
 
  This may fall into the realm of a religious war (I hope not!), but 
 recently several people on this list have
  said/implied that ZFS was only acceptable for production use on FreeBSD 
 (or Solaris, of course) rather than Linux
  with ZoL.
 
  I'm working on a project at work involving a large(-ish) amount of 
 data, about 5TB, working its way up to 12-15TB
 This is pretty small by today's standards.  With 4TB disks, that is only 3-4 
 disks + redundancy.
 True. At my last job, we were used to researchers asking for individual 4-5TB 
 filesystems, and 1-2TB increases in size. When I left, there was over a 100TB 
 online (in '07).

100TB is medium sized for today's systems, about 4RU or less :-)

  eventually, spread among a dozen or so nodes. There may or may not be a 
 clustered filesystem involved (probably
  gluster if we use anything).
 I wouldn't dream of building a clustered file system that small. Maybe when 
 you get into the
 multiple-PB range, then it might make sense.
 
 The point of a clustered filesystem was to be able to spread our data out 
 among all nodes and still have access from any node without having to run 
 NFS. Size of the data set (once you get past the point where you can 
 replicate it on each node) is irrelevant.

Interesting, something more complex than NFS to avoid the complexities of NFS? 
;-)

  I've been looking at ZoL as the primary filesystem for this data. We're 
 a Linux shop, so I'd rather not switch to
  FreeBSD, or any of the Solaris-derived distros--although I have no 
 problem with them, I just don't want to
  introduce another OS into the mix if I can avoid it.
 
  So, the actual questions are:
 
  Is ZoL really not ready for production use?
 
  If not, what is holding it back? Features? Performance? Stability?
 The computer science behind ZFS is sound. But it was also developed for 
 Solaris which
 is quite different than Linux under the covers. So the Linux and other OS 
 ports have issues
 around virtual memory system differences and fault management differences. 
 This is the
 classic getting it to work is 20% of the effort, getting it to work when 
 all else is failing is
 the other 80% case.
  -- richard
 
 I understand the 80/20 rule. But this doesn't really answer the question(s). 
 If there weren't any major differences among operating systems, the project 
 probably would have been done long ago.

The issues are not only technical :-(

 To put it slightly differently, if I used ZoL in production, would I be 
 likely to experience performance or stability problems? Or would it be 
 lacking in features that I would likely need?

It seems reasonably stable for the casual use cases. 

As for the features, that is a much more difficult question to answer. For 
example, if
you use ACLs, you might find that some userland tools on some distros have full 
or 
no support for ACLs.

Let us know how it works out for you.
 -- richard

--
ZFS Performance and Training
richard.ell...@richardelling.com
+1-760-896-4422







___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Jordan Schwartz
To put it slightly differently, if I used ZoL in production, would I be
likely to experience performance or stability problems?

I saw one team revert from ZoL (CentOS 6) back to ext on some backup
servers for an application project, the killer  was
stat times (find running slow etc.), perhaps more layer 2 cache could have
solved the problem, but it was easier to deploy ext/lvm2.
The source filesystems were ext so zfs send/rcv was not an option.

You may want to check with the ZoL project about where there development is
with respect to performance, I heard that the focus
was on stability.


Jordan



On Wed, Apr 25, 2012 at 10:59 AM, Paul Archer p...@paularcher.org wrote:

 9:59am, Richard Elling wrote:

  On Apr 25, 2012, at 5:48 AM, Paul Archer wrote:

  This may fall into the realm of a religious war (I hope not!), but
 recently several people on this list have
  said/implied that ZFS was only acceptable for production use on
 FreeBSD (or Solaris, of course) rather than Linux
  with ZoL.

  I'm working on a project at work involving a large(-ish) amount of
 data, about 5TB, working its way up to 12-15TB


 This is pretty small by today's standards.  With 4TB disks, that is only
 3-4 disks + redundancy.

  True. At my last job, we were used to researchers asking for individual
 4-5TB filesystems, and 1-2TB increases in size. When I left, there was over
 a 100TB online (in '07).



   eventually, spread among a dozen or so nodes. There may or may not
 be a clustered filesystem involved (probably
  gluster if we use anything).


 I wouldn't dream of building a clustered file system that small. Maybe
 when you get into the
 multiple-PB range, then it might make sense.

  The point of a clustered filesystem was to be able to spread our data
 out among all nodes and still have access from any node without having to
 run NFS. Size of the data set (once you get past the point where you can
 replicate it on each node) is irrelevant.




   I've been looking at ZoL as the primary filesystem for this data.
 We're a Linux shop, so I'd rather not switch to
  FreeBSD, or any of the Solaris-derived distros--although I have no
 problem with them, I just don't want to
  introduce another OS into the mix if I can avoid it.

  So, the actual questions are:

  Is ZoL really not ready for production use?

  If not, what is holding it back? Features? Performance? Stability?


 The computer science behind ZFS is sound. But it was also developed for
 Solaris which
 is quite different than Linux under the covers. So the Linux and other OS
 ports have issues
 around virtual memory system differences and fault management
 differences. This is the
 classic getting it to work is 20% of the effort, getting it to work when
 all else is failing is
 the other 80% case.
  -- richard


 I understand the 80/20 rule. But this doesn't really answer the
 question(s). If there weren't any major differences among operating
 systems, the project probably would have been done long ago.

 To put it slightly differently, if I used ZoL in production, would I be
 likely to experience performance or stability problems? Or would it be
 lacking in features that I would likely need?
 ___
 zfs-discuss mailing list
 zfs-discuss@opensolaris.org
 http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Stefan Ring
 I saw one team revert from ZoL (CentOS 6) back to ext on some backup servers
 for an application project, the killer  was
 stat times (find running slow etc.), perhaps more layer 2 cache could have
 solved the problem, but it was easier to deploy ext/lvm2.

But stat times (think directory traversal) are horrible on ZFS/Solaris
as well, at least on a workstation-class machine that doesn't run
24/7. Maybe on an always-on server with 256GB RAM or more, things
would be different. For me, that's really the only pain point of using
ZFS.

Sorry for not being able to contribute any ZoL experience. I've been
pondering whether it's worth trying for a few months myself already.
Last time I checked, it didn't support the .zfs directory (for
snapshot access), which you really don't want to miss after getting
used to it.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Paul Archer

To put it slightly differently, if I used ZoL in production, would I be likely 
to experience performance or stability
problems?

I saw one team revert from ZoL (CentOS 6) back to ext on some backup servers 
for an application project, the killer  was
stat times (find running slow etc.), perhaps more layer 2 cache could have 
solved the problem, but it was easier to deploy
ext/lvm2.


Hmm... I've got 1.4TB in about 70K files in 2K directories, and a simple find 
on a cold FS took me about 6 seconds:


root@hoard22:/hpool/12/db# time find . -type d | wc
df -h
   20822082   32912

real0m5.923s
user0m0.052s
sys 0m1.012s


So I'd say I'm doing OK there. But I've got 10K disks and a fast SSD for 
caching.___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Paul Archer

9:08pm, Stefan Ring wrote:


Sorry for not being able to contribute any ZoL experience. I've been
pondering whether it's worth trying for a few months myself already.
Last time I checked, it didn't support the .zfs directory (for
snapshot access), which you really don't want to miss after getting
used to it.

Actually, rc8 (or was it rc7?) introduced/implemented the .zfs directory. If 
you're upgrading, you need to reboot,  but other than that, it works 
perfectly.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] ZFS on Linux vs FreeBSD

2012-04-25 Thread Nico Williams
As I understand it LLNL has very large datasets on ZFS on Linux.  You
could inquire with them, as well as
http://groups.google.com/a/zfsonlinux.org/group/zfs-discuss/topics?pli=1
.  My guess is that it's quite stable for at least some use cases
(most likely: LLNL's!), but that may not be yours.  You could
always... test it, but if you do then please tell us how it went :)

Nico
--
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss