Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-27 Thread Pawel Jakub Dawidek
On Tue, Sep 12, 2006 at 03:56:00PM -0700, Matthew Ahrens wrote: Matthew Ahrens wrote: [...] Given the overwhelming criticism of this feature, I'm going to shelve it for now. I'd really like to see this feature. You say ZFS should change our view on filesystems, I say be consequent. In ZFS

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-20 Thread Wout Mertens
Just a me too mail: On 13 Sep 2006, at 08:30, Richard Elling wrote: Is this use of slightly based upon disk failure modes? That is, when disks fail do they tend to get isolated areas of badness compared to complete loss? I would suggest that complete loss should include someone tripping over

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-19 Thread David Dyer-Bennet
On 9/19/06, Richard Elling - PAE [EMAIL PROTECTED] wrote: [pardon the digression] David Dyer-Bennet wrote: On 9/18/06, Richard Elling - PAE [EMAIL PROTECTED] wrote: Interestingly, the operation may succeed and yet we will get an error which recommends replacing the drive. For example, if

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-19 Thread Torrey McMahon
Richard Elling - PAE wrote: This question was asked many times in this thread. IMHO, it is the single biggest reason we should implement ditto blocks for data. We did a study of disk failures in an enterprise RAID array a few years ago. One failure mode stands heads and shoulders above the

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-19 Thread Richard Elling - PAE
reply below... Torrey McMahon wrote: Richard Elling - PAE wrote: This question was asked many times in this thread. IMHO, it is the single biggest reason we should implement ditto blocks for data. We did a study of disk failures in an enterprise RAID array a few years ago. One failure mode

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-19 Thread Torrey McMahon
Richard Elling - PAE wrote: Non-recoverable reads may not represent permanent failures. In the case of a RAID array, the data should be reconstructed and a rewrite + verify attempted with the possibility of sparing the sector. ZFS can reconstruct the data and relocate the block. True but

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-19 Thread Richard Elling - PAE
Torrey McMahon wrote: Richard Elling - PAE wrote: Non-recoverable reads may not represent permanent failures. In the case of a RAID array, the data should be reconstructed and a rewrite + verify attempted with the possibility of sparing the sector. ZFS can reconstruct the data and relocate

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-18 Thread Richard Elling - PAE
[appologies for being away from my data last week] David Dyer-Bennet wrote: The more I look at it the more I think that a second copy on the same disk doesn't protect against very much real-world risk. Am I wrong here? Are partial(small) disk corruptions more common than I think? I don't have

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-18 Thread David Dyer-Bennet
On 9/18/06, Richard Elling - PAE [EMAIL PROTECTED] wrote: [appologies for being away from my data last week] David Dyer-Bennet wrote: The more I look at it the more I think that a second copy on the same disk doesn't protect against very much real-world risk. Am I wrong here? Are

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-18 Thread Richard Elling - PAE
more below... David Dyer-Bennet wrote: On 9/18/06, Richard Elling - PAE [EMAIL PROTECTED] wrote: [appologies for being away from my data last week] David Dyer-Bennet wrote: The more I look at it the more I think that a second copy on the same disk doesn't protect against very much

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-14 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Ahrens wrote: Out of curiosity, what would you guys think about addressing this same problem by having the option to store some filesystems unreplicated on an mirrored (or raid-z) pool? This would have the same issues of unexpected space

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-14 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Neil A. Wilson wrote: This is unfortunate. As a laptop user with only a single drive, I was looking forward to it since I've been bitten in the past by data loss caused by a bad area on the disk. I don't care about the space consumption because

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Matthew Ahrens
Torrey McMahon wrote: Matthew Ahrens wrote: The problem that this feature attempts to address is when you have some data that is more important (and thus needs a higher level of redundancy) than other data. Of course in some situations you can use multiple pools, but that is antithetical to

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread eric kustarz
David Dyer-Bennet wrote: On 9/12/06, eric kustarz [EMAIL PROTECTED] wrote: So it seems to me that having this feature per-file is really useful. Say i have a presentation to give in Pleasanton, and the presentation lives on my single-disk laptop - I want all the meta-data and the actual

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread eric kustarz
Torrey McMahon wrote: eric kustarz wrote: Matthew Ahrens wrote: Matthew Ahrens wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Thanks everyone for your input. The problem that this feature attempts to

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Mike Gerdts
On 9/13/06, Richard Elling [EMAIL PROTECTED] wrote: * Mirroring offers slightly better redundancy, because one disk from each mirror can fail without data loss. Is this use of slightly based upon disk failure modes? That is, when disks fail do they tend to get isolated areas of badness

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Tobias Schacht
On 9/13/06, Mike Gerdts [EMAIL PROTECTED] wrote: The only part of the proposal I don't like is space accounting. Double or triple charging for data will only confuse those apps and users that check for free space or block usage. Why exactly isn't reporting the free space divided by the copies

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Al Hopper
On Tue, 12 Sep 2006, Matthew Ahrens wrote: Torrey McMahon wrote: Matthew Ahrens wrote: The problem that this feature attempts to address is when you have some data that is more important (and thus needs a higher level of redundancy) than other data. Of course in some situations you can

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Bill Sommerfeld
On Wed, 2006-09-13 at 02:30, Richard Elling wrote: The field data I have says that complete disk failures are the exception. I hate to leave this as a teaser, I'll expand my comments later. That matches my anecdotal experience with laptop drives; maybe I'm just lucky, or maybe I'm just paying

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Torrey McMahon
eric kustarz wrote: I want per pool, per dataset, and per file - where all are done by the filesystem (ZFS), not the application. I was talking about a further enhancement to copies than what Matt is currently proposing - per file copies, but its more work (one thing being we don't have

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Bart Smaalders
Torrey McMahon wrote: eric kustarz wrote: I want per pool, per dataset, and per file - where all are done by the filesystem (ZFS), not the application. I was talking about a further enhancement to copies than what Matt is currently proposing - per file copies, but its more work (one thing

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-13 Thread Torrey McMahon
Bart Smaalders wrote: Torrey McMahon wrote: eric kustarz wrote: I want per pool, per dataset, and per file - where all are done by the filesystem (ZFS), not the application. I was talking about a further enhancement to copies than what Matt is currently proposing - per file copies, but

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Darren J Moffat
Mike Gerdts wrote: On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: B. DESCRIPTION A new property will be added, 'copies', which specifies how many copies of the given filesystem will be stored. Its value must be 1, 2, or 3. Like other properties (eg. checksum, compression), it only

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Dick Davies
On 12/09/06, Matthew Ahrens [EMAIL PROTECTED] wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Your comments are appreciated! Flexibility is always nice, but this seems to greatly complicate things, both

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Dick Davies
On 12/09/06, Darren J Moffat [EMAIL PROTECTED] wrote: Dick Davies wrote: The only real use I'd see would be for redundant copies on a single disk, but then why wouldn't I just add a disk? Some systems have physical space for only a single drive - think most laptops! True - I'm a laptop

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Jeff Victor
This proposal would benefit greatly by a problem statement. As it stands, it feels like a solution looking for a problem. The Introduction mentions a different problem and solution, but then pretends that there is value to this solution. The Description section mentions some benefits of

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread David Dyer-Bennet
On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Your comments are appreciated! I've read the proposal, and followed the discussion so far. I have to say that I

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Neil A. Wilson
Darren J Moffat wrote: While encryption of existing data is not in scope for the first ZFS crypto phase I am being careful in the design to ensure that it can be done later if such a ZFS framework becomes available. The biggest problem I see with this is one of observability, if not all of

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Darren J Moffat
Neil A. Wilson wrote: Darren J Moffat wrote: While encryption of existing data is not in scope for the first ZFS crypto phase I am being careful in the design to ensure that it can be done later if such a ZFS framework becomes available. The biggest problem I see with this is one of

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Nicolas Williams
On Tue, Sep 12, 2006 at 10:36:30AM +0100, Darren J Moffat wrote: Mike Gerdts wrote: Is there anything in the works to compress (or encrypt) existing data after the fact? For example, a special option to scrub that causes the data to be re-written with the new properties could potentially do

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Matthew Ahrens
Matthew Ahrens wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Thanks everyone for your input. The problem that this feature attempts to address is when you have some data that is more important (and thus

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread David Dyer-Bennet
On 9/12/06, Matthew Ahrens [EMAIL PROTECTED] wrote: Matthew Ahrens wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Thanks everyone for your input. The problem that this feature attempts to address is when you

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread Neil A. Wilson
Matthew Ahrens wrote: Matthew Ahrens wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Thanks everyone for your input. The problem that this feature attempts to address is when you have some data that is more

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread eric kustarz
Matthew Ahrens wrote: Matthew Ahrens wrote: Here is a proposal for a new 'copies' property which would allow different levels of replication for different filesystems. Thanks everyone for your input. The problem that this feature attempts to address is when you have some data that is

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-12 Thread David Dyer-Bennet
On 9/12/06, eric kustarz [EMAIL PROTECTED] wrote: So it seems to me that having this feature per-file is really useful. Say i have a presentation to give in Pleasanton, and the presentation lives on my single-disk laptop - I want all the meta-data and the actual presentation to be replicated.

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-11 Thread Mike Gerdts
On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: B. DESCRIPTION A new property will be added, 'copies', which specifies how many copies of the given filesystem will be stored. Its value must be 1, 2, or 3. Like other properties (eg. checksum, compression), it only affects newly-written

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-11 Thread Matthew Ahrens
James Dickens wrote: On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: B. DESCRIPTION A new property will be added, 'copies', which specifies how many copies of the given filesystem will be stored. Its value must be 1, 2, or 3. Like other properties (eg. checksum, compression), it only

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-11 Thread Matthew Ahrens
Mike Gerdts wrote: Is there anything in the works to compress (or encrypt) existing data after the fact? For example, a special option to scrub that causes the data to be re-written with the new properties could potentially do this. This is a long-term goal of ours, but with snapshots, this

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-11 Thread James Dickens
On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: James Dickens wrote: On 9/11/06, Matthew Ahrens [EMAIL PROTECTED] wrote: B. DESCRIPTION A new property will be added, 'copies', which specifies how many copies of the given filesystem will be stored. Its value must be 1, 2, or 3. Like

Re: [zfs-discuss] Proposal: multiple copies of user data

2006-09-11 Thread Matthew Ahrens
James Dickens wrote: though I think this is a cool feature, I think i needs more work. I think there sould be an option to make extra copies expendible. So the extra copies are a request, if the space is availible make them, if not complete the write, and log the event. Are you asking for the