Re: Ideas for a feature implementation
On Aug 13, 2014, at 5:01 AM, David Pottage wrote: > At present, you can write RAID 5 or 6 data, but if anything goes wrong, btrfs > cannot use the parity information to help you get your data back, so in > general you are better off with RAID 1 or 10. Btrfs RAID5/6 normally mounted will reconstruct data from parity if there's a data checksum mismatch indicating data block(s) are corrupt. It'll also reconstruct from parity in the case of degraded mounts, and when doing a balance. What's missing is scrub. > Also, I don't think I/O done in parallel so you get no speed advantage from > having multiple discs either. Hmm I thought that was just single and raid1? Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 2014/08/12 17:52, David Pottage wrote: [snip] ... if it does not then the file-system has broken the contract to secure delete a file when you asked it to. This is a technicality - and it has not necessarily "broken the contract". I think the correct thing to do would be to securely delete the metadata referring to that file. That would satisfy the concept that there is no evidence that the file ever existed in that location. The fact that it actually does still legitimately exist elsewhere is not a caveat - it is simply acting within standard behaviour. -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 13/8/2014 2:01 μμ, David Pottage wrote: On 12/08/14 12:00, Konstantinos Skarlatos wrote: Maybe help with Andrea Mazzoleni's New RAID library supporting up to six parities? It seems to be a great feature for btrfs. https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html That would be very cool, but at present vanila RAID 5 or 6 does not work properly, so I think getting that fully working would be a better idea. (Unless it would make more sense to merge the whole lot into one bit of work where RAID 5 or 6 are just a special case of arbitrary parity level support). At present, you can write RAID 5 or 6 data, but if anything goes wrong, btrfs cannot use the parity information to help you get your data back, so in general you are better off with RAID 1 or 10. Also, I don't think I/O done in parallel so you get no speed advantage from having multiple discs either. Yeah, thats one of the features I am waiting to get finished, because I already have 5 multi disk systems that i would prefer to migrate to RAID5/6 from RAID1/JBOD that they are now. I dont know what is the best sequencing, I just think that these are great patches/features and its a pity for them to languish. -- Konstantinos Skarlatos -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 12/08/14 12:00, Konstantinos Skarlatos wrote: Maybe help with Andrea Mazzoleni's New RAID library supporting up to six parities? It seems to be a great feature for btrfs. https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html That would be very cool, but at present vanila RAID 5 or 6 does not work properly, so I think getting that fully working would be a better idea. (Unless it would make more sense to merge the whole lot into one bit of work where RAID 5 or 6 are just a special case of arbitrary parity level support). At present, you can write RAID 5 or 6 data, but if anything goes wrong, btrfs cannot use the parity information to help you get your data back, so in general you are better off with RAID 1 or 10. Also, I don't think I/O done in parallel so you get no speed advantage from having multiple discs either. -- David Pottage -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 2014-08-12 11:52, David Pottage wrote: > > On 11/08/14 03:53, Austin S Hemmelgarn wrote: > >> Another thing that isn't listed there, that I would personally love to >> see is support for secure file deletion. To be truly secure though, >> this would need to hook into the COW logic so that files marked for >> secure deletion can't be reflinked (maybe make the automatically NOCOW >> instead, and don't allow snapshots?), and when they get written to, the >> blocks that get COW'ed have the old block overwritten. > How would secure deletion interact with file de-duplication? > > For example suppose you and I are both users on a multi user system. We > both obtain copies of the same file independently, and save that file to > our home directories. > > A background process notices that both files are the same and > de-duplicates them. This means that both your file and mine point to the > same blocks on disc. This is exactly the same as would happen if you > made a COW copy of your file, transferred ownership to me, and I moved > it into my home dir. > > You then decide to secure delete your copy of the file. What happens to > mine? If it gets removed, then you have just deleted a file you don't > own, if it does not then the file-system has broken the contract to > secure delete a file when you asked it to. > > Also, what happens if the two files have similar portions, but they are > not identical. For example, if you download and ISO image for ubuntu, > and I download the ISO for kubuntu (at the same version). There will be > a lot of sections that are the same, because they will contain a lot of > packages in common, so there will be large gains in de-duplicating the > similar parts, but most people would consider the files to be different. > > Could this mean that if you secure delete your ubuntu iso, then portions > of my kubuntu iso might become corrupt? > You could work around this by marking the extent, instead of the file (marking a file would mark all of it's extents), and then checking for that marking when the extent is freed (ie, nobody refers to it anymore). While this approach might not seem useful to most people, there are practical use cases for it (even without whole disk encryption). It would be pretty easy actually to integrate this globally for a file-system as a mount option. > Even if we limit secure delete to root, then we still leave the risk of > unintentonaly breaking user files, because non-one realised that all or > part of the file appears in other files via de-duplication. In any case > if secure delete is limited to root, then most people would not find it > useful. (or they would use sudo to do it, which brings us back to the > same problems). > > Basically, I think that file secure deletion as a concept is not > compatible with a 5th generation file system. If you relay want to > securely remove a file, then copy the stuff you need elsewhere, and put > the disc in the crusher. Alternatively put the filesystem in an encypted > container, and then reformat the disc with a different encryption key. > While I agree that the traditional notion of secure deletion doesn't fit in the current generation of file systems, there is still a need for COW filesystems to be able to prevent sensitive data from being exposed during run-time. On any current BTRFS filesystem, it is still possible to find blocks that have been COW'ed (assuming discard is turned off) and have no referents, possibly long after the block itself is freed, and especially if the volume is much larger than the stored data set (like a large majority of desktop users these days) or the workload is not write intensive. smime.p7s Description: S/MIME Cryptographic Signature
Re: Ideas for a feature implementation
On 11/08/14 03:53, Austin S Hemmelgarn wrote: Another thing that isn't listed there, that I would personally love to see is support for secure file deletion. To be truly secure though, this would need to hook into the COW logic so that files marked for secure deletion can't be reflinked (maybe make the automatically NOCOW instead, and don't allow snapshots?), and when they get written to, the blocks that get COW'ed have the old block overwritten. How would secure deletion interact with file de-duplication? For example suppose you and I are both users on a multi user system. We both obtain copies of the same file independently, and save that file to our home directories. A background process notices that both files are the same and de-duplicates them. This means that both your file and mine point to the same blocks on disc. This is exactly the same as would happen if you made a COW copy of your file, transferred ownership to me, and I moved it into my home dir. You then decide to secure delete your copy of the file. What happens to mine? If it gets removed, then you have just deleted a file you don't own, if it does not then the file-system has broken the contract to secure delete a file when you asked it to. Also, what happens if the two files have similar portions, but they are not identical. For example, if you download and ISO image for ubuntu, and I download the ISO for kubuntu (at the same version). There will be a lot of sections that are the same, because they will contain a lot of packages in common, so there will be large gains in de-duplicating the similar parts, but most people would consider the files to be different. Could this mean that if you secure delete your ubuntu iso, then portions of my kubuntu iso might become corrupt? Even if we limit secure delete to root, then we still leave the risk of unintentonaly breaking user files, because non-one realised that all or part of the file appears in other files via de-duplication. In any case if secure delete is limited to root, then most people would not find it useful. (or they would use sudo to do it, which brings us back to the same problems). Basically, I think that file secure deletion as a concept is not compatible with a 5th generation file system. If you relay want to securely remove a file, then copy the stuff you need elsewhere, and put the disc in the crusher. Alternatively put the filesystem in an encypted container, and then reformat the disc with a different encryption key. -- David Pottage. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
Austin S Hemmelgarn writes: > Another thing that isn't listed there, that I would personally love to > see is support for secure file deletion. As discussed in the followups, it is difficult to do a really secure deletion and a security-conscious person would opt to use encryption instead. Ability to encrypt subvolumes or files would probably serve those needs sufficiently, though it sounds like it could easily be a longer task than 4 months. -- _ / __// /__ __ http://www.modeemi.fi/~flux/\ \ / /_ / // // /\ \/ /\ / /_/ /_/ \___/ /_/\_\@modeemi.fi \/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 10/8/2014 10:21 μμ, Vimal A R wrote: Hello, I came across the to-do list at https://btrfs.wiki.kernel.org/index.php/Project_ideas and would like to know if this list is updated and recent. I am looking for a project idea for my under graduate degree which can be completed in around 3-4 months. Are there any suggestions and ideas to help me further? Maybe help with Andrea Mazzoleni's New RAID library supporting up to six parities? It seems to be a great feature for btrfs. https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html Thank you, Vimal -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Konstantinos Skarlatos -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On Aug 11, 2014, at 8:27 PM, Austin S Hemmelgarn wrote: > On 08/11/2014 04:27 PM, Chris Murphy wrote: >> >> On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn >> wrote: >> >>> >>> Another thing that isn't listed there, that I would personally >>> love to see is support for secure file deletion. To be truly >>> secure though, this would need to hook into the COW logic so that >>> files marked for secure deletion can't be reflinked (maybe make >>> the automatically NOCOW instead, and don't allow snapshots?), and >>> when they get written to, the blocks that get COW'ed have the old >>> block overwritten. >> >> If the file is reflinked or snapshot, then it can it be secure >> deleted? Because what does it mean to secure delete a file when >> there's a completely independent file pointing to the same physical >> blocks? What if someone else owns that independent file? Does the >> reflink copy get rm'd as well? Or does the file remain, but its >> blocks are zero'd/corrupted? > The semantics that I would expect would be that the extents can't be > reflinked, and when snapshotted the whole file gets COW'ed, and then > inherits the secure deletion flag, possibly with another flag saying > that the user can't disable the secure deletion flag. Ahh OK I was thinking of a secure delete command (or an option to rm that indicates secure delete). You're suggesting one or more flags that makes for secure file handling, not just delete, affecting: a.) copied b.) moved, c.) snapshot/reflinked, d.) deleted. So if deleted, a regular rm would "see" the xattr and do a secure delete; and the xattr would inhibit or limit the others. While a reflink or normal copy could be inhibited, the snapshot case seems more difficult because it just creates a new tree. It's not scanning the tree for files/folders with xattr, which would have to be done to go retroactively remove the file set with the secure delete flag - could be really slow. And what if the snapshot is made read-only? Strictly secure delete, e.g. rm -s, would be more straightforward than a flag affecting other filesystem operations. >> For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open >> question when the data is actually irretrievable. It may be >> seconds, but could be much longer (hours?) so I'm not sure if it's >> useful. On HDD's using SMR it's not necessarily a given an >> overwrite will work there either. > By secure deletion, I don't mean make the data absolutely > unrecoverable by any means, I mean make it functionally impractical > for someone without low-level access to and/or extensive knowledge of > the hardware to recover the data; that is, more secure than simply > unlinking the file, but obviously less than (for example) the > application of thermite to the disk platters. I'm talking the rough > equivalent of wiping the data from RAM. > > Anyone who is truly security minded should be using whole disk > encryption anyway, but even then you have the data accessible from the > running OS. Seems straightforward for any file system already supporting discard. This even has a useful application for thinly provisioned storage and large files where you'd want the underlying logical layer to free up extents sooner than later - even if you didn't care about the security aspect. But for that matter, on SSDs right now you can rm the file and then fstrim the file system to get the same effect. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 08/11/2014 04:27 PM, Chris Murphy wrote: > > On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn > wrote: > >> >> Another thing that isn't listed there, that I would personally >> love to see is support for secure file deletion. To be truly >> secure though, this would need to hook into the COW logic so that >> files marked for secure deletion can't be reflinked (maybe make >> the automatically NOCOW instead, and don't allow snapshots?), and >> when they get written to, the blocks that get COW'ed have the old >> block overwritten. > > If the file is reflinked or snapshot, then it can it be secure > deleted? Because what does it mean to secure delete a file when > there's a completely independent file pointing to the same physical > blocks? What if someone else owns that independent file? Does the > reflink copy get rm'd as well? Or does the file remain, but its > blocks are zero'd/corrupted? The semantics that I would expect would be that the extents can't be reflinked, and when snapshotted the whole file gets COW'ed, and then inherits the secure deletion flag, possibly with another flag saying that the user can't disable the secure deletion flag. > > For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open > question when the data is actually irretrievable. It may be > seconds, but could be much longer (hours?) so I'm not sure if it's > useful. On HDD's using SMR it's not necessarily a given an > overwrite will work there either. By secure deletion, I don't mean make the data absolutely unrecoverable by any means, I mean make it functionally impractical for someone without low-level access to and/or extensive knowledge of the hardware to recover the data; that is, more secure than simply unlinking the file, but obviously less than (for example) the application of thermite to the disk platters. I'm talking the rough equivalent of wiping the data from RAM. Anyone who is truly security minded should be using whole disk encryption anyway, but even then you have the data accessible from the running OS. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn wrote: > > Another thing that isn't listed there, that I would personally love to > see is support for secure file deletion. To be truly secure though, > this would need to hook into the COW logic so that files marked for > secure deletion can't be reflinked (maybe make the automatically NOCOW > instead, and don't allow snapshots?), and when they get written to, the > blocks that get COW'ed have the old block overwritten. If the file is reflinked or snapshot, then it can it be secure deleted? Because what does it mean to secure delete a file when there's a completely independent file pointing to the same physical blocks? What if someone else owns that independent file? Does the reflink copy get rm'd as well? Or does the file remain, but its blocks are zero'd/corrupted? For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open question when the data is actually irretrievable. It may be seconds, but could be much longer (hours?) so I'm not sure if it's useful. On HDD's using SMR it's not necessarily a given an overwrite will work there either. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ideas for a feature implementation
On 08/10/2014 03:21 PM, Vimal A R wrote: > Hello, > > I came across the to-do list at > https://btrfs.wiki.kernel.org/index.php/Project_ideas and would like to know > if this list is updated and recent. > > I am looking for a project idea for my under graduate degree which can be > completed in around 3-4 months. Are there any suggestions and ideas to help > me further? > > Thank you, > Vimal It's not really listed there (though some of the projects there might be considered subsets of it), but improved parallelization for the multi-device setups is one thing that I know that a lot of people would like to see. Another thing that isn't listed there, that I would personally love to see is support for secure file deletion. To be truly secure though, this would need to hook into the COW logic so that files marked for secure deletion can't be reflinked (maybe make the automatically NOCOW instead, and don't allow snapshots?), and when they get written to, the blocks that get COW'ed have the old block overwritten. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html