Re: svn commit: r214431 - head/bin/rm
On Thu, 28 Oct 2010 14:11:14 -0700 Garrett Cooper gcoo...@freebsd.org wrote: Unfortunately it's implied superficially by the 3 pass tort; but as most people who understand magnetic disks know, unless you completely obliterate a disk, wipe over it with random data enough times, whatever, the content is still on the disk and retrievable via various methods... I agree that this advice should be placed near the flag itself so that people completely understand the implications of the feature. I believe the only method 10-15 years ago would have been a scanning electron microscope, but that probably isn't possible with today's disks. Simply writing zeros once is enough to obliterate all the data (except for any remapped sectors). -- Bruce Cran ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Fri, 29 Oct 2010, Bruce Cran wrote: On Thu, 28 Oct 2010 14:11:14 -0700 Garrett Cooper gcoo...@freebsd.org wrote: Unfortunately it's implied superficially by the 3 pass tort; but as most people who understand magnetic disks know, unless you completely obliterate a disk, wipe over it with random data enough times, whatever, the content is still on the disk and retrievable via various methods... I agree that this advice should be placed near the flag itself so that people completely understand the implications of the feature. I believe the only method 10-15 years ago would have been a scanning electron microscope, but that probably isn't possible with today's disks. Simply writing zeros once is enough to obliterate all the data (except for any remapped sectors). Especially given modern magnetic disk densities. However, the problem with flash memory is quite interesting -- because of device wear-leveling. People who really care about forensic extraction of flash disks bypass the normal interface in order to (a) follow best practices on not powering on devices and (b) bypass the wear-levelling abstraction, revealing the underlying disk blocks and wear-levelling meta-data. As a result, they often have access to large numbers of believed-deleted and even believed-overwritten blocks (although how much depends on the algorithm, fill ratio, usage patterns, etc). Robert ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
Quoting Garrett Cooper gcoo...@freebsd.org (from Thu, 28 Oct 2010 14:11:14 -0700): Unfortunately it's implied superficially by the 3 pass tort; but as most people who understand magnetic disks know, unless you completely obliterate a disk, wipe over it with random data enough times, whatever, the content is still on the disk and retrievable via various methods... Can you please elaborate? I've read somewhere that someone (some kind of computer magazine) made a test of this (yes, to much somewhere and someone, sorry). They took a full harddisk with random data (random as in pictures, music, ...) and overwrote it once with zeros and gave it to one (or several) data rescue companies. The result was that for a price which makes still sense to pay for a business (like a bank), only a fraction of data (as in non-zero bytes) was recoverable, and that none of the original files, and no useful sequence of data, was recovered. For myself I kept the info that I've did not read it at an untrusty place and that for any normal person (even with some malicious intent to remove traces of the existence of something) overwritting a harddisk one time completely is enough. Bye, Alexander. -- Christ died for our sins, so let's not disappoint Him. http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Thu, 28 Oct 2010 16:22:05 +1100 (EST) Bruce Evans b...@optusnet.com.au wrote: On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. -- Gary Jennejohn ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Thu, Oct 28, 2010 at 09:55:38AM +0200, Gary Jennejohn wrote: On Thu, 28 Oct 2010 16:22:05 +1100 (EST) Bruce Evans b...@optusnet.com.au wrote: On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. I was about to suggest the same thing. Sounds like -P requires far too many ducks to be lined up in order to work. Ceri -- Haffely, Gaffely, Gaffely, Gonward. pgp0Y9GyN1bZ5.pgp Description: PGP signature
Re: svn commit: r214431 - head/bin/rm
On Thu Oct 28 10, Gary Jennejohn wrote: On Thu, 28 Oct 2010 16:22:05 +1100 (EST) Bruce Evans b...@optusnet.com.au wrote: On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. that might be the ultimate solution. also one could use security/srm instead. +1 here. ;) question is: should -P be removed entirely or be made a no op? cheers. alex -- Gary Jennejohn -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Thu, Oct 28, 2010 at 10:25:47AM +, Alexander Best wrote: On Thu Oct 28 10, Gary Jennejohn wrote: On Thu, 28 Oct 2010 16:22:05 +1100 (EST) Bruce Evans b...@optusnet.com.au wrote: On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. that might be the ultimate solution. also one could use security/srm instead. +1 here. ;) question is: should -P be removed entirely or be made a no op? Probably best that it fail. Ceri -- Haffely, Gaffely, Gaffely, Gonward. pgplCIkL78xYi.pgp Description: PGP signature
Re: svn commit: r214431 - head/bin/rm
On Thu Oct 28 10, Ceri Davies wrote: On Thu, Oct 28, 2010 at 10:25:47AM +, Alexander Best wrote: On Thu Oct 28 10, Gary Jennejohn wrote: On Thu, 28 Oct 2010 16:22:05 +1100 (EST) Bruce Evans b...@optusnet.com.au wrote: On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. that might be the ultimate solution. also one could use security/srm instead. +1 here. ;) question is: should -P be removed entirely or be made a no op? Probably best that it fail. how about the following patch? cheers. alex Ceri -- Haffely, Gaffely, Gaffely, Gonward. -- a13x diff --git a/bin/rm/rm.1 b/bin/rm/rm.1 index 11bc77d..569e7db 100644 --- a/bin/rm/rm.1 +++ b/bin/rm/rm.1 @@ -84,22 +84,6 @@ directory is being recursively removed. This is a far less intrusive option than .Fl i yet provides almost the same level of protection against mistakes. -.It Fl P -Overwrite regular files before deleting them. -Files are overwritten three times, first with the byte pattern 0xff, -then 0x00, and then 0xff again, before they are deleted. -Files with multiple links will not be overwritten nor deleted -and a warning will be issued. -If the -.Fl f -option is specified, files with multiple links will also be overwritten -and deleted. -No warning will be issued. -.Pp -Specifying this flag for a read only file will cause -.Nm -to generate an error message and exit. -The file will not be removed or overwritten. .It Fl R Attempt to remove the file hierarchy rooted in each .Ar file @@ -182,12 +166,6 @@ For example: .Pp .Dl rm /home/user/-filename .Dl rm ./-filename -.Pp -When -.Fl P -is specified with -.Fl f -the file will be overwritten and removed even if it has hard links. .Sh COMPATIBILITY The .Nm @@ -203,6 +181,23 @@ Also, historical .Bx implementations prompted on the standard output, not the standard error output. +.Pp +Previous +.Bx +versions of +.Nm +featured a +.Fl P +option which instructed +.Nm +to overwrite regular files three times with the byte pattern +0xff, then 0x00, and then 0xff again, before they were deleted. +Support for the +.Fl P +flag was removed in +.Fx 9.0 , +because overwriting specific blocks cannot be assured +on certain filesystems and media. .Sh SEE ALSO .Xr chflags 1 , .Xr rmdir 1 , @@ -226,11 +221,3 @@ A .Nm command appeared in .At v1 . -.Sh BUGS -The -.Fl P -option assumes that the underlying file system updates existing blocks -in-place and does not store new data in a new location. -This is true for UFS but not for ZFS, which is using a Copy-On-Write strategy. -In addition, only regular files are overwritten, other types of files -are not. diff --git a/bin/rm/rm.c b/bin/rm/rm.c index 653833a..1aff60f 100644 --- a/bin/rm/rm.c +++
Re: svn commit: r214431 - head/bin/rm
On Thu, 28 Oct 2010, Gary Jennejohn wrote: The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. I'm not sure I entirely agree with this conclusion: there are times when -P can be quite effective at removing data from a system, and not having it available when it could work might also be problematic. Part of the problem here is that we have successfully abstracted away a lot of information going up the storage stack. One possibly solution to this problem is to add ways for that information to propagate better -- in the same way that the end-to-end argument in network design is moderated by the reality that performance can be dramatically improved if information gathering can be done with an awareness of lower layers (for example, PMTU discovery), even if decisions about how to use that information are at the endpoints only. For example, you could imagine a pathconf() call that asks the file system if overwriting a file is likely to result in the data going away -- UFS could answer yes unless it's snapshotting the file system, and ZFS could simply answer no. In FreeBSD 9, we're going to have two common file system configurations: ZFS and UFS+SUJ; the latter will do overwriting of files just fine on most current media. Robert ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Thu Oct 28 10, Robert Watson wrote: On Thu, 28 Oct 2010, Gary Jennejohn wrote: The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. All this discussion leads me to the conclusion that we should just remove the -P functionality and add a remark to the man page that that was done because it isn't guaranteed to work on all file systems. Why give users a false sense of security? If they're concerned about data security then they should use geli or something similar. I'm not sure I entirely agree with this conclusion: there are times when -P can be quite effective at removing data from a system, and not having it available when it could work might also be problematic. Part of the problem here is that we have successfully abstracted away a lot of information going up the storage stack. One possibly solution to this problem is to add ways for that information to propagate better -- in the same way that the end-to-end argument in network design is moderated by the reality that performance can be dramatically improved if information gathering can be done with an awareness of lower layers (for example, PMTU discovery), even if decisions about how to use that information are at the endpoints only. For example, you could imagine a pathconf() call that asks the file system if overwriting a file is likely to result in the data going away -- UFS could answer yes unless it's snapshotting the file system, and ZFS could simply answer no. In FreeBSD 9, we're going to have two common file system configurations: ZFS and UFS+SUJ; the latter will do overwriting of files just fine on most current media. i agree. however from this discussion it's become quite obvious that securely deleting any data on a certain media is a very complex topic. when the -P option got implemented into rm, securely deleting files was quite an easy and intuitive task. this has clearly changed. so the question arises, if securely removing data (maybe also directory entries) shouldn't be handled by a specially designed utility? the pathconf() idea looks very primising. should the return values (yes or no) simply be hardcoded for every fs? or should they be returned by a routine which actually veryfies on the fly if overwriting files is possible? alex ps: please keep in mind that an import of fusefs might happen in the next few months, so the fs situation on freebsd might change. Robert -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Thu, 28 Oct 2010, Alexander Best wrote: For example, you could imagine a pathconf() call that asks the file system if overwriting a file is likely to result in the data going away -- UFS could answer yes unless it's snapshotting the file system, and ZFS could simply answer no. In FreeBSD 9, we're going to have two common file system configurations: ZFS and UFS+SUJ; the latter will do overwriting of files just fine on most current media. i agree. however from this discussion it's become quite obvious that securely deleting any data on a certain media is a very complex topic. when the -P option got implemented into rm, securely deleting files was quite an easy and intuitive task. this has clearly changed. so the question arises, if securely removing data (maybe also directory entries) shouldn't be handled by a specially designed utility? the pathconf() idea looks very primising. should the return values (yes or no) simply be hardcoded for every fs? or should they be returned by a routine which actually veryfies on the fly if overwriting files is possible? Deleting data wasn't simple when rm -P was implemented -- in fact, if I recall, there was significant information available even then that overwrite patterns didn't fully remove file system data from magnetic media in such a way that a qualified attacker couldn't recover it. Bad block relocation in software and hardware is also relevant, and wear-levelling has been used in certain storage devices for quite a while now as well. Today the story has gotten even more complicated: file system snapshot features, copy-on-write file systems, block device-level snapshot features, widespread use of wear-levelling, full data journalling, etc. But that doesn't mean that rm -P still isn't useful :-). I see the point of the new pathconf() being to indicate whether trying to overwrite file data is fundamentally useless. That information could be used in one of two ways: (1) to warn the user that the requested overwrite is unlikely to have helped and (2) to cause the removal to fail if it couldn't possibly have worked. I find (1) more appealing but can see arguments for (2); possibly both should be supported in some or another form. Since there's a whole spectrum of confidence possible in whether an overwrite succeeded, any mechanism like the one I've suggested is necessarily going to involve fairly arbitrary heuristics. In a first generation implementation, I suggest that rm -P should generate a warning only in the following case: - An attempt to overwrite a file on a file system actively performing copy-on-write that would affect the file, such as ZFS, or UFS when there's an active snapshot. After some further cogitation, and if people remain interested, the following further case seems reasonable as well: - An attempt to overwrite a file on a file system is mounted from a geom stack in which at least one component in the stack is performing copy-on-write that would affect the file. The warning should read something along the lines of: rm: /foo/bar: file system may not support data overwrite (-P Or similar. However, rm should still perform the overwrite. To be honest, I'm also fine with there just being a nice caveat in the rm(1) man page. :-) Robert ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed Oct 27 10, Dag-Erling Smorgrav wrote: Author: des Date: Wed Oct 27 18:48:23 2010 New Revision: 214431 URL: http://svn.freebsd.org/changeset/base/214431 Log: Language cleanup. Modified: head/bin/rm/rm.1 Modified: head/bin/rm/rm.1 == --- head/bin/rm/rm.1 Wed Oct 27 18:46:56 2010(r214430) +++ head/bin/rm/rm.1 Wed Oct 27 18:48:23 2010(r214431) @@ -231,6 +231,6 @@ The .Fl P option assumes that the underlying file system updates existing blocks in-place and does not store new data in a new location. -This is true for UFS but not for ZFS, which is using a Copy-On-Write strategy. -In addition, only regular files are overwritten, other types of files -are not. +This is true for UFS, but not for ZFS or other file systems which use +copy-on-write semantics. +In addition, only regular files are overwritten. are in fact COW fs the only exception where the -P flag won't work? before r213582 LFS was mentioned here and that the block size must be fixed. also the comment in rm.c says that -P won't work for any logging file systems. i'm not a fs expert, but i think mentioning that -P won't work for COW fs isn't enough. -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On 10/27/10 14:26, Alexander Best wrote: are in fact COW fs the only exception where the -P flag won't work? before r213582 LFS was mentioned here and that the block size must be fixed. also the comment in rm.c says that -P won't work for any logging file systems. i'm not a fs expert, but i think mentioning that -P won't work for COW fs isn't enough. What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. hth, Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed Oct 27 10, Doug Barton wrote: On 10/27/10 14:26, Alexander Best wrote: are in fact COW fs the only exception where the -P flag won't work? before r213582 LFS was mentioned here and that the block size must be fixed. also the comment in rm.c says that -P won't work for any logging file systems. i'm not a fs expert, but i think mentioning that -P won't work for COW fs isn't enough. What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. i don't think that's a good approach, because then the rm(1) has to be changed everytime freebsd gets a new fs which works with the -P option. i think it's better to list which fs semantics DON'T work. so if freebsd gets a new fs, users simply have to know which semantics the new fs is based on and can decide for themselves whether the -P switch will work or not. so far the -P option doesn't seem to work for: - COW fs and/or - fs with a variable block size and/or - fs which do journaling please correct me if i got anything wrong. so i think having such a list in the rm(1) manual would be very nice (maybe improving the comment in rm.c too). cheers. alex hth, Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed, Oct 27, 2010 at 14:48, Alexander Best arun...@freebsd.org wrote: On Wed Oct 27 10, Doug Barton wrote: What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. i don't think that's a good approach, because then the rm(1) has to be changed everytime freebsd gets a new fs which works with the -P option. i think it's better to list which fs semantics DON'T work. so if freebsd gets a new fs, users simply have to know which semantics the new fs is based on and can decide for themselves whether the -P switch will work or not. so far the -P option doesn't seem to work for: - COW fs and/or - fs with a variable block size and/or - fs which do journaling I really don't want to ask the average user to know whether their filesystem is in-place block-rewriting or not. That's just silly. In this case Doug is right; I don't think FreeBSD gets new file systems as often as you think that it would be a big burden. Having a general description of the types of filesystem it can work on might be useful, but a list seems more useful still. Listing the types it can't work on is backwards because that requires a user to understand the dichotomy as well as knowing what kind of filesystem they don't have / do have. And for them to never get it backwards. At least mount(8) will tell you what filesystem you are using; there's no tool to tell you the properties of your filesystem, and good luck easily-mining an answer to the question of whether your filesystem fits into that category from a manpage without introducing substantial confusion. Maybe there should be substantial confusion around this feature, though, since that's what it seems to be there for. Juli. ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 10/27/10 14:48, Alexander Best wrote: On Wed Oct 27 10, Doug Barton wrote: On 10/27/10 14:26, Alexander Best wrote: are in fact COW fs the only exception where the -P flag won't work? before r213582 LFS was mentioned here and that the block size must be fixed. also the comment in rm.c says that -P won't work for any logging file systems. i'm not a fs expert, but i think mentioning that -P won't work for COW fs isn't enough. What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. i don't think that's a good approach, because then the rm(1) has to be changed everytime freebsd gets a new fs which works with the -P option. i think it's better to list which fs semantics DON'T work. so if freebsd gets a new fs, users simply have to know which semantics the new fs is based on and can decide for themselves whether the -P switch will work or not. so far the -P option doesn't seem to work for: - COW fs and/or - fs with a variable block size and/or - fs which do journaling please correct me if i got anything wrong. so i think having such a list in the rm(1) manual would be very nice (maybe improving the comment in rm.c too). I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Cheers, - -- Xin LI delp...@delphij.nethttp://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (FreeBSD) iQEcBAEBCAAGBQJMyKicAAoJEATO+BI/yjfBElYIAMP70g1a+YheuKD14NXugVTU sG4KEWAjRSZCe808f46AXU+wJePnRFkYVKD+A+6aH63y/r2V0e3CVMUYZZXr4l/d HJRnZjJK9e/YJv8pcCpq7PgnmPzMa4m4BQNYVJoNGbPd75V27wMi3hgBzzPrJxWL aBuB31hpU32PcpvzQgBPLiNzjEuLRq5be42HjgTPT1qGwSMEQcLgXOfG9l6TS27s I5n5KPU7dEFt0Z+3ljQM+F3Fk2wmy/EOAeRcZL89xvFZIAYmtVrL3UcniHALPRSn CAbGrNpCbvh2RZvJX1Cwu3H+PVIlIcl2uG/aiOEC7m/tA29LfPPXG0IzUN9qVLc= =LQen -END PGP SIGNATURE- ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed Oct 27 10, Juli Mallett wrote: On Wed, Oct 27, 2010 at 14:48, Alexander Best arun...@freebsd.org wrote: On Wed Oct 27 10, Doug Barton wrote: What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. i don't think that's a good approach, because then the rm(1) has to be changed everytime freebsd gets a new fs which works with the -P option. i think it's better to list which fs semantics DON'T work. so if freebsd gets a new fs, users simply have to know which semantics the new fs is based on and can decide for themselves whether the -P switch will work or not. so far the -P option doesn't seem to work for: - COW fs and/or - fs with a variable block size and/or - fs which do journaling I really don't want to ask the average user to know whether their filesystem is in-place block-rewriting or not. That's just silly. In this case Doug is right; I don't think FreeBSD gets new file systems as often as you think that it would be a big burden. Having a general description of the types of filesystem it can work on might be useful, but a list seems more useful still. Listing the types it can't work on is backwards because that requires a user to understand the dichotomy as well as knowing what kind of filesystem they don't have / do have. And for them to never get it backwards. At least mount(8) will tell you what filesystem you are using; there's no tool to tell you the properties of your filesystem, and good luck easily-mining an answer to the question of whether your filesystem fits into that category from a manpage without introducing substantial confusion. Maybe there should be substantial confusion around this feature, though, since that's what it seems to be there for. how about fusefs? i think there's an ongoing disussion about importing it into HEAD on arch@ or f...@. won't this bring in support for a number of new filesystems which then all have to be documented in the rm(1) manual? but if in fact all working fs should get mentioned...what are they? - UFS1/2 - ext2fs - FAT12/16/32 ...any more? what about memory backed fs like tmpfs? cheers. alex Juli. -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed Oct 27 10, Xin LI wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 10/27/10 14:48, Alexander Best wrote: On Wed Oct 27 10, Doug Barton wrote: On 10/27/10 14:26, Alexander Best wrote: are in fact COW fs the only exception where the -P flag won't work? before r213582 LFS was mentioned here and that the block size must be fixed. also the comment in rm.c says that -P won't work for any logging file systems. i'm not a fs expert, but i think mentioning that -P won't work for COW fs isn't enough. What may be a better approach is to confirm the fs' that DO work, list them, and then add something to the effect of, This feature is unlikely to work on other file systems. i don't think that's a good approach, because then the rm(1) has to be changed everytime freebsd gets a new fs which works with the -P option. i think it's better to list which fs semantics DON'T work. so if freebsd gets a new fs, users simply have to know which semantics the new fs is based on and can decide for themselves whether the -P switch will work or not. so far the -P option doesn't seem to work for: - COW fs and/or - fs with a variable block size and/or - fs which do journaling please correct me if i got anything wrong. so i think having such a list in the rm(1) manual would be very nice (maybe improving the comment in rm.c too). I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === personally i like this approach a lot better than listing all working fs, because that list would be quite long since e.g. for UFS2 (as you mentioned) one cannot asume that it will support the -P option at all times. cheers. alex Cheers, - -- Xin LI delp...@delphij.net http://www.delphij.net/ FreeBSD - The Power to Serve!Live free or die -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.16 (FreeBSD) iQEcBAEBCAAGBQJMyKicAAoJEATO+BI/yjfBElYIAMP70g1a+YheuKD14NXugVTU sG4KEWAjRSZCe808f46AXU+wJePnRFkYVKD+A+6aH63y/r2V0e3CVMUYZZXr4l/d HJRnZjJK9e/YJv8pcCpq7PgnmPzMa4m4BQNYVJoNGbPd75V27wMi3hgBzzPrJxWL aBuB31hpU32PcpvzQgBPLiNzjEuLRq5be42HjgTPT1qGwSMEQcLgXOfG9l6TS27s I5n5KPU7dEFt0Z+3ljQM+F3Fk2wmy/EOAeRcZL89xvFZIAYmtVrL3UcniHALPRSn CAbGrNpCbvh2RZvJX1Cwu3H+PVIlIcl2uG/aiOEC7m/tA29LfPPXG0IzUN9qVLc= =LQen -END PGP SIGNATURE- -- a13x ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed, Oct 27, 2010 at 03:33:01PM -0700, Xin LI wrote: Maybe something like: BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. This sounds very nice and FreeBSD-ish. ./danfe ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r214431 - head/bin/rm
On Wed, 27 Oct 2010, Xin LI wrote: I think what really defeats -P is the fact that the file system or underlying data storage would not overwrite data on a file at sync(). COW is of course one of the case, journaling MAY defeat -P but is not guaranteed. FS with variable block size - I believe this really depends on the implementation. If I understood the code correctly, UFS, UFS+SU, UFS+SUJ, msdosfs and ext2fs supports rm -P as long as they are not being put on gjournal'ed disk, ZFS zvol, etc., and no snapshot is being used. And that the underlying data storage us dumb. Any flash drive now tries to minimise writes. It wouldn't take much buffering to defeat the 0xff, 0,0xff pattern. Wear leveling should result in different physical blocks being written each time if the writes get to the lowest level of storage. And that block reallocation (done by ffs1 and ffs2) doesn't choose different blocks. It seems to be hard for me to conclude all cases in short, plain English but I'm all for improvements to the manual page to describe that in an elegant and precise manner. Maybe something like: === BUGS The -P option assumes that the underlying storage overwrites file block when data is written on existing offset. Several factors including the file system and its backing store could defeat the assumption, this includes, but is not limited to file systems that uses Copy-On-Write strategy (e.g. ZFS or UFS when snapshot is being used), or backing datastore that does journaling, etc. In addition, only regular files are overwritten, other types of files are not. === Summary: it is very hard to tell whether -P works, even when you think you know what all the subsystems are doing. Bruce ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
