Re: cp command - problem with sparse
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to James Youngman on 2/1/2005 3:17 AM: Unix systems automatically generate sparse files when programs seek forwards on their output file. There is no need to have a sparse attribute. This is what coreutils' cp does. Right now, the tests/du/8gb test uses dd to try to create a sparse file; and strace'ing that on cygwin shows that it uses just lseek() followed by ftruncate() (no intervening write()). But the code in src/copy.c goes to great lengths to write() before calling ftruncate(), with the comment that the kernel would truncate the file at the end of the last write operation. Which is it? Is copy doing more work than it should, or should dd also be doing a write before truncate? POSIX does say that ftruncate shall increase the size of the file in XSI systems, but allows it return an error and keep the size unchanged on non-XSI systems. I ask, because at the moment, cygwin's implementation only makes a sparse file on write() after lseek(), although the developers are considering making ftruncate() after lseek() also create a sparse file. - -- Life is short - so eat dessert first! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCAaOB84KuGfSFAYARAoXxAJ9nbKLwI8fGcdAJ9vVggGwehujbFwCgnIuv kmBzebHsRNu4iHb7q1vPGE0= =Wpzz -END PGP SIGNATURE- ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
Eric Blake [EMAIL PROTECTED] wrote: According to James Youngman on 2/1/2005 3:17 AM: Unix systems automatically generate sparse files when programs seek forwards on their output file. There is no need to have a sparse attribute. This is what coreutils' cp does. Right now, the tests/du/8gb test uses dd to try to create a sparse file; and strace'ing that on cygwin shows that it uses just lseek() followed by ftruncate() (no intervening write()). But the code in src/copy.c goes to great lengths to write() before calling ftruncate(), with the comment that the kernel would truncate the file at the end of the last write operation. Which is it? Is copy doing more work than it should, or should dd also be doing a write before truncate? POSIX does say that ftruncate shall increase the size of the file in XSI systems, but allows it return an error and keep the size unchanged on non-XSI systems. Well, to make that test case stop skipping on cygwin, we could apply the band-aid of simply using cp --sparse=always to copy the dd-created file. I'm reluctant to modify dd solely to accommodate Cygwin when it may be jumping on the XSI bandwagon soon anyhow, but if the change is not too ugly, I'll consider it. I ask, because at the moment, cygwin's implementation only makes a sparse file on write() after lseek(), although the developers are considering making ftruncate() after lseek() also create a sparse file. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
Eric Blake [EMAIL PROTECTED] wrote: ... According to the cygwin mailing list, http://sources.redhat.com/ml/cygwin/2005-02/msg00013.html, cygwin already supports sparse files when you do lseek beyond EOF during writes. The trick, however, is that NTFS on Windows XP does not create a hole until 128k. Therefore, this patch is needed in the testsuite to turn a SKIP into a PASS on cygwin: 2005-02-01 Eric Blake [EMAIL PROTECTED] (tiny change) * tests/du/8gb: Detect sparse files on NTFS under cygwin. Thanks. I've applied that. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
Eric Blake [EMAIL PROTECTED] writes: Index: tests/du/8gb === RCS file: /cvsroot/coreutils/coreutils/tests/du/8gb,v retrieving revision 1.6 diff -u -p -r1.6 8gb --- tests/du/8gb 3 May 2003 14:24:37 - 1.6 +++ tests/du/8gb 2 Feb 2005 03:19:31 - @@ -26,7 +26,8 @@ fi # If this file system doesn't support sparse files, # don't try to create a file that'd end up consuming 8GB. # This happens on Darwin6.5 with a file system of type `hfs'. -dd bs=1 seek=64K of=t /dev/null 2 /dev/null +# NTFS requires 128K before a hole appears in a sparse file. +dd bs=1 seek=128K of=t /dev/null 2 /dev/null set x `du -sk t` if test $2 = 64; then ^^ Doesn't that need to be changed to 128 as well? Andreas. -- Andreas Schwab, SuSE Labs, [EMAIL PROTECTED] SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
Andreas Schwab [EMAIL PROTECTED] wrote: Eric Blake [EMAIL PROTECTED] writes: Index: tests/du/8gb === RCS file: /cvsroot/coreutils/coreutils/tests/du/8gb,v retrieving revision 1.6 diff -u -p -r1.6 8gb --- tests/du/8gb 3 May 2003 14:24:37 - 1.6 +++ tests/du/8gb 2 Feb 2005 03:19:31 - @@ -26,7 +26,8 @@ fi # If this file system doesn't support sparse files, # don't try to create a file that'd end up consuming 8GB. # This happens on Darwin6.5 with a file system of type `hfs'. -dd bs=1 seek=64K of=t /dev/null 2 /dev/null +# NTFS requires 128K before a hole appears in a sparse file. +dd bs=1 seek=128K of=t /dev/null 2 /dev/null set x `du -sk t` if test $2 = 64; then ^^ Doesn't that need to be changed to 128 as well? Yes. Thanks. Fixed. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
Jim Meyering [EMAIL PROTECTED] writes: Is this just theoretical, or do you know of a file system type that'd cause trouble (that doesn't support sparse files, yet for which meta data can take up extra space)? It's just a guess, I don't know of any example. It might well be that going from 64K to 128K on hfs could result in enough meta data to be allocated so that the total size rounds up to 129 (assuming that such meta data is actually accounted for in the stat data). Andreas. -- Andreas Schwab, SuSE Labs, [EMAIL PROTECTED] SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
On Mon, Jan 31, 2005 at 05:30:22PM +0100, RE wrote: cp --sparse=always c:\test.cfg c:\test2.cfg Everything works fine with that cp command, except the fact that I do not get a sparse file. Even when I copy a sparse file, the sparse attribute is no longer present in the copy an the occupied space on my HD is the same as with the original file. What am I doing wrong? I tried already different PCs with NTFS (OS = Win2k SP4) Unix systems automatically generate sparse files when programs seek forwards on their output file. There is no need to have a sparse attribute. This is what coreutils' cp does. Windows and NTFS don't work in this way. Under NTFS, there is, as you say, a sparse attribute which must be set. GNU coreutils runs on Windows under Cygwin and am not sure if Cygwin exposes any form of API which might allow cp to set the sparse attribute. It's certainly a lot more complex to do this under Windows. See http://www.ntfs.com/ntfs-sparse.htm and http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/fsctl_set_zero_data.asp ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: cp command - problem with sparse
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 According to James Youngman on 2/1/2005 3:17 AM: Unix systems automatically generate sparse files when programs seek forwards on their output file. There is no need to have a sparse attribute. This is what coreutils' cp does. Windows and NTFS don't work in this way. Under NTFS, there is, as you say, a sparse attribute which must be set. GNU coreutils runs on Windows under Cygwin and am not sure if Cygwin exposes any form of API which might allow cp to set the sparse attribute. It's certainly a lot more complex to do this under Windows. According to the cygwin mailing list, http://sources.redhat.com/ml/cygwin/2005-02/msg00013.html, cygwin already supports sparse files when you do lseek beyond EOF during writes. The trick, however, is that NTFS on Windows XP does not create a hole until 128k. Therefore, this patch is needed in the testsuite to turn a SKIP into a PASS on cygwin: 2005-02-01 Eric Blake [EMAIL PROTECTED] (tiny change) * tests/du/8gb: Detect sparse files on NTFS under cygwin. - -- Life is short - so eat dessert first! Eric Blake [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (Cygwin) Comment: Public key at home.comcast.net/~ericblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCAEZA84KuGfSFAYARAiFSAKDF7lb6zJq6ADLsFyHPrgkQ30tDaACcDT7P 8lGA+YY7czPjlGfVQYRANaQ= =E+76 -END PGP SIGNATURE- Index: tests/du/8gb === RCS file: /cvsroot/coreutils/coreutils/tests/du/8gb,v retrieving revision 1.6 diff -u -p -r1.6 8gb --- tests/du/8gb3 May 2003 14:24:37 - 1.6 +++ tests/du/8gb2 Feb 2005 03:19:31 - @@ -26,7 +26,8 @@ fi # If this file system doesn't support sparse files, # don't try to create a file that'd end up consuming 8GB. # This happens on Darwin6.5 with a file system of type `hfs'. -dd bs=1 seek=64K of=t /dev/null 2 /dev/null +# NTFS requires 128K before a hole appears in a sparse file. +dd bs=1 seek=128K of=t /dev/null 2 /dev/null set x `du -sk t` if test $2 = 64; then echo $0: skipping this test, since this file system doesn't support 12 ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
