bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread George Valkov
> On 2023-03-21, at 8:21 PM, Pádraig Brady wrote: > > On 21/03/2023 18:08, George Valkov wrote: >>> On 2023-03-21, at 7:32 PM, Paul Eggert wrote: >>> >>> On 3/21/23 08:27, George Valkov wrote: However since the old m4-1.4.19 in under tools/m4, coreutils builds with the legacy

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread Pádraig Brady
On 21/03/2023 18:08, George Valkov wrote: On 2023-03-21, at 7:32 PM, Paul Eggert wrote: On 3/21/23 08:27, George Valkov wrote: However since the old m4-1.4.19 in under tools/m4, coreutils builds with the legacy version: If you're building from that tarball, coreutils doesn't need m4. I

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread George Valkov
> On 2023-03-21, at 7:32 PM, Paul Eggert wrote: > > On 3/21/23 08:27, George Valkov wrote: >> However since the old m4-1.4.19 in under tools/m4, coreutils builds with the >> legacy version: > > If you're building from that tarball, coreutils doesn't need m4. I just now > double-checked

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread Paul Eggert
On 3/21/23 08:27, George Valkov wrote: However since the old m4-1.4.19 in under tools/m4, coreutils builds with the legacy version: If you're building from that tarball, coreutils doesn't need m4. I just now double-checked that, in an environment that didn't have m4. "./configure && make

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread George Valkov
> On 2023-03-21, at 4:49 PM, Paul Eggert wrote: > Easiest might be to build from the latest release: > https://ftp.gnu.org/pub/gnu/coreutils/coreutils-9.2.tar.xz > If you do that, you shouldn't need to run m4, and all the gnulib files will > be there already. Thanks, Paul! Yes, this is what

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread Paul Eggert
On 3/21/23 06:55, George Valkov wrote: Is there a source download link or should be clone from the Savanna git repo? Easiest might be to build from the latest release: https://ftp.gnu.org/pub/gnu/coreutils/coreutils-9.2.tar.xz If you do that, you shouldn't need to run m4, and all the gnulib

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-21 Thread George Valkov
> On 2023-03-11, at 4:28 PM, Pádraig Brady wrote: > > On 11/03/2023 07:29, George Valkov wrote: >> Hi Paul! >> Here are the latest test results on macOS 12.6.3: >> threshold.sh: skipped test: block size of a directory is smaller than 4 bytes >> SKIP: tests/du/threshold.sh >> It was also

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-11 Thread Pádraig Brady
On 11/03/2023 07:29, George Valkov wrote: Hi Paul! Here are the latest test results on macOS 12.6.3: threshold.sh: skipped test: block size of a directory is smaller than 4 bytes SKIP: tests/du/threshold.sh It was also skipped on macOS before the patch, so no change.

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-10 Thread George Valkov
> On 2023-03-07, at 11:53 PM, Paul Eggert wrote: > > On 3/6/23 16:31, Pádraig Brady wrote: > >> syntax check now looks good thanks. > > You're welcome. I'm getting a failure on tests/du/threshold.sh, though, on > Fedora 37 x86-64. Are you seeing that? See attached (compressed) log. > > >>

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-07 Thread Pádraig Brady
On 07/03/2023 21:53, Paul Eggert wrote: On 3/6/23 16:31, Pádraig Brady wrote: syntax check now looks good thanks. You're welcome. I'm getting a failure on tests/du/threshold.sh, though, on Fedora 37 x86-64. Are you seeing that? See attached (compressed) log. I see this as a more

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-07 Thread Paul Eggert
On 3/6/23 16:31, Pádraig Brady wrote: syntax check now looks good thanks. You're welcome. I'm getting a failure on tests/du/threshold.sh, though, on Fedora 37 x86-64. Are you seeing that? See attached (compressed) log. I see this as a more fundamental operational thing than a limit

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-06 Thread Pádraig Brady
On 06/03/2023 23:48, Paul Eggert wrote: On 3/6/23 03:13, Pádraig Brady wrote: On 06/03/2023 07:37, Paul Eggert wrote: The lseek /dev/null issue was only in your undef SEEK_DATA test patch, and already addressed in your final gnulib patch in the area, as discussed at:

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-06 Thread Paul Eggert
On 3/6/23 03:13, Pádraig Brady wrote: On 06/03/2023 07:37, Paul Eggert wrote: The lseek /dev/null issue was only in your undef SEEK_DATA test patch, and already addressed in your final gnulib patch in the area, as discussed at:

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-06 Thread George Valkov
> On 2023-03-06, at 9:37 AM, Paul Eggert wrote: > > I recall reading somewhere in this thread that a 'split' test was failing on > macOS because it doesn't let you lseek on /dev/null. I fixed that problem > here: > >

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-06 Thread Pádraig Brady
On 06/03/2023 07:37, Paul Eggert wrote: I recall reading somewhere in this thread that a 'split' test was failing on macOS because it doesn't let you lseek on /dev/null. I fixed that problem here:

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-03-05 Thread Paul Eggert
I recall reading somewhere in this thread that a 'split' test was failing on macOS because it doesn't let you lseek on /dev/null. I fixed that problem here: https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=aa266f1b3dc4e12acdc46cc0f562adc03c2c0b8f and fixed some other 'split' issues

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-25 Thread Pádraig Brady
On 24/02/2023 23:23, George Valkov wrote: Are we in a state where I can update OpenWRT to the latest coreutils master or perhaps it would be better to wait until you fix the tests and coreutils-9.2 is released? You mentioned the release is coming soon? Probably best to wait for the 9.2

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread George Valkov
> On 2023-02-25, at 12:47 AM, Pádraig Brady wrote: > > On 24/02/2023 22:06, George Valkov wrote: >> If I revert a0803c4bad6f8e11bb05effcc42ef433f4fc3f9b, the requirement to >> press enter after PASS: tests/rm/isatty.sh is fixed. > > Ah very useful info. > I'll test this on my macOS 13.2

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread Pádraig Brady
On 24/02/2023 22:06, George Valkov wrote: If I revert a0803c4bad6f8e11bb05effcc42ef433f4fc3f9b, the requirement to press enter after PASS: tests/rm/isatty.sh is fixed. Ah very useful info. I'll test this on my macOS 13.2 system. Sometimes it might randomly hang: gdb after PASS:

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread George Valkov
If I revert a0803c4bad6f8e11bb05effcc42ef433f4fc3f9b, the requirement to press enter after PASS: tests/rm/isatty.sh is fixed. Sometimes it might randomly hang: gdb after PASS: tests/rm/r-4.sh ^Cmake[1]: *** Deleting file 'tests/rm/r-root.log' ^C^C^C ps -A |grep gdb 55051 ttys0200:00.09 gdb

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread George Valkov
> On 2023-02-24, at 5:43 PM, Pádraig Brady wrote: > > On 24/02/2023 14:33, George Valkov wrote: >>> On 2023-02-24, at 12:23 AM, Paul Eggert wrote: >>> >>> On 2/20/23 13:14, Pádraig Brady wrote: Since https://github.com/coreutils/gnulib/commit/4db8db34 gnulib has been

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread Pádraig Brady
On 24/02/2023 14:33, George Valkov wrote: On 2023-02-24, at 12:23 AM, Paul Eggert wrote: On 2/20/23 13:14, Pádraig Brady wrote: Since https://github.com/coreutils/gnulib/commit/4db8db34 gnulib has been unconditionally replacing lseek() on macos. Now with SEEK_DATA undefined that replaced

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-24 Thread George Valkov
> On 2023-02-24, at 12:23 AM, Paul Eggert wrote: > > On 2/20/23 13:14, Pádraig Brady wrote: >> Since https://github.com/coreutils/gnulib/commit/4db8db34 >> gnulib has been unconditionally replacing lseek() on macos. >> Now with SEEK_DATA undefined that replaced lseek() >> will run the code

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-23 Thread Pádraig Brady
On 23/02/2023 22:23, Paul Eggert wrote: On 2/20/23 13:14, Pádraig Brady wrote: Since https://github.com/coreutils/gnulib/commit/4db8db34 gnulib has been unconditionally replacing lseek() on macos. Now with SEEK_DATA undefined that replaced lseek() will run the code intended for BeOS. So either

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-23 Thread Paul Eggert
On 2/20/23 13:14, Pádraig Brady wrote: Since https://github.com/coreutils/gnulib/commit/4db8db34 gnulib has been unconditionally replacing lseek() on macos. Now with SEEK_DATA undefined that replaced lseek() will run the code intended for BeOS. So either the lseek.m4 or lseek.c needs adjusting

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-22 Thread George Valkov
I received an update from Apple > We reproduced the issue and are investigating. > Our engineers are investigating the root cause of the issue you reported. If > we need more information from you, we’ll add a comment and send you an email. Georgi Valkov httpstorm.com nano RTOS

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread George Valkov
> On 2023-02-20, at 11:14 PM, Pádraig Brady wrote: > > On 20/02/2023 19:35, George Valkov wrote: >>> On 2023-02-20, at 7:49 PM, Pádraig Brady wrote: >>> >>> On 20/02/2023 15:02, George Valkov wrote: Hi Paul, the following tests fail after your patch: FAIL: tests/split/filter.sh

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread Pádraig Brady
On 20/02/2023 19:35, George Valkov wrote: On 2023-02-20, at 7:49 PM, Pádraig Brady wrote: On 20/02/2023 15:02, George Valkov wrote: Hi Paul, the following tests fail after your patch: FAIL: tests/split/filter.sh FAIL: tests/split/b-chunk.sh FAIL: tests/split/l-chunk.sh These look

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread George Valkov
> On 2023-02-20, at 7:49 PM, Pádraig Brady wrote: > > On 20/02/2023 15:02, George Valkov wrote: >> Hi Paul, the following tests fail after your patch: >> FAIL: tests/split/filter.sh >> FAIL: tests/split/b-chunk.sh >> FAIL: tests/split/l-chunk.sh > > These look unrelated and may be due to some

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread Pádraig Brady
On 20/02/2023 15:02, George Valkov wrote: Hi Paul, the following tests fail after your patch: FAIL: tests/split/filter.sh FAIL: tests/split/b-chunk.sh FAIL: tests/split/l-chunk.sh These look unrelated and may be due to some other change in your environment. Specifically lseek() is failing on

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread George Valkov
Hi Paul, the following tests fail after your patch: FAIL: tests/split/filter.sh FAIL: tests/split/b-chunk.sh FAIL: tests/split/l-chunk.sh FAIL: tests/cp/sparse-perf.sh Before patch

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-20 Thread George Valkov
> On 2023-02-19, at 8:08 AM, Paul Eggert wrote: > > George, given what you've written I suppose we should give up the idea of > copying sparse files efficiently on macOS (and on FreeBSD 13.0-RELEASE, as it > has a similar bug with SEEK_HOLE and SEEK_DATA), in cases where fclonefileat > does

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-19 Thread Pádraig Brady
On 19/02/2023 06:08, Paul Eggert wrote: George, given what you've written I suppose we should give up the idea of copying sparse files efficiently on macOS (and on FreeBSD 13.0-RELEASE, as it has a similar bug with SEEK_HOLE and SEEK_DATA), in cases where fclonefileat does not work. Please try

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-18 Thread Paul Eggert
George, given what you've written I suppose we should give up the idea of copying sparse files efficiently on macOS (and on FreeBSD 13.0-RELEASE, as it has a similar bug with SEEK_HOLE and SEEK_DATA), in cases where fclonefileat does not work. Please try the attached Gnulib patch; it uses

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-16 Thread Paul Eggert
On 2/16/23 13:32, Pádraig Brady wrote: +  /* any desired permissions that weren't cloned.  */    extra_permissions = desired_mode & ~cloned_mode; That comment is a tad redundant, but I did add some commentary along those lines and installed the patch into

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-16 Thread Pádraig Brady
On 15/02/2023 10:56, Paul Eggert wrote: Attached is an updated proposal for the fclonefileat patch. CVE-2021-30995 confirmed my suspicion that coreutils 9.1 and the current bleeding-edge coreutils on Savannah both have an exploitable security bug on macOS. Although I hope this patch fixes the

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-16 Thread George Valkov
Gentlemen, I found a solution, as well as the reason why lseek(SEEK_DATA) returns invalid data. I found a way to craft a file that gets corrupted when we try to make a sparse copy. It’s based on mmap: 1. Open src and dst. 2. Obtain the size of src, and truncate dst to the same size. 3. Use mmap

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread George Valkov
> On 2023-02-15, at 9:48 PM, Paul Eggert wrote: > > On 2023-02-15 07:26, George Valkov wrote: >> I tested your patch: both overwrite existing and clone to new produce a >> working copy. Here are the test results: >>

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread George Valkov
> On 2023-02-15, at 9:26 PM, Paul Eggert wrote: > > On 2023-02-15 06:05, George Valkov wrote: >> gcc d.c && ./a.out >> src 3 dst 4 >> c 14053376 p 20344832 h 20344832 d 6291456 >> total bytes copied 14053376 / 27551296 > > Thanks, this is due to a known incompatibility in macOS lseek that

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread Paul Eggert
On 2023-02-15 07:26, George Valkov wrote: I tested your patch: both overwrite existing and clone to new produce a working copy. Here are the test results: https://httpstorm.com/share/.openwrt/test/2023-02-06_coreutils-9.1/test-04-cf80f988eeb97cc3f8c65ae58e735d36f865277b-clone.txt I see some

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread Paul Eggert
On 2023-02-15 06:05, George Valkov wrote: gcc d.c && ./a.out src 3 dst 4 c 14053376 p 20344832 h 20344832 d 6291456 total bytes copied 14053376 / 27551296 Thanks, this is due to a known incompatibility in macOS lseek that coreutils is supposed to work around. See

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread George Valkov
> On 2023-02-15, at 12:56 PM, Paul Eggert wrote: > > The disabling SEEK_HOLE on Apple is OK if a release is imminent. Otherwise > I'd like to get to the bottom of it first. This can be done by having George > use dtrace or (if that's not possible) adding debugging printfs. Hi Paul! Sure,

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread George Valkov
> On 2023-02-14, at 2:12 PM, Pádraig Brady wrote: > > On 14/02/2023 06:12, Paul Eggert wrote: >> On 2023-02-13 09:16, George Valkov wrote: >>> 1. We apply my original patch to disable sparse-copy on macOS. Otherwise >>> since fclonefileat is not used whenever we overwrite a file, corruption

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread Paul Eggert
Attached is an updated proposal for the fclonefileat patch. CVE-2021-30995 confirmed my suspicion that coreutils 9.1 and the current bleeding-edge coreutils on Savannah both have an exploitable security bug on macOS. Although I hope this patch fixes the bug, it could use more pairs of eyes,

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-15 Thread Pádraig Brady
On 14/02/2023 19:02, Paul Eggert wrote: On 2023-02-14 04:12, Pádraig Brady wrote: I suspect it's a similar issue to the one that openzfs had: https://github.com/openzfs/zfs/issues/11900 Yes, quite likely. Given how closed / uncommunicative Apple are in general and specifically for this

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-14 Thread Paul Eggert
On 2023-02-14 04:12, Pádraig Brady wrote: I suspect it's a similar issue to the one that openzfs had: https://github.com/openzfs/zfs/issues/11900 Yes, quite likely. Given how closed / uncommunicative Apple are in general and specifically for this already reported bug, I'm inclined to

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-14 Thread Paul Eggert
On 2023-02-13 09:16, George Valkov wrote: 1. We apply my original patch to disable sparse-copy on macOS. Otherwise since fclonefileat is not used whenever we overwrite a file, corruption will still occur. I'm not entirely sold on this patch, because I still don't understand what's

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-14 Thread George Valkov
> On 2023-02-14, at 8:12 AM, Paul Eggert wrote: > > On 2023-02-13 09:16, George Valkov wrote: >> 1. We apply my original patch to disable sparse-copy on macOS. Otherwise >> since fclonefileat is not used whenever we overwrite a file, corruption will >> still occur. > > I'm not entirely sold

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-14 Thread Pádraig Brady
On 14/02/2023 06:12, Paul Eggert wrote: On 2023-02-13 09:16, George Valkov wrote: 1. We apply my original patch to disable sparse-copy on macOS. Otherwise since fclonefileat is not used whenever we overwrite a file, corruption will still occur. I'm not entirely sold on this patch, because I

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-13 Thread Pádraig Brady
On 13/02/2023 17:16, George Valkov wrote: On 2023-02-13, at 4:05 PM, Pádraig Brady wrote: This amendment seems to make the treatment of umask and setuid consistent between using fclonefileat() and not. diff --git a/src/copy.c b/src/copy.c index 782fab98d..8370f55bd 100644 --- a/src/copy.c

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-13 Thread George Valkov
Test results on macOS 12.6.3 https://httpstorm.com/share/.openwrt/test/2023-02-06_coreutils-9.1/test-02-d374d32ccf12f8cf33c8f823d573498b7c8b27a4-clone.txt Georgi Valkov httpstorm.com nano RTOS

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-13 Thread George Valkov
> On 2023-02-13, at 4:05 PM, Pádraig Brady wrote: > > On 12/02/2023 20:36, Pádraig Brady wrote: >> On 12/02/2023 14:15, Pádraig Brady wrote: >>> On 11/02/2023 21:53, Paul Eggert wrote: On 2023-02-10 17:21, George Valkov wrote: > remember to trace all code paths errors.

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-13 Thread Pádraig Brady
On 12/02/2023 23:37, George Valkov wrote: I can confirm that. CLONE_ACL is 4 when building on macOS 12.6.3. I rand the compiled sample on the recovery environment, which is equivalent to macOS 13.3. The flag is supported, though I did not observe any difference with or without it. The sample

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-13 Thread Pádraig Brady
On 12/02/2023 20:36, Pádraig Brady wrote: On 12/02/2023 14:15, Pádraig Brady wrote: On 11/02/2023 21:53, Paul Eggert wrote: On 2023-02-10 17:21, George Valkov wrote: remember to trace all code paths errors. Yes, I've been doing that. However, I don't use macOS and the two of us are

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-12 Thread George Valkov
> On 2023-02-12, at 10:36 PM, Pádraig Brady wrote: > > On 12/02/2023 14:15, Pádraig Brady wrote: >> On 11/02/2023 21:53, Paul Eggert wrote: >>> On 2023-02-10 17:21, George Valkov wrote: >>> remember to trace all code paths errors. >>> >>> Yes, I've been doing that. However, I don't use

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-12 Thread Pádraig Brady
On 12/02/2023 14:15, Pádraig Brady wrote: On 11/02/2023 21:53, Paul Eggert wrote: On 2023-02-10 17:21, George Valkov wrote: remember to trace all code paths errors. Yes, I've been doing that. However, I don't use macOS and the two of us are debugging remotely where I write the code and you

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-12 Thread Pádraig Brady
On 11/02/2023 21:53, Paul Eggert wrote: On 2023-02-10 17:21, George Valkov wrote: remember to trace all code paths errors. Yes, I've been doing that. However, I don't use macOS and the two of us are debugging remotely where I write the code and you debug it but neither of us know how macOS

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread George Valkov
> On 2023-02-12, at 2:47 AM, Paul Eggert wrote: > > On 2023-02-11 16:38, George Valkov wrote: >> This might help: >> https://github.com/apple/darwin-xnu/blob/main/bsd/sys/clonefile.h > > It doesn't help, because it doesn't mention CLONE_ACL. Here is what I found: The version of

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread Paul Eggert
On 2023-02-11 16:38, George Valkov wrote: This might help: https://github.com/apple/darwin-xnu/blob/main/bsd/sys/clonefile.h It doesn't help, because it doesn't mention CLONE_ACL. The kernel should be open source. No, the macOS kernel is not free software. Nor is iOS's kernel.

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread George Valkov
> On 2023-02-11, at 11:53 PM, Paul Eggert wrote: > > On 2023-02-10 17:21, George Valkov wrote: > >> remember to trace all code paths errors. > > Yes, I've been doing that. However, I don't use macOS and the two of us are > debugging remotely where I write the code and you debug it but

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread Paul Eggert
On 2023-02-10 17:21, George Valkov wrote: remember to trace all code paths errors. Yes, I've been doing that. However, I don't use macOS and the two of us are debugging remotely where I write the code and you debug it but neither of us know how macOS works. Of course it would be much more

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread George Valkov
> On 2023-02-11, at 8:19 PM, Paul Eggert wrote: > > On 2023-02-10 17:27, George Valkov wrote: >> I’d prefer a timestamp consistent with source. > > You can get that with 'cp -p' (or with 'cp --preserve=timestamps' etc.). So > this is not an issue of whether you can get what you prefer. It's

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-11 Thread Paul Eggert
On 2023-02-10 17:27, George Valkov wrote: I’d prefer a timestamp consistent with source. You can get that with 'cp -p' (or with 'cp --preserve=timestamps' etc.). So this is not an issue of whether you can get what you prefer. It's merely an issue about what cp's default behavior is, when the

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-11, at 12:02 AM, Pádraig Brady wrote: > > On 10/02/2023 21:50, Paul Eggert wrote: >> On 2/10/23 13:35, George Valkov wrote: >>> Since the source and it’s clone have separate metadata, >>> it should be possible to change it on the clone, to comply with standards. >> Attached is a

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-11, at 2:48 AM, Paul Eggert wrote: > > I want to emphasize the fact that we need to get to the bottom of the > SEEK_HOLE / SEEK_DATA situation on macOS. Core programs other than 'cp' use > these options, and working around the bug in 'cp' won't fix the bug elsewhere. > > One

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 11:50 PM, Paul Eggert wrote: > > On 2/10/23 13:35, George Valkov wrote: > >> Since the source and it’s clone have separate metadata, >> it should be possible to change it on the clone, to comply with standards. > > Attached is a hacky patch to do that. It also uses the

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Paul Eggert
I want to emphasize the fact that we need to get to the bottom of the SEEK_HOLE / SEEK_DATA situation on macOS. Core programs other than 'cp' use these options, and working around the bug in 'cp' won't fix the bug elsewhere. One possibility is to modify Gnulib so that SEEK_HOLE and SEEK_DATA

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 11:46 PM, Pádraig Brady wrote: > > On 10/02/2023 20:45, Paul Eggert wrote: >> On 2/10/23 10:58, Pádraig Brady wrote: >>> I was considering "touch"ing the timestamps after also, >>> but it's better to just maintain them as we're >>> pointing to the same data after all. >>

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Paul Eggert
On 2/10/23 13:46, Pádraig Brady wrote: Maybe. Though POSIX says cp "shall copy" and we're not making a copy, we're making a reflink. If that were an important reason not to clone, then cp should not have made --reflink=auto the default, as clones would not be considered to be copies.

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 21:50, Paul Eggert wrote: On 2/10/23 13:35, George Valkov wrote: Since the source and it’s clone have separate metadata, it should be possible to change it on the clone, to comply with standards. Attached is a hacky patch to do that. It also uses the new CLONE_ACL flag. The old

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Paul Eggert
On 2/10/23 13:35, George Valkov wrote: Since the source and it’s clone have separate metadata, it should be possible to change it on the clone, to comply with standards. Attached is a hacky patch to do that. It also uses the new CLONE_ACL flag. The old code apparently mishandled ACLs; the

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 20:45, Paul Eggert wrote: On 2/10/23 10:58, Pádraig Brady wrote: I was considering "touch"ing the timestamps after also, but it's better to just maintain them as we're pointing to the same data after all. For POSIX conformance we must touch if the user has specified only POSIX

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 10:45 PM, Paul Eggert wrote: > > On 2/10/23 10:58, Pádraig Brady wrote: >> I was considering "touch"ing the timestamps after also, >> but it's better to just maintain them as we're >> pointing to the same data after all. > > For POSIX conformance we must touch if the user

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Paul Eggert
On 2/10/23 10:58, Pádraig Brady wrote: I was considering "touch"ing the timestamps after also, but it's better to just maintain them as we're pointing to the same data after all. For POSIX conformance we must touch if the user has specified only POSIX options (and has not specified -p). And

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 8:58 PM, Pádraig Brady wrote: > > On 10/02/2023 17:24, George Valkov wrote: >>> On 2023-02-10, at 4:02 PM, Pádraig Brady wrote: >>> >>> On 10/02/2023 12:13, George Valkov wrote: > On 2023-02-10, at 11:18 AM, Pádraig Brady wrote: > > I'll apply the simple

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 17:24, George Valkov wrote: On 2023-02-10, at 4:02 PM, Pádraig Brady wrote: On 10/02/2023 12:13, George Valkov wrote: On 2023-02-10, at 11:18 AM, Pádraig Brady wrote: I'll apply the simple patch later I think. I have an interesting idea: If I copy a large file, say the 16

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 4:02 PM, Pádraig Brady wrote: > > On 10/02/2023 12:13, George Valkov wrote: >>> On 2023-02-10, at 11:18 AM, Pádraig Brady wrote: >>> >>> I'll apply the simple patch later I think. >> I have an interesting idea: If I copy a large file, say the 16 GB disk image >> where I

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 12:13, George Valkov wrote: On 2023-02-10, at 11:18 AM, Pádraig Brady wrote: I'll apply the simple patch later I think. I have an interesting idea: If I copy a large file, say the 16 GB disk image where I compiled OpenWRT. So I copy this on the same filesystem, and check disk

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread George Valkov
> On 2023-02-10, at 11:18 AM, Pádraig Brady wrote: > > On 10/02/2023 03:57, Paul Eggert wrote: >> On 2/9/23 01:20, George Valkov wrote: >>> -#ifdef SEEK_HOLE >>> +#if defined(SEEK_HOLE) && !defined(__APPLE__) >> Instead of always disabling the SEEK_HOLE optimization, how about doing >> it only

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 03:57, Paul Eggert wrote: On 2/9/23 01:20, George Valkov wrote: -#ifdef SEEK_HOLE +#if defined(SEEK_HOLE) && !defined(__APPLE__) Instead of always disabling the SEEK_HOLE optimization, how about doing it only on APFS files? Something like the attached, perhaps (this is against

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-10 Thread Pádraig Brady
On 10/02/2023 01:03, George Valkov wrote: On 2023-02-09, at 11:35 PM, Pádraig Brady wrote: On 09/02/2023 17:23, George Valkov wrote: On 2023-02-09, at 6:32 PM, Pádraig Brady wrote: On 09/02/2023 15:57, George Valkov wrote: On 2023-02-09, at 1:56 PM, Pádraig Brady wrote: On 09/02/2023

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread Paul Eggert
On 2/9/23 01:20, George Valkov wrote: -#ifdef SEEK_HOLE +#if defined(SEEK_HOLE) && !defined(__APPLE__) Instead of always disabling the SEEK_HOLE optimization, how about doing it only on APFS files? Something like the attached, perhaps (this is against Savannah master). If APFS is pretty

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread George Valkov
> On 2023-02-09, at 11:35 PM, Pádraig Brady wrote: > > On 09/02/2023 17:23, George Valkov wrote: >>> On 2023-02-09, at 6:32 PM, Pádraig Brady wrote: >>> >>> On 09/02/2023 15:57, George Valkov wrote: > On 2023-02-09, at 1:56 PM, Pádraig Brady wrote: > > On 09/02/2023 09:20,

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread Pádraig Brady
On 09/02/2023 17:23, George Valkov wrote: On 2023-02-09, at 6:32 PM, Pádraig Brady wrote: On 09/02/2023 15:57, George Valkov wrote: On 2023-02-09, at 1:56 PM, Pádraig Brady wrote: On 09/02/2023 09:20, George Valkov wrote: Due to a bug in macOS, sparse copies are corrupted on virtual

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread George Valkov
> On 2023-02-09, at 6:32 PM, Pádraig Brady wrote: > > On 09/02/2023 15:57, George Valkov wrote: >>> On 2023-02-09, at 1:56 PM, Pádraig Brady wrote: >>> >>> On 09/02/2023 09:20, George Valkov wrote: Due to a bug in macOS, sparse copies are corrupted on virtual disks formatted with

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread Pádraig Brady
On 09/02/2023 15:57, George Valkov wrote: On 2023-02-09, at 1:56 PM, Pádraig Brady wrote: On 09/02/2023 09:20, George Valkov wrote: Due to a bug in macOS, sparse copies are corrupted on virtual disks formatted with APFS. HFS is not affected. Affected are coreutils install, and gcp when

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread Pádraig Brady
On 09/02/2023 09:20, George Valkov wrote: Due to a bug in macOS, sparse copies are corrupted on virtual disks formatted with APFS. HFS is not affected. Affected are coreutils install, and gcp when compiled with SEEK_HOLE, as well as macOS Finder. While reading the entire file returns valid

bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

2023-02-09 Thread George Valkov
Due to a bug in macOS, sparse copies are corrupted on virtual disks formatted with APFS. HFS is not affected. Affected are coreutils install, and gcp when compiled with SEEK_HOLE, as well as macOS Finder. While reading the entire file returns valid data, scanning for allocated segments may return