> On 2023-02-10, at 4:02 PM, Pádraig Brady <p...@draigbrady.com> wrote: > > On 10/02/2023 12:13, George Valkov wrote: >>> On 2023-02-10, at 11:18 AM, Pádraig Brady <p...@draigbrady.com> wrote: >>> >>> I'll apply the simple patch later I think. >> I have an interesting idea: If I copy a large file, say the 16 GB disk image >> where I compiled OpenWRT. So I copy this on the same filesystem, and check >> disk usage before an after: no change. Also the copy is instant. That’s >> because >> APFS supports copy-on-write. Initially both files share the same sectors on >> disk, >> any changes made after that are copied to new sectors. >> This is the most efficient way to copy files on APFS, and should produce no >> corruption. >> Let’s implement it. I would assume there is a system cal that does the >> entire copy, >> And we don’t need to read and write any data. >> Does the trace contain any interesting calls that might be related to that? > > When you say "I copy a large file", is that with gcp or something else?
Finder: option + drag in the same directory > Since coreutils 9.1 we try the CoW by default with fclonefileat() > which is available since macOS 10.12 (2016). I can confirms fclonefileat works on macOS 12.6.3 and solves the issue. File attributes are also preserved. To clarify: these observations are from the sample below. I don’t have internal experience with cloreutils, but I can test it if you provide some description. > Note that works only when src and dest are within the same file system, > but that is the case for you if I'm reading your original report correctly. Correct. fclonefileat only works on the same file system. If we attempt to clone to another volume, fclonefileat fails 18 Cross-device link. When building OpenWRT everything works on the same file system, so fclonefileat is applicable. > When I mentioned that earlier I thought your macOS version was too old to > support that, > but in fact your 12.6.3 should support fclonefileat() fine. Thankfully it is supported. The latest security update came in January 2023. > So that's another variable. Is that call failing completely for you? > You might be quicker to add a couple of printfs around the > fclonefileat() calls in the coreutils from latest git you compliled. > Note there is new error handling related to that call in the latest git, > compared to what was released in coreutils 9.1. Coreutils tests on master git clone git://git.savannah.gnu.org/coreutils.git mv coreutils coreutils-clone cd coreutils-clone git submodule foreach git pull origin master ./bootstrap ./configure make -j 16 git log commit d374d32ccf12f8cf33c8f823d573498b7c8b27a4 (HEAD -> clone, origin/master, origin/HEAD, master) cd .. ./coreutils-clone/src/cp cc1 cc1-test printf("HAVE_FCLONEFILEAT %u USE_XATTR %u\n", HAVE_FCLONEFILEAT, USE_XATTR); HAVE_FCLONEFILEAT 1 USE_XATTR 0 int fc_flags = x->preserve_ownership ? 0 : CLONE_NOOWNERCOPY; printf( "data_copy_required %u x->reflink_mode %u\n" "x->preserve_mode %u x->preserve_timestamps %u\n" "x->preserve_ownership %u CLONE_NOOWNERCOPY %u\n", data_copy_required, x->reflink_mode, x->preserve_mode, x->preserve_timestamps, x->preserve_ownership, CLONE_NOOWNERCOPY ); if (data_copy_required && x->reflink_mode && x->preserve_mode && x->preserve_timestamps && (x->preserve_ownership || CLONE_NOOWNERCOPY)) { int a = fclonefileat (source_desc, dst_dirfd, dst_relname, fc_flags); int e = errno; printf("fclonefileat %i %i\n", a, e); if (a == 0) goto close_src_desc; data_copy_required 1 x->reflink_mode 1 x->preserve_mode 0 x->preserve_timestamps 0 x->preserve_ownership 0 CLONE_NOOWNERCOPY 2 fclonefileat is not used because x->preserve_mode = 0, x->preserve_timestamps = 0 That design seems wrong. The fact than no one requested to preserve that data, doesn’t mean preserving it via fclonefileat is a bad thing. I think we should always call fclonefileat and proceed to other means if it fails. It is more efficient, since it requires no additional space or data to be copied. And always produces a valid copy. > Note also I don't see any fclonefileat() syscalls in your Finder logs at > least. Great news: fclonefileat works on the same file system, and if we need to copy outside, then just disable sparse copy according to my patch. One option is to start blindly with fclonefileat, and if that fails, fall back to normal copy. If you can use the trace to find what Finder is doing, we can try it. It might bring some improvements. Else, disable sparse and do a regular copy. It was easier for me to write a small sample: // clone.c #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <sys/clonefile.h> int main(int argc, char ** argv) { int fd = open("cc1", O_RDONLY); int dir = open("./", O_RDONLY); int a = fclonefileat(fd, dir, "cc1-clone", 0); close(fd); close(dir); printf("fd %i dir %i fclonefileat %i\n", fd, dir, a); return 0; } ./coreutils/src/cp cc1 cc1-ori-before gcc clone.c && ./a.out ./coreutils/src/cp cc1 cc1-ori-after ll cc1* -rwxr-xr-x 1 g staff 27551296 Feb 10 17:35 cc1* -rwxr-xr-x 1 g staff 27551296 Feb 10 17:35 cc1-clone* -rwxr-xr-x 1 g staff 27551296 Feb 10 17:39 cc1-ori-after* -rwxr-xr-x 1 g staff 27551296 Feb 10 17:39 cc1-ori-before* sha1sum 812cb0591b000f56bdf75c0f3f94c622e371b970 cc1 812cb0591b000f56bdf75c0f3f94c622e371b970 cc1-clone 812cb0591b000f56bdf75c0f3f94c622e371b970 cc1-ori-after 7b447132f56f3b549ef62a4780945e82d26f7d2c cc1-ori-before df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1s1 1.1Ti 1.0Ti 58Gi 95% 2400403 605165920 0% /System/Volumes/Data # clone a large file 15 GB using fclonefileat ./a.out # copy a large file 15 GB using Finder: option + drag in the same directory # both complete instantly df -h Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1s1 1.1Ti 1.0Ti 58Gi 95% 2400401 605207240 0% /System/Volumes/Data ll -h /Users/g/vhd/OpenWRT.wrt3200* -rw-r--r--@ 1 g staff 15G Feb 10 04:06 /Users/g/vhd/OpenWRT.wrt3200-clone.sparseimage -rw-r--r--@ 1 g staff 15G Feb 10 04:06 /Users/g/vhd/OpenWRT.wrt3200-finder.sparseimage -rw-r--r--@ 1 g staff 15G Feb 10 04:06 /Users/g/vhd/OpenWRT.wrt3200.sparseimage Cheers mate! Georgi Valkov httpstorm.com nano RTOS