Re: Bug 252358 - cp(1) of large files is causing 100% CPU utilization and poor transfer of ~168M/minute

2021-01-03 Thread Rick Macklem
Well, it depends on what "stressdisk" (never heard of it) does. If it: 1 - Reads a large non-sparse file on a UFS file system. 2 - Uses lseek(SEEK_DATA/SEEK_HOLE) with a small blocksize or copy_file_range(2) with a small length. (copy_file_range(2) does SEEK_DATA/SEEK_HOLE internally.)

Bug 252358 - cp(1) of large files is causing 100% CPU utilization and poor transfer of ~168M/minute

2021-01-03 Thread Graham Perrin
On 02/01/2021 20:05, Matthias Apitz wrote: Here we go:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252358 Thank you. I plan to build and install 82397d791966b09d344251bc709cd9db2b3a1902; In the

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
LGTM! This patch also fixes another problem: the previous version of cp, when copying a large sparse file on UFS, would create some UFS indirect blocks (because it would keep truncating the file to larger sizes). The output file would still be sparse, but it would take up more space than the

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Rick Macklem
The attached small patch seems to fix the problem. My hunch is that, for a large non-sparse file, SEEK_DATA SEEK_HOLE takes a fairly long time. These are done for each copy_file_range(2) syscall. cp was doing lots of them because of the small len argument. Bumping the len up to SSIZE_MAX results

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Matthias Apitz
El día sábado, enero 02, 2021 a las 11:29:36a. m. -0700, Alan Somers escribió: > > El día sábado, enero 02, 2021 a las 05:06:05p. m. +, Rick Macklem > > escribió: > > > > > Just fyi, I've reproduced the problem. > > > All I did was create a 20Gbyte file > > > on UFS on a slow (4Gbyte or RAM,

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
On Sat, Jan 2, 2021 at 11:28 AM Matthias Apitz wrote: > El día sábado, enero 02, 2021 a las 05:06:05p. m. +, Rick Macklem > escribió: > > > Just fyi, I've reproduced the problem. > > All I did was create a 20Gbyte file > > on UFS on a slow (4Gbyte or RAM, > > slow spinning disk) laptop. > >

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Matthias Apitz
El día sábado, enero 02, 2021 a las 05:06:05p. m. +, Rick Macklem escribió: > Just fyi, I've reproduced the problem. > All I did was create a 20Gbyte file > on UFS on a slow (4Gbyte or RAM, > slow spinning disk) laptop. > (The UFS file system is just what the installer creates these days.) >

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
It seems pretty clear to me, based on my experiments, that FIOSEEKHOLE is O(n) in filesize on UFS (though not on ZFS). And vn_generic_copy_file_range calls VOP_IOCTL(..., FIOSEEKHOLE) once for every block it copies, where blocks can range from 4KB to 1 MB. So I think there are three required

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Rick Macklem
Just fyi, I've reproduced the problem. All I did was create a 20Gbyte file on UFS on a slow (4Gbyte or RAM, slow spinning disk) laptop. (The UFS file system is just what the installer creates these days.) cp still hasn't finished and is definitely taking a looott longer than dd did. I'll start

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
On Sat, Jan 2, 2021 at 9:12 AM Matthias Apitz wrote: > El día sábado, enero 02, 2021 a las 09:06:24a. m. -0700, Alan Somers > escribió: > > > > As I said, it can be reproduced using only the local file system. This > > > was setup recently on a SSD: > > > > > > # dmesg | grep ada0 > > > ada0 at

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Matthias Apitz
El día sábado, enero 02, 2021 a las 09:06:24a. m. -0700, Alan Somers escribió: > > As I said, it can be reproduced using only the local file system. This > > was setup recently on a SSD: > > > > # dmesg | grep ada0 > > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > > ada0: ACS-2 ATA SATA 3.x

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
On Sat, Jan 2, 2021 at 9:02 AM Matthias Apitz wrote: > El día sábado, enero 02, 2021 a las 08:42:01a. m. -0700, Alan Somers > escribió: > > > > # dd if=guru-20210102.tar.gz > of=/mnt/AcerC720/backups/guru-20210102.tar.gz > > > bs=8m > > > 4601+1 records in > > > 4601+1 records out > > >

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Matthias Apitz
El día sábado, enero 02, 2021 a las 08:42:01a. m. -0700, Alan Somers escribió: > > # dd if=guru-20210102.tar.gz of=/mnt/AcerC720/backups/guru-20210102.tar.gz > > bs=8m > > 4601+1 records in > > 4601+1 records out > > 38603862368 bytes transferred in 506.778929 secs (76174956 bytes/sec) > > # ls

Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Alan Somers
On Sat, Jan 2, 2021 at 7:59 AM Matthias Apitz wrote: > > This is with: > > # uname -a > FreeBSD c720-r368166 13.0-CURRENT FreeBSD 13.0-CURRENT #23 r368166M: Thu > Dec 17 13:12:37 CET 2020 > guru@c720-r368166:/usr/obj/usr/src/amd64.amd64/sys/GENERIC > amd64 > > I copy often large backup

cp(1) of large files is causing 100% CPU utilization and poor transfer

2021-01-02 Thread Matthias Apitz
This is with: # uname -a FreeBSD c720-r368166 13.0-CURRENT FreeBSD 13.0-CURRENT #23 r368166M: Thu Dec 17 13:12:37 CET 2020 guru@c720-r368166:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I copy often large backup files to an external USB disk and hit the following problem since updating