Re: Selinux slow / How to disable selinux labeling from SPEC Was: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On Friday, 28 January 2022 00:43:07 CET Robert-André Mauchin wrote:
> On 1/24/22 01:30, Robert-André Mauchin wrote:
>
> > Hi,
> >
> > So I have an annoying bug that started near the beginnings of F35.
> > My papirus-icon-theme became very slow to install:
> > https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
> >
> > During the installation, all the files are copied, then renamed by rpm
> > (no idea why it works like this).
> >
> > It works fast in a Mock chroot but incredibly slow on bare metal.
> >
> > I've done a small test of moving files:
> >
> > [root@cassini icons]# mkdir test
> > [root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
> > done
> > [root@cassini icons]# cd test
> >
> > On host:
> >
> > time $(for f in *; do mv "$f" "${f%}.txt"; done)
> > real2m3,500s
> > user0m3,966s
> > sys 2m0,431s
> >
> > In nspawn container:
> >
> > sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt";
> > done)
> > real0m6.702s
> > user0m4.237s
> > sys 0m3.344s
> >
> > Since papirus-icon-theme contains more than 100,000 (small) files, it is
> > a problem. One minute of waiting is ok, 20 mn is not.
> >
> > What can cause this? I read that nspawn virtualizes the file system,
> > could it be file system related on the host? (I use btrfs btw)
> >
> > Any input welcome!
> >
> > Best regards,
> >
> > Robert-André
>
>
>
> Thanks to Panu, it has been determined that the install process of
> papirus-icon-theme spends most of its time labeling the 100,000 files.
> Installing the rpm with rpm -i --nocontexts makes it happen in a
> reasonable time.
> Is there a way to bypass this step from within the SPEC itself?
> It started to happen around F35, do you think I should try to raise a
> bug with SELinux? I don't know how to diagnose this better.
>
> Any input appreciated.
Did you log a bug? Updating texlive is horribly slow ...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 31/01/2022 15:47, Kevin P. Fleming wrote: DNF consumed an entire CPU during that time. https://bugzilla.redhat.com/show_bug.cgi?id=2048168 -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
I just saw this today, btrfs+LUKS, Micron MTFDHBA256TDV. May not have been five minutes, but it was a long time, and DNF consumed an entire CPU during that time. On Sat, Jan 29, 2022 at 3:39 PM Chris Murphy wrote: > On Sat, Jan 29, 2022 at 4:12 AM Vitaly Zaitsev via devel > wrote: > > > > On 24/01/2022 01:52, Tom Hughes via devel wrote: > > > Probably so the file is replaced atomically, and you either > > > have the old or new version but never a partial version. > > > > Can be easily reproduced with FEDORA-2022-a581f05398 update: > > installation of breeze-icon-theme will hang dnf for 5 minutes. Tested on > > 2 different PCs. Both has NVMe SSD and ext4+LUKS. > > Huh, interesting we're getting such mixed results with ext4. We need > more data to see what the pattern is. Some flash drives, as fast as > they are in normal write workloads, can get extremely bogged down and > take minutes to recover. So I wouldn't discount hardware as a > contributing factor, even if it is also a regression. > > > -- > Chris Murphy > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On Sat, Jan 29, 2022 at 4:12 AM Vitaly Zaitsev via devel wrote: > > On 24/01/2022 01:52, Tom Hughes via devel wrote: > > Probably so the file is replaced atomically, and you either > > have the old or new version but never a partial version. > > Can be easily reproduced with FEDORA-2022-a581f05398 update: > installation of breeze-icon-theme will hang dnf for 5 minutes. Tested on > 2 different PCs. Both has NVMe SSD and ext4+LUKS. Huh, interesting we're getting such mixed results with ext4. We need more data to see what the pattern is. Some flash drives, as fast as they are in normal write workloads, can get extremely bogged down and take minutes to recover. So I wouldn't discount hardware as a contributing factor, even if it is also a regression. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 24/01/2022 01:52, Tom Hughes via devel wrote: Probably so the file is replaced atomically, and you either have the old or new version but never a partial version. Can be easily reproduced with FEDORA-2022-a581f05398 update: installation of breeze-icon-theme will hang dnf for 5 minutes. Tested on 2 different PCs. Both has NVMe SSD and ext4+LUKS. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 24/01/2022 01:30, Robert-André Mauchin wrote: So I have an annoying bug that started near the beginnings of F35. My papirus-icon-theme became very slow to install: Can confirm this issue on breeze-icon-theme. The update process hangs for 5 minutes both on my desktop and laptop. Configuration: Samsung NVMe SSD, AMD Ryzen 5800X, ext4+LUKS. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Selinux slow / How to disable selinux labeling from SPEC Was: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 1/24/22 01:30, Robert-André Mauchin wrote:
Hi,
So I have an annoying bug that started near the beginnings of F35.
My papirus-icon-theme became very slow to install:
https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
During the installation, all the files are copied, then renamed by rpm
(no idea why it works like this).
It works fast in a Mock chroot but incredibly slow on bare metal.
I've done a small test of moving files:
[root@cassini icons]# mkdir test
[root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
done
[root@cassini icons]# cd test
On host:
time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 2m3,500s
user 0m3,966s
sys 2m0,431s
In nspawn container:
sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 0m6.702s
user 0m4.237s
sys 0m3.344s
Since papirus-icon-theme contains more than 100,000 (small) files, it is
a problem. One minute of waiting is ok, 20 mn is not.
What can cause this? I read that nspawn virtualizes the file system,
could it be file system related on the host? (I use btrfs btw)
Any input welcome!
Best regards,
Robert-André
Thanks to Panu, it has been determined that the install process of
papirus-icon-theme spends most of its time labeling the 100,000 files.
Installing the rpm with rpm -i --nocontexts makes it happen in a
reasonable time.
Is there a way to bypass this step from within the SPEC itself?
It started to happen around F35, do you think I should try to raise a
bug with SELinux? I don't know how to diagnose this better.
Any input appreciated.
Best regards,
Robert-André
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
Chris Murphy wrote on Mon, Jan 24, 2022 at 02:56:09PM -0700: > Should be true, and if this nspawn container is running on the host > then they should share the same btrfs file system. And even if nspawn > is creating separate subvolumes for the mock build (?not sure if it > does) then because it's a nested subvolume, not mounted, there's no > mount point boundary to cross so you *do* get reflink copies between > subvolumes. For what it's worth the example given does renames in the same directory (since that's what rpm does when installing things anyway), so there is no reflink involved, just a plain rename call I got curious so I tried reproducing but unfortunately I get similar times whether it's in mock or native, but it varies from a run to another. Running with perf record -g, I get the following hog: - 95.11% 0.01% mv [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe 95.10% entry_SYSCALL_64_after_hwframe - do_syscall_64 - 92.40% __x64_sys_renameat2 - 92.40% do_renameat2 - 92.36% vfs_rename - 92.34% btrfs_rename2 - 92.09% btrfs_log_new_name - btrfs_log_inode_parent - 53.42% log_new_dir_dentries - 47.27% btrfs_log_inode + 13.78% btrfs_log_all_xattrs + 11.60% copy_items.isra.0 + 11.58% drop_objectid_items + 4.31% btrfs_search_forward + 2.39% btrfs_search_slot 1.21% btrfs_release_path + 1.00% btrfs_commit_inode_delayed_inode + 2.26% btrfs_search_forward + 1.04% btrfs_iget 0.89% read_extent_buffer - 38.65% btrfs_log_inode - 38.41% log_directory_changes - log_dir_items - 35.01% overwrite_item + 19.04% btrfs_search_slot + 3.59% btrfs_release_path + 3.32% kfree 3.07% __kmalloc 2.84% read_extent_buffer 1.40% btrfs_get_32 0.63% memcmp 1.67% read_extent_buffer So basically the directory modification is costly for each rename and it doesn't seem to get batched, maybe that would give a hint as to somewhere that could be improved. (I wasn't able to reliably produce the slightly shorter times I got to check with perf record, but I did get one run with ~20s when it's usually much longer so there's probably something... Never went as far down as sub-10s though) -- Dominique ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On Mon, Jan 24, 2022 at 11:44 AM Robert-André Mauchin wrote: > Apparently this happens also on ext4 filesystems. :facepalm: haha! ok well then the previous email i just sent can be ignored as it relates to this problem! -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On Mon, Jan 24, 2022 at 11:41 AM Robert-André Mauchin wrote: > > On 1/24/22 05:14, Chris Murphy wrote: > > What file system is being used in each case? > > > > Everything is btrfs. > > > This is a bit obscure but... cp and mv use reflink=auto. On XFS and > > Btrfs this means it'll make reflinks (copies metadata, doesn't > > duplicate the data extents) if it can. Falling back to a full copy > > (metadata and data extents). > > > But both the host and the nspawn container are using btrfs? Should be true, and if this nspawn container is running on the host then they should share the same btrfs file system. And even if nspawn is creating separate subvolumes for the mock build (?not sure if it does) then because it's a nested subvolume, not mounted, there's no mount point boundary to cross so you *do* get reflink copies between subvolumes. > > It might not be possible due to an obscure VFS rule that disallows > > reflinks (for reasons I don't understand) when the copy or move > > crosses mount point boundaries. This includes bind mounts of > > directories. Bind mounts are also what are employed behind the scene > > with 'mount -o subvol' mount option on Btrfs, which we use by default > > in Fedora Workstation and Cloud Edition, and all the desktop spins. > > > > The nspawn container, I'm not super familiar with how it works. I > > think on Btrfs, it will create nested subvolumes, i.e. they are not > > mounted with the subvol mount option, hence no mount point boundary. > > But on other file systems, I think nspawn creates a loop mounted file > > system? > > > > > I've got two subvol: > > UUID=ee9eec69-8710-4503-b389-e16fcde8a0a5 / btrfs >subvol=root,compress=zstd:1 0 0 > > UUID=d7e21336-6ac6-483a-b4f2-aaeecabd8f1f /home btrfs >subvol=home,compress=zstd:1 0 0 > > but when I do my tests there is no subvol crossing, everything happens > on the root subvol? It might be there's a nested subvolume created by nspawn (I'm not sure), so maybe part of it happens in some other subvolume. But there should still be an efficient (reflink) copy. If cp or mv aren't literally invoked, and the copy is done by some library then we'd need to find out what ioctl is actually being called. For example upstream coreutils only just recently cut a new release v9.0 (only in rawhide) that has the enhancement for cp to use reflink=auto. It was previously reflink=never which is what's used most everywhere else other than Fedora. $ strace cp --reflink=always A B ... ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0 $ strace cp --reflink=never A B ... fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faf80f5e000 read(3, "2022/01/12 13:48:21 Starting Blu"..., 131072) = 1756 write(4, "2022/01/12 13:48:21 Starting Blu"..., 1756) = 1756 Sorry though if this is a goose chase. I can't tell if it's a factor in what's going on. But maybe someone else will find this interesting :D There is a mostly reliable way to determine if a file is a reflink copy. Before the copy, look at the file: $ filefrag -v A ... ext: logical_offset:physical_offset: length: expected: flags: 0:0.. 14: 10103641.. 10103655: 15: last,encoded,eof ... The key take away is 10103641. Let's copy it within the same directory: $ cp A B $ filefrag -v B ... 0:0.. 14: 10103641.. 10103655: 15: last,encoded,shared,eof ... Again 1013641. So the data extent location is the same, which is only possible with a reflink copy, and hence how reflinks go by a more technical name, shared extents. And you also see in the flags column "shared". That flag is only there because both A and B exist. If I remove A or B, such that there is only one file using those extents, they're no longer shared, so the "shared" flag won't be there. Hence my emphasis on the address. There *is* logical block address reuse in Btrfs but due to COW, it's not going to be reused less than about a minute. $ cp --reflink=never A C $ filefrag -v C ... 0:0.. 14: 10398358.. 10398372: 15: last,encoded,eof Different location because the data extents were duplicated, not shared. This is the same on XFS too. The subtle differences maybe don't matter here much. A btrfs subvolume does have it's own st_dev, so things like rsync -x and borg will not cross subvolume boundaries. Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 1/24/22 01:30, Robert-André Mauchin wrote:
Hi,
So I have an annoying bug that started near the beginnings of F35.
My papirus-icon-theme became very slow to install:
https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
During the installation, all the files are copied, then renamed by rpm
(no idea why it works like this).
It works fast in a Mock chroot but incredibly slow on bare metal.
I've done a small test of moving files:
[root@cassini icons]# mkdir test
[root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
done
[root@cassini icons]# cd test
On host:
time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 2m3,500s
user 0m3,966s
sys 2m0,431s
In nspawn container:
sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 0m6.702s
user 0m4.237s
sys 0m3.344s
Since papirus-icon-theme contains more than 100,000 (small) files, it is
a problem. One minute of waiting is ok, 20 mn is not.
What can cause this? I read that nspawn virtualizes the file system,
could it be file system related on the host? (I use btrfs btw)
Any input welcome!
Best regards,
Robert-André
Apparently this happens also on ext4 filesystems.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 1/24/22 05:14, Chris Murphy wrote:
On Sun, Jan 23, 2022 at 5:30 PM Robert-André Mauchin wrote:
Hi,
So I have an annoying bug that started near the beginnings of F35.
My papirus-icon-theme became very slow to install:
https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
During the installation, all the files are copied, then renamed by rpm
(no idea why it works like this).
It works fast in a Mock chroot but incredibly slow on bare metal.
I've done a small test of moving files:
[root@cassini icons]# mkdir test
[root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
done
[root@cassini icons]# cd test
On host:
time $(for f in *; do mv "$f" "${f%}.txt"; done)
real2m3,500s
user0m3,966s
sys 2m0,431s
In nspawn container:
sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
real0m6.702s
user0m4.237s
sys 0m3.344s
Since papirus-icon-theme contains more than 100,000 (small) files, it is
a problem. One minute of waiting is ok, 20 mn is not.
What can cause this? I read that nspawn virtualizes the file system,
could it be file system related on the host? (I use btrfs btw)
Any input welcome!
What file system is being used in each case?
Everything is btrfs.
This is a bit obscure but... cp and mv use reflink=auto. On XFS and
Btrfs this means it'll make reflinks (copies metadata, doesn't
duplicate the data extents) if it can. Falling back to a full copy
(metadata and data extents).
But both the host and the nspawn container are using btrfs?
It might not be possible due to an obscure VFS rule that disallows
reflinks (for reasons I don't understand) when the copy or move
crosses mount point boundaries. This includes bind mounts of
directories. Bind mounts are also what are employed behind the scene
with 'mount -o subvol' mount option on Btrfs, which we use by default
in Fedora Workstation and Cloud Edition, and all the desktop spins.
The nspawn container, I'm not super familiar with how it works. I
think on Btrfs, it will create nested subvolumes, i.e. they are not
mounted with the subvol mount option, hence no mount point boundary.
But on other file systems, I think nspawn creates a loop mounted file
system?
I've got two subvol:
UUID=ee9eec69-8710-4503-b389-e16fcde8a0a5 / btrfs
subvol=root,compress=zstd:1 0 0
UUID=d7e21336-6ac6-483a-b4f2-aaeecabd8f1f /home btrfs
subvol=home,compress=zstd:1 0 0
but when I do my tests there is no subvol crossing, everything happens
on the root subvol?
Thanks for your input.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On Sun, Jan 23, 2022 at 5:30 PM Robert-André Mauchin wrote:
>
> Hi,
>
> So I have an annoying bug that started near the beginnings of F35.
> My papirus-icon-theme became very slow to install:
> https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
>
> During the installation, all the files are copied, then renamed by rpm
> (no idea why it works like this).
>
> It works fast in a Mock chroot but incredibly slow on bare metal.
>
> I've done a small test of moving files:
>
> [root@cassini icons]# mkdir test
> [root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
> done
> [root@cassini icons]# cd test
>
> On host:
>
> time $(for f in *; do mv "$f" "${f%}.txt"; done)
> real2m3,500s
> user0m3,966s
> sys 2m0,431s
>
> In nspawn container:
>
> sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
> real0m6.702s
> user0m4.237s
> sys 0m3.344s
>
> Since papirus-icon-theme contains more than 100,000 (small) files, it is
> a problem. One minute of waiting is ok, 20 mn is not.
>
> What can cause this? I read that nspawn virtualizes the file system,
> could it be file system related on the host? (I use btrfs btw)
>
> Any input welcome!
What file system is being used in each case?
This is a bit obscure but... cp and mv use reflink=auto. On XFS and
Btrfs this means it'll make reflinks (copies metadata, doesn't
duplicate the data extents) if it can. Falling back to a full copy
(metadata and data extents).
It might not be possible due to an obscure VFS rule that disallows
reflinks (for reasons I don't understand) when the copy or move
crosses mount point boundaries. This includes bind mounts of
directories. Bind mounts are also what are employed behind the scene
with 'mount -o subvol' mount option on Btrfs, which we use by default
in Fedora Workstation and Cloud Edition, and all the desktop spins.
The nspawn container, I'm not super familiar with how it works. I
think on Btrfs, it will create nested subvolumes, i.e. they are not
mounted with the subvol mount option, hence no mount point boundary.
But on other file systems, I think nspawn creates a loop mounted file
system?
--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 1/24/22 01:52, Tom Hughes wrote:
Do you have the nosync plugin enabled in your mock? That
will shim system calls that try and sync to disk and suppress
the actual sync in the name of greater performance.
Tom
I don't think so, my config is
config_opts['chroothome'] = '/builddir'
config_opts['basedir'] = '/var/lib/mock'
config_opts['rpmbuild_timeout'] = 86400
config_opts['chroot_setup_cmd'] = 'install @buildsys-build pigz lbzip2'
config_opts['target_arch'] = 'x86_64'
config_opts['legal_host_arches'] = ('x86_64',)
config_opts['root'] = 'f36-candidate-x86_64'
config_opts['extra_chroot_dirs'] = [ '/run/lock', ]
config_opts['package_manager'] = 'dnf'
config_opts['use_bootstrap'] = True
config_opts['environment']['LANG'] = 'C.UTF-8'
config_opts['releasever'] = '36'
config_opts['rpmbuild_networking'] = False
config_opts['use_host_resolv'] = False
config_opts['dynamic_buildrequires'] = True
config_opts['dynamic_buildrequires_max_loops'] = 10
config_opts['plugin_conf']['root_cache_enable'] = True
config_opts['plugin_conf']['yum_cache_enable'] = True
config_opts['plugin_conf']['ccache_enable'] = True
config_opts['macros']['%_host'] = 'x86_64-koji-linux-gnu'
config_opts['macros']['%_host_cpu'] = 'x86_64'
config_opts['macros']['%vendor'] = 'Koji'
config_opts['macros']['%distribution'] = 'fc36'
config_opts['macros']['%_topdir'] = '/builddir/build'
config_opts['macros']['%_rpmfilename'] =
'%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'
config_opts['macros']['%packager'] = 'Koji'
config_opts['plugin_conf']['package_state_enable'] = False
config_opts['plugin_conf']['ccache_opts']['compress'] = True
config_opts['plugin_conf']['root_cache_opts']['compress_program'] = "pigz"
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'
config_opts['yum.conf'] = """
[main]
keepcache=1
debuglevel=2
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=
install_weak_deps=0
metadata_timer_sync=8
metadata_expire=8
mdpolicy=group:primary
best=1
module_platform_id=platform:f36
fastestmirror=1
max_parallel_downloads=8
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
On 24/01/2022 00:30, Robert-André Mauchin wrote:
During the installation, all the files are copied, then renamed by rpm
(no idea why it works like this).
Probably so the file is replaced atomically, and you either
have the old or new version but never a partial version.
It works fast in a Mock chroot but incredibly slow on bare metal.
I've done a small test of moving files:
[root@cassini icons]# mkdir test
[root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
done
[root@cassini icons]# cd test
On host:
time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 2m3,500s
user 0m3,966s
sys 2m0,431s
In nspawn container:
sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
real 0m6.702s
user 0m4.237s
sys 0m3.344s
Since papirus-icon-theme contains more than 100,000 (small) files, it is
a problem. One minute of waiting is ok, 20 mn is not.
What can cause this? I read that nspawn virtualizes the file system,
could it be file system related on the host? (I use btrfs btw)
Do you have the nosync plugin enabled in your mock? That
will shim system calls that try and sync to disk and suppress
the actual sync in the name of greater performance.
Tom
--
Tom Hughes (t...@compton.nu)
http://compton.nu/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
mv is massively slower on the host rather than in a nspawn chroot, regression somewhere?
Hi,
So I have an annoying bug that started near the beginnings of F35.
My papirus-icon-theme became very slow to install:
https://bugzilla.redhat.com/show_bug.cgi?id=2029709#c18
During the installation, all the files are copied, then renamed by rpm
(no idea why it works like this).
It works fast in a Mock chroot but incredibly slow on bare metal.
I've done a small test of moving files:
[root@cassini icons]# mkdir test
[root@cassini icons]# for (( i = 0; i < 1; i++ )) do > test/file_$i;
done
[root@cassini icons]# cd test
On host:
time $(for f in *; do mv "$f" "${f%}.txt"; done)
real2m3,500s
user0m3,966s
sys 2m0,431s
In nspawn container:
sh-5.1# time $(for f in *; do mv "$f" "${f%}.txt"; done)
real0m6.702s
user0m4.237s
sys 0m3.344s
Since papirus-icon-theme contains more than 100,000 (small) files, it is
a problem. One minute of waiting is ok, 20 mn is not.
What can cause this? I read that nspawn virtualizes the file system,
could it be file system related on the host? (I use btrfs btw)
Any input welcome!
Best regards,
Robert-André
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
