Re: valgrinded executable seem to run far more slowly in Rawhide
On Wed, Jun 09, 2021 at 10:20:28AM +0200, Mark Wielaard wrote: > Hi Rich, > > On Tue, Jun 08, 2021 at 04:09:33PM +0100, Richard W.M. Jones wrote: > > I haven't filed a bug yet because I'm not sure what component to > > assign this too. I upgraded to Rawhide this morning and noticed that > > some tests that previously ran OK are now timing out. It turned out > > that the tests were failing because a valgrinded executable runs much > > more slowly in Rawhide today. > > > > Here's my test: > > > > Upgrade to Rawhide and install these additional packages: > > > > # dnf install libnbd nbdkit valgrind > > > > Unpack the certificates from > > http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp > > > > Run this command as NON-root (it's self-contained and just connects > > the two commands together over localhost port 10809): > > > > $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run > > 'valgrind nbdinfo nbds://localhost' > > > > For me this takes over 40 seconds in Rawhide, versus a couple of > > seconds in Fedora 33/34. Also if you remove "valgrind" it runs > > instantaneously even in Rawhide. If you run it as root, it runs > > quickly again (WTF!) > > > > The perf flamegraph seems to indicate it's doing a lot of stuff with > > DWARF 3 symbols. > > > > http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg > > > > Uninstalling all relevant debuginfo packages didn't seem to help. > > Thanks for using valgrind and sorry we made it even slower. I am > working on it. As you already figured out, it has to do with > debuginfo. Basically valgrind should only really need to read the line > tables, but it does a lot more. This is the fedora bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1662656 > > This has become more urgent now that debuginfod has become the default > in rawhide (valgrind supports debuginfod since 3.17.0): > https://fedoraproject.org/wiki/Changes/DebuginfodByDefault > > That also explains your observations, it depends on whether or not > DEBUGINFOD_URLS is set in the environment (and whether > elfutils-debuginfod-client is installed). FWIW yes it is (to "https://debuginfod.fedoraproject.org/;) Unsetting the environment variable does improve the performance back to expected levels. This also explains the mystery of why "sudo" fixed the problem, since sudo won't pass this environment through to the subprocess. Rich. > I'll work on making valgrind fast again, as far as valgrind is fast to > begin with :) > > Cheers, > > Mark > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: valgrinded executable seem to run far more slowly in Rawhide
Hi Rich, On Tue, Jun 08, 2021 at 04:09:33PM +0100, Richard W.M. Jones wrote: > I haven't filed a bug yet because I'm not sure what component to > assign this too. I upgraded to Rawhide this morning and noticed that > some tests that previously ran OK are now timing out. It turned out > that the tests were failing because a valgrinded executable runs much > more slowly in Rawhide today. > > Here's my test: > > Upgrade to Rawhide and install these additional packages: > > # dnf install libnbd nbdkit valgrind > > Unpack the certificates from > http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp > > Run this command as NON-root (it's self-contained and just connects > the two commands together over localhost port 10809): > > $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run > 'valgrind nbdinfo nbds://localhost' > > For me this takes over 40 seconds in Rawhide, versus a couple of > seconds in Fedora 33/34. Also if you remove "valgrind" it runs > instantaneously even in Rawhide. If you run it as root, it runs > quickly again (WTF!) > > The perf flamegraph seems to indicate it's doing a lot of stuff with > DWARF 3 symbols. > > http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg > > Uninstalling all relevant debuginfo packages didn't seem to help. Thanks for using valgrind and sorry we made it even slower. I am working on it. As you already figured out, it has to do with debuginfo. Basically valgrind should only really need to read the line tables, but it does a lot more. This is the fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1662656 This has become more urgent now that debuginfod has become the default in rawhide (valgrind supports debuginfod since 3.17.0): https://fedoraproject.org/wiki/Changes/DebuginfodByDefault That also explains your observations, it depends on whether or not DEBUGINFOD_URLS is set in the environment (and whether elfutils-debuginfod-client is installed). I'll work on making valgrind fast again, as far as valgrind is fast to begin with :) Cheers, Mark ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
valgrinded executable seem to run far more slowly in Rawhide
I haven't filed a bug yet because I'm not sure what component to assign this too. I upgraded to Rawhide this morning and noticed that some tests that previously ran OK are now timing out. It turned out that the tests were failing because a valgrinded executable runs much more slowly in Rawhide today. Here's my test: Upgrade to Rawhide and install these additional packages: # dnf install libnbd nbdkit valgrind Unpack the certificates from http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp Run this command as NON-root (it's self-contained and just connects the two commands together over localhost port 10809): $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run 'valgrind nbdinfo nbds://localhost' For me this takes over 40 seconds in Rawhide, versus a couple of seconds in Fedora 33/34. Also if you remove "valgrind" it runs instantaneously even in Rawhide. If you run it as root, it runs quickly again (WTF!) The perf flamegraph seems to indicate it's doing a lot of stuff with DWARF 3 symbols. http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg Uninstalling all relevant debuginfo packages didn't seem to help. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
