Re: Artifacts of failed builds for debugging
Thanks for the tips. I eventually narrowed errors to just NUMA builders, and confirmed that once borrowed multicore machine from RH. Turned out the problem is internal to BIND only, reported under MR [1]. It is problematic to hide details on kyua unit testing. Unit tests are built using cmocka and can print nice output. Unfortunately it uses kyua to group multiple results and that tool hides any details. Without easy way to provide them. Not sure which tool should be updated, kyua is the best candidate for more detailed output. It should be always easy to make building more verbose, especially in such weird errors. Thanks for help. 1. https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/4582 On 1/19/21 5:32 PM, Vít Ondruch wrote: > It is tedious, but you can try gdb in batch mode, e.g.: > > ~~~ > gdb /usr/bin/ruby-mri -batch \ > -ex 'set breakpoint pending on' \ > -ex 'r -Ilib:test -e '\''Dir.glob %|./test/**/*_test.rb|, > (:require)'\'' -- -v' \ > -ex 'bt full' \ > -ex c \ > -ex 'bt full' \ > -ex quit > ~~~ > > (and of course modify this to fit your use case). > > > Vít > > > Dne 18. 01. 21 v 23:22 Petr Menšík napsal(a): >> Hello, >> >> I am trying to build new bind on Rawhide, but I suddenly get quite high >> rate of failures during builds. Strange is the build is always >> successful, when I try fedpkg mockbuild on my machine. >> >> I tried several tries [1][2][3] and scratch builds fail often. Even >> production builds fail, but I think they haven't failed until recently. >> I were unable to reproduce it at least once in mockbuild. >> >> Is it possible to record crashes during builds using abrt or faf? Is >> there way to obtain more details of failed builds? Is there anything >> similar to mockbuild --no-cleanup-after, which could help me identify >> the cause? >> >> Do you have any hints? I tried deleting /var/lib/mock and >> /var/cache/mock, but that haven't changed anything. COPR [4] haven't >> failed also. >> >> Regards, >> Petr >> >> 1. https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310 >> 2. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548 >> 3. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586 >> 4. >> https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/ >> >> ___ >> devel mailing list -- devel@lists.fedoraproject.org >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org >> > > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > -- Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemen...@redhat.com PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Artifacts of failed builds for debugging
It is tedious, but you can try gdb in batch mode, e.g.: ~~~ gdb /usr/bin/ruby-mri -batch \ -ex 'set breakpoint pending on' \ -ex 'r -Ilib:test -e '\''Dir.glob %|./test/**/*_test.rb|, (:require)'\'' -- -v' \ -ex 'bt full' \ -ex c \ -ex 'bt full' \ -ex quit ~~~ (and of course modify this to fit your use case). Vít Dne 18. 01. 21 v 23:22 Petr Menšík napsal(a): Hello, I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine. I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild. Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause? Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also. Regards, Petr 1. https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310 2. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548 3. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586 4. https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org OpenPGP_0x0CE09EE79917B87C.asc Description: application/pgp-keys OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Artifacts of failed builds for debugging
On Mon, Jan 18, 2021 at 11:22:10PM +0100, Petr Menšík wrote: > Hello, > > I am trying to build new bind on Rawhide, but I suddenly get quite high > rate of failures during builds. Strange is the build is always > successful, when I try fedpkg mockbuild on my machine. > > I tried several tries [1][2][3] and scratch builds fail often. Even > production builds fail, but I think they haven't failed until recently. > I were unable to reproduce it at least once in mockbuild. > > Is it possible to record crashes during builds using abrt or faf? Is > there way to obtain more details of failed builds? Is there anything > similar to mockbuild --no-cleanup-after, which could help me identify > the cause? > > Do you have any hints? I tried deleting /var/lib/mock and > /var/cache/mock, but that haven't changed anything. COPR [4] haven't > failed also. I note that all of these seem to have failed on buildhw-x86 builders. Those are bare metal hardware with tons of cpu and memory. Could be something particular to them? Otherwise it looks like a toolchain issue perhaps? You could try adding more debugging to the test suite and doing scratch builds? Perhaps the tools folks will have some subbestion... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Artifacts of failed builds for debugging
Hello, I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine. I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild. Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause? Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also. Regards, Petr 1. https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310 2. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548 3. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586 4. https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/ -- Petr Menšík Software Engineer, Red Hat, http://www.redhat.com/ email: pemen...@redhat.com PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
