On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
lists.openembedded.org wrote:
> I've spent quite a bit of time trying to reproduce/debug this:
>
> https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
>
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
>
> on debian11-vk-1.
>
> I've a successful test and a failed test testimage output for
> comparison:
>
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
>
> along with qemu serial output:
>
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
>
> I've tried rerunning that exact series of tests on that worker with
> no
> success reproducing the issue. The image was overwritten by a later
> test so we can't retest the exact image. I've checked the journal on
> that system and there isn't anything related around the time this
> happened. The two failures were on different network interfaces and
> both interfaces work on later tests.
This is happening on most builds on debian11 for my test branch. If I
add new changes, the issue doesn't happen so it looks to be timing
related and needs a populated sstate cache.
I put auditing into runqemu's tap locks codepaths and the devices are
being locked/released correctly, there is no duplicate device usage.
I also went through the non-tap/tun codepaths and I can't spot any
issues, we do use slirp in some tests.
Since we have a trigger point where we know it is failing (the runtime
ping and ssh tests), I added a os.system("ps awx") into it. That gave
me a process dump of what was running when this happens.
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146
>From those we can conclude that there is only one qemu-system-* running
using the appropriate interface. I couldn't spot any other particularly
untoward processes running. That tells us a lot of things it isn't I
guess.
In addition to the ps, I've now added a netstat -tunape to see if we
can see something holding a network connection open. I'm wondering
about our httpserver used in some tests, or debuginfod, or something.
That build is ongoing.
> I'm open to ideas...
I still am, this one is proving to be a pain...
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206082):
https://lists.openembedded.org/g/openembedded-core/message/206082
Mute This Topic: https://lists.openembedded.org/mt/108984574/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
