On Sun, Jan 29, 2023 at 9:41 AM Richard Purdie
<[email protected]> wrote:
>
> Hi All,
>
> I've going to be a distracted this next week with other project related
> things and the autobuilder is showing a number of problems. I've spent
> part of my weekend trying to stabilise things so I could not have the
> distraction and whilst I've had partial success, things aren't
> resolved. I'm sending this out in the hope others may be able to
> help/assist. The issues:
>
> Kernel 6.1 reproducibility issue in perf
> ========================================
>
> We've had two fixes which do seem to have fixed the kernel-devsrc issue
> but not the perf one. The diffoscope:
>
> http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230129-8bp_1b_z/packages/diff-html/
>
> The autobuilder failure:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/2314/steps/13/logs/stdio
>
> Part of the issue is this seems to be host contamination, so you do
> need to build on a different host. The first time the test runs, it
> does so on the same host and the output is unchanged which is why we've
> had green builds but then failures.
>
> I did try removing the sstate for perf, the issue came back, so it
> isn't a cache contamination issue afaik.
>
>
> bitbake timeout on sending commands
> ===================================
>
> Bitbake runCommand appears to be able to hang in the send since
> there is no timeout set there. This can cause things to block. We need
> to put a worse case timeout/exit in there.
>
>
> unfs tests leave nfs server running
> ===================================
>
> After the unfs NFS test runs, we seem to leave an unfs server hanging
> around after the builds are long gone. These build up on the
> autobuilder workers.
>
>
> http-server in dnf tests breaking
> =================================
>
> We've started seeing a lot of short packets with the http-server used
> by the dnf package manager tests. dnf retries and succeeds leaving a
> failure in the logs which parse_logs picks up.
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/6592/steps/14/logs/stdio
> Central error: 2023-01-29T11:43:32+0000 INFO Error during transfer: Curl
> error (18): Transferred a partial file for
> http://192.168.7.1:36183/cortexa15t2hf_neon/libc6-2.36-r0.cortexa15t2hf_neon.rpm
> [transfer closed with 15758 bytes remaining to read]
> https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6592/steps/14/logs/stdio
> Central error: 2023-01-29T11:07:31+0000 INFO Error during transfer: Curl
> error (18): Transferred a partial file for
> http://192.168.7.3:42417/core2_32/busybox-1.35.0-r0.core2_32.rpm [transfer
> closed with 14627 bytes remaining to read]
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/6598/steps/13/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6532
>
> and a load more. Why the sudden increase in frequency, not sure. Could
> be 6.1 kernel related.
>
>
> asyncio issues with PR Serv
> ===========================
>
> I've found at least one bug with what the code was doing and fixing
> that should help some of the test failures we've seen. When looking
> into that I found more problems though.
>
> If you run "oe-selftest -r prservice -j 3 -K" and then look at the
> bitbake-cookerdaemon.log, it shows a load of python asyncio issues. The
> first one is:
>
> /usr/lib/python3.10/asyncio/base_events.py:685: ResourceWarning:
> unclosed event loop <_UnixSelectorEventLoop running=False closed=False
> debug=False>
> _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
>
> showing connections to the PR service aren't being cleaned up.
I can take a look at this if no one else gets there first, but not
until next week
>
> Also:
>
> /home/pokybuild/yocto-worker/oe-selftest-
> debian/build/bitbake/lib/bb/codeparser.py:472: ResourceWarning:
> unclosed <socket.socket fd=15, family=AddressFamily.AF_INET,
> type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 47910),
> raddr=('127.0.0.1', 40751)>
> self.process_tokens(more_tokens)
>
> but the codeparser reference isn't accurate, it is just from the
> garbage collection point.
>
> The second issue is that in running the selftests, a PR Service process
> often seems to be left behind without bitbake running. It suggests some
> kind of lifecycle issue somewhere. It might only be happening in
> failure cases, not sure.
>
> The third issue seems to be around event loop shutdown. On debian11
> with py 3.9, when bitbake is shutting down we're seeing:
>
> 412934 07:12:26.081334 Exiting (socket: True)
> /usr/lib/python3.9/signal.py:30: ResourceWarning: unclosed
> <socket.socket fd=22, family=AddressFamily.AF_INET,
> type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 36287), r
> addr=('127.0.0.1', 36422)>
> return enum_klass(value)
> ResourceWarning: Enable tracemalloc to get the object allocation
> traceback
> /usr/lib/python3.9/asyncio/selector_events.py:704: ResourceWarning:
> unclosed transport <_SelectorSocketTransport fd=22>
> _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
> ResourceWarning: Enable tracemalloc to get the object allocation
> traceback
> Traceback (most recent call last):
> File "/home/pokybuild/yocto-worker/oe-selftest-
> debian/build/bitbake/lib/bb/asyncrpc/serv.py", line 71, in
> process_requests
> d = await self.read_message()
> GeneratorExit
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/home/pokybuild/yocto-worker/oe-selftest-
> debian/build/bitbake/lib/bb/asyncrpc/serv.py", line 201, in
> handle_client
> await client.process_requests()
> File "/home/pokybuild/yocto-worker/oe-selftest-
> debian/build/bitbake/lib/bb/asyncrpc/serv.py", line 79, in
> process_requests
> self.writer.close()
> File "/usr/lib/python3.9/asyncio/streams.py", line 353, in close
> return self._transport.close()
> File "/usr/lib/python3.9/asyncio/selector_events.py", line 700, in
> close
> self._loop.call_soon(self._call_connection_lost, None)
> File "/usr/lib/python3.9/asyncio/base_events.py", line 746, in
> call_soon
> self._check_closed()
> File "/usr/lib/python3.9/asyncio/base_events.py", line 510, in
> _check_closed
> raise RuntimeError('Event loop is closed')
> RuntimeError: Event loop is closed
> Task was destroyed but it is pending!
> task: <Task pending name='Task-2540' coro=<AsyncServer.h
>
>
> If anyone can help with any of these please reply to the email. We
> probably need to transfer some into bugzilla, I just wanted to at least
> get something written down.
>
> Cheers,
>
> Richard
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#176553):
https://lists.openembedded.org/g/openembedded-core/message/176553
Mute This Topic: https://lists.openembedded.org/mt/96607332/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
