To reproduce we can set:
# the real port is 8687 and with the 18687 we can simulate a firewall block
BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:18687"
With the V2 patch and we get on the log:
WARNING: BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash
equivalence server at 'typhoon.yocto.io:18687': timeout('timed out')
and bitbake deadlock in Initialising tasks:
Initialising tasks: 44%
|####################################################
| ETA:
0:00:00
on bitbake-cookerdaemon.log we have:
3 19:32:26.434306 Running command ['buildTargets', ['core-image-minimal'],
'build']
63 19:32:26.434386 Command Completed
Error talking to server: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Error talking to server: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Error talking to server: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Error talking to server: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Error from client: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Traceback (most recent call last):
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 57, in
_send_wrapper
await self.connect()
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 43, in connect
(self.reader, self.writer) = await self._connect_sock()
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 26, in
connect_sock
return await asyncio.open_connection(address, port)
File "/usr/lib/python3.7/asyncio/streams.py", line 77, in open_connection
lambda: protocol, host, port, **kwds)
File "/usr/lib/python3.7/asyncio/base_events.py", line 959, in
create_connection
raise exceptions[0]
File "/usr/lib/python3.7/asyncio/base_events.py", line 946, in
create_connection
await self.sock_connect(sock, address)
File "/usr/lib/python3.7/asyncio/selector_events.py", line 464, in
sock_connect
return await fut
File "/usr/lib/python3.7/asyncio/selector_events.py", line 494, in
_sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('35.233.185.178', 18687)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/poky/bitbake/lib/bb/asyncrpc/serv.py", line 192, in
handle_client
await client.process_requests()
File "/work/poky/bitbake/lib/hashserv/server.py", line 200, in
process_requests
await super().process_requests()
File "/work/poky/bitbake/lib/bb/asyncrpc/serv.py", line 72, in
process_requests
await self.dispatch_message(d)
File "/work/poky/bitbake/lib/hashserv/server.py", line 210, in
dispatch_message
await self.handlers[k](msg[k])
File "/work/poky/bitbake/lib/hashserv/server.py", line 355, in
handle_get_stream
upstream = await self.upstream_client.get_unihash(method, taskhash)
File "/work/poky/bitbake/lib/hashserv/client.py", line 57, in get_unihash
await self._set_mode(self.MODE_GET_STREAM)
File "/work/poky/bitbake/lib/hashserv/client.py", line 46, in _set_mode
r = await self.send_message({"get-stream": None})
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 110, in
send_message
return await self._send_wrapper(proc)
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 68, in
_send_wrapper
raise ConnectionError(str(e))
ConnectionError: [Errno 110] Connect call failed ('35.233.185.178', 18687)
Error talking to server: Connection closed
Traceback (most recent call last):
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 57, in
_send_wrapper
await self.connect()
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 43, in connect
(self.reader, self.writer) = await self._connect_sock()
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 26, in
connect_sock
return await asyncio.open_connection(address, port)
File "/usr/lib/python3.7/asyncio/streams.py", line 77, in open_connection
lambda: protocol, host, port, **kwds)
File "/usr/lib/python3.7/asyncio/base_events.py", line 959, in
create_connection
raise exceptions[0]
File "/usr/lib/python3.7/asyncio/base_events.py", line 946, in
create_connection
await self.sock_connect(sock, address)
File "/usr/lib/python3.7/asyncio/selector_events.py", line 464, in
sock_connect
return await fut
File "/usr/lib/python3.7/asyncio/selector_events.py", line 494, in
_sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('35.233.185.178', 18687)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/poky/bitbake/lib/bb/asyncrpc/serv.py", line 192, in
handle_client
await client.process_requests()
File "/work/poky/bitbake/lib/hashserv/server.py", line 200, in
process_requests
await super().process_requests()
File "/work/poky/bitbake/lib/bb/asyncrpc/serv.py", line 72, in
process_requests
await self.dispatch_message(d)
File "/work/poky/bitbake/lib/hashserv/server.py", line 210, in
dispatch_message
await self.handlers[k](msg[k])
File "/work/poky/bitbake/lib/hashserv/server.py", line 355, in
handle_get_stream
upstream = await self.upstream_client.get_unihash(method, taskhash)
File "/work/poky/bitbake/lib/hashserv/client.py", line 57, in get_unihash
await self._set_mode(self.MODE_GET_STREAM)
File "/work/poky/bitbake/lib/hashserv/client.py", line 46, in _set_mode
r = await self.send_message({"get-stream": None})
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 110, in
send_message
return await self._send_wrapper(proc)
File "/work/poky/bitbake/lib/bb/asyncrpc/client.py", line 68, in
_send_wrapper
raise ConnectionError(str(e))
ConnectionError: [Errno 110] Connect call failed ('35.233.185.178', 18687)
Error talking to server: [Errno 110] Connect call failed ('35.233.185.178',
18687)
Jose Quaresma via lists.openembedded.org <quaresma.jose=
[email protected]> escreveu no dia quinta, 4/11/2021 à(s)
19:31:
>
>
> Jose Quaresma via lists.openembedded.org <quaresma.jose=
> [email protected]> escreveu no dia domingo, 17/10/2021
> à(s) 22:14:
>
>>
>>
>> Joshua Watt <[email protected]> escreveu no dia sábado, 16/10/2021
>> à(s) 20:15:
>>
>>>
>>>
>>> On Fri, Oct 15, 2021, 6:47 PM Jose Quaresma <[email protected]>
>>> wrote:
>>>
>>>> When the user specify an upstream hash equivalence server in
>>>> BB_HASHSERVE_UPSTREAM abort if we can't connect otherwise the
>>>> user will never be notified about the problem.
>>>>
>>>> Signed-off-by: Jose Quaresma <[email protected]>
>>>> ---
>>>> bitbake/lib/bb/cooker.py | 12 +++++++++++-
>>>> 1 file changed, 11 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>>>> index af794b4c42..34603c5ea2 100644
>>>> --- a/bitbake/lib/bb/cooker.py
>>>> +++ b/bitbake/lib/bb/cooker.py
>>>> @@ -388,12 +388,22 @@ class BBCooker:
>>>> # Create a new hash server bound to a unix domain socket
>>>> if not self.hashserv:
>>>> dbfile = (self.data.getVar("PERSISTENT_DIR") or
>>>> self.data.getVar("CACHE")) + "/hashserv.db"
>>>> + upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM")
>>>> or None
>>>> + if upstream:
>>>> + import socket
>>>> + try:
>>>> + sock =
>>>> socket.create_connection(upstream.split(":"), 5)
>>>> + sock.close()
>>>> + except socket.error as e:
>>>> + bb.fatal("BB_HASHSERVE_UPSTREAM is not valid,
>>>> unable to connect hash equivalence server at '%s': %s"
>>>> + % (upstream, repr(e)))
>>>> +
>>>>
>>>
>>> Perhaps this should just be a warning? The upstream missing should not
>>> be fatal for a given server, if it is we probably need to fix that
>>>
>>
> I have changed it to be a warning in V2
>
>
>> In my opinion this needs to be fatal or fixed with a different approach.
>>
>> If we launch bitbake in a clean build without build/cache with an invalid
>> server it will deadlock printing the follow message after several minutes:
>> WARNING: Error contacting Hash Equivalence Server
>> unix:///build/hashserve.sock: Connection closed
>>
>> The build/bitbake-cookerdaemon.log have all the connections errors as
>> well as exceptions.
>> This cooker log is a bit ugly imho and I will try to improved it in a
>> follow up patch.
>>
>>
>>> self.hashservaddr = "unix://%s/hashserve.sock" %
>>>> self.data.getVar("TOPDIR")
>>>> self.hashserv = hashserv.create_server(
>>>> self.hashservaddr,
>>>> dbfile,
>>>> sync=False,
>>>> - upstream=self.data.getVar("BB_HASHSERVE_UPSTREAM")
>>>> or None,
>>>> + upstream=upstream,
>>>> )
>>>> self.hashserv.serve_as_process()
>>>> self.data.setVar("BB_HASHSERVE", self.hashservaddr)
>>>> --
>>>> 2.33.1
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Best regards,
>>
>> José Quaresma
>>
>>
>>
>>
>
> --
> Best regards,
>
> José Quaresma
>
>
>
>
--
Best regards,
José Quaresma
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#157861):
https://lists.openembedded.org/g/openembedded-core/message/157861
Mute This Topic: https://lists.openembedded.org/mt/86362510/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
