Well.. the specific fix we need here *might* be backportable, although the full patch is not. I can look into it next week, if the issues continue.
On Fri, Jun 18, 2021 at 3:46 PM Daniel Alley <dal...@redhat.com> wrote: > Yup, that's the issue. I've seen all 3 of those errors before while I was > testing my fix for this issue (retriable downloads). The cdn.redhat.com > seems to be having a very bad day, today. > > It's a feature though, and not backportable, because the changes touch the > database and several different areas of code. My only suggestions are to > either reduce the download_concurrency further, use on_demand sync so that > it doesn't immediately download packages at all, or wait (because the CDN > issue is likely temporary). And upgrade once you get the chance (once > Katello packages pulpcore 3.14). > > On Fri, Jun 18, 2021 at 3:03 PM Bin Li (BLOOMBERG/ 120 PARK) < > bli...@bloomberg.net> wrote: > >> Still have errors on this repo >> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os >> >> >> Got 3 different errors for last 3 runs. >> Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR: >> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >> completed >> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:17:22 pulpserver rq: rv = job.perform() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:17:22 pulpserver rq: self._result = self._execute() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:17:22 pulpserver rq: dv.create() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:17:22 pulpserver rq: return future.result() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:17:22 pulpserver rq: await self.run() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:17:22 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:17:22 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:17:22 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 89, in _run >> Jun 18 14:17:22 pulpserver rq: to_return = await >> self._handle_response(response) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", >> line 180, in _handle_response >> Jun 18 14:17:22 pulpserver rq: chunk = await >> response.content.read(1048576) # 1 megabyte >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 368, in read >> Jun 18 14:17:22 pulpserver rq: await self._wait('read') >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 296, in _wait >> Jun 18 14:17:22 pulpserver rq: await waiter >> Jun 18 14:17:22 pulpserver rq: >> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >> completed >> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:17:22 pulpserver rq: rv = job.perform() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:17:22 pulpserver rq: self._result = self._execute() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:17:22 pulpserver rq: dv.create() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:17:22 pulpserver rq: return future.result() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:17:22 pulpserver rq: await self.run() >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:17:22 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:17:22 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:17:22 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 89, in _run >> Jun 18 14:17:22 pulpserver rq: to_return = await >> self._handle_response(response) >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", >> line 180, in _handle_response >> Jun 18 14:17:22 pulpserver rq: chunk = await >> response.content.read(1048576) # 1 megabyte >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 368, in read >> Jun 18 14:17:22 pulpserver rq: await self._wait('read') >> Jun 18 14:17:22 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 296, in _wait >> Jun 18 14:17:22 pulpserver rq: await waiter >> Jun 18 14:17:22 pulpserver rq: >> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >> completed >> ================================== >> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 936, in _wrap_create_connection >> Jun 18 14:34:09 pulpserver rq: return await >> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in >> create_connection >> Jun 18 14:34:09 pulpserver rq: >> ssl_handshake_timeout=ssl_handshake_timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in >> _create_connection_transport >> Jun 18 14:34:09 pulpserver rq: await waiter >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in >> data_received >> Jun 18 14:34:09 pulpserver rq: ssldata, appdata = >> self._sslpipe.feed_ssldata(data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in >> feed_ssldata >> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake >> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: >> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056) >> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause >> of the following exception: >> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:34:09 pulpserver rq: rv = job.perform() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:34:09 pulpserver rq: self._result = self._execute() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:34:09 pulpserver rq: dv.create() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:34:09 pulpserver rq: return future.result() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:34:09 pulpserver rq: await self.run() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:34:09 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:34:09 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:34:09 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 87, in _run >> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, >> proxy=self.proxy, auth=self.auth) as response: >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 1012, in __aenter__ >> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 483, in _request >> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 523, in connect >> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req, >> traces, timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 856, in _create_connection >> Jun 18 14:34:09 pulpserver rq: req, traces, timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 1097, in _create_proxy_connection >> Jun 18 14:34:09 pulpserver rq: req=req) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 941, in _wrap_create_connection >> Jun 18 14:34:09 pulpserver rq: raise >> ClientConnectorSSLError(req.connection_key, exc) from exc >> Jun 18 14:34:09 pulpserver rq: >> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host >> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] >> sslv3 alert unexpected message (_ssl.c:1056)] >> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 936, in _wrap_create_connection >> Jun 18 14:34:09 pulpserver rq: return await >> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in >> create_connection >> Jun 18 14:34:09 pulpserver rq: >> ssl_handshake_timeout=ssl_handshake_timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in >> _create_connection_transport >> Jun 18 14:34:09 pulpserver rq: await waiter >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in >> data_received >> Jun 18 14:34:09 pulpserver rq: ssldata, appdata = >> self._sslpipe.feed_ssldata(data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in >> feed_ssldata >> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake >> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: >> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056) >> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause >> of the following exception: >> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:34:09 pulpserver rq: rv = job.perform() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:34:09 pulpserver rq: self._result = self._execute() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:34:09 pulpserver rq: dv.create() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:34:09 pulpserver rq: return future.result() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:34:09 pulpserver rq: await self.run() >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:34:09 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:34:09 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:34:09 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 87, in _run >> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, >> proxy=self.proxy, auth=self.auth) as response: >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 1012, in __aenter__ >> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 483, in _request >> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 523, in connect >> Jun 18 14:34:09 pulpserver rq: proto = await self._create_connection(req, >> traces, timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 856, in _create_connection >> Jun 18 14:34:09 pulpserver rq: req, traces, timeout) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 1097, in _create_proxy_connection >> Jun 18 14:34:09 pulpserver rq: req=req) >> Jun 18 14:34:09 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >> line 941, in _wrap_create_connection >> Jun 18 14:34:09 pulpserver rq: raise >> ClientConnectorSSLError(req.connection_key, exc) from exc >> Jun 18 14:34:09 pulpserver rq: >> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host >> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] >> sslv3 alert unexpected message (_ssl.c:1056)] >> >> ================================= >> Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR: >> aiohttp.client_exceptions.ServerDisconnectedError >> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:47:28 pulpserver rq: rv = job.perform() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:47:28 pulpserver rq: self._result = self._execute() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:47:28 pulpserver rq: dv.create() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:47:28 pulpserver rq: return future.result() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:47:28 pulpserver rq: await self.run() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:47:28 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:47:28 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:47:28 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 87, in _run >> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, >> proxy=self.proxy, auth=self.auth) as response: >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 1012, in __aenter__ >> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 504, in _request >> Jun 18 14:47:28 pulpserver rq: await resp.start(conn) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", >> line 847, in start >> Jun 18 14:47:28 pulpserver rq: message, payload = await >> self._protocol.read() # type: ignore # noqa >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 591, in read >> Jun 18 14:47:28 pulpserver rq: await self._waiter >> Jun 18 14:47:28 pulpserver rq: >> aiohttp.client_exceptions.ServerDisconnectedError >> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last): >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >> line 886, in perform_job >> Jun 18 14:47:28 pulpserver rq: rv = job.perform() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 664, in perform >> Jun 18 14:47:28 pulpserver rq: self._result = self._execute() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >> 670, in _execute >> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, **self.kwargs) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >> line 266, in synchronize >> Jun 18 14:47:28 pulpserver rq: dv.create() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >> line 148, in create >> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >> run_until_complete >> Jun 18 14:47:28 pulpserver rq: return future.result() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 225, in create_pipeline >> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >> line 43, in __call__ >> Jun 18 14:47:28 pulpserver rq: await self.run() >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 152, in run >> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >> line 178, in _handle_content_unit >> Jun 18 14:47:28 pulpserver rq: await >> asyncio.gather(*downloaders_for_content) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >> line 88, in download >> Jun 18 14:47:28 pulpserver rq: download_result = await >> downloader.run(extra_data=self.extra_data) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >> line 227, in run >> Jun 18 14:47:28 pulpserver rq: return await >> self._run(extra_data=extra_data) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >> line 87, in _run >> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, >> proxy=self.proxy, auth=self.auth) as response: >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 1012, in __aenter__ >> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >> line 504, in _request >> Jun 18 14:47:28 pulpserver rq: await resp.start(conn) >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", >> line 847, in start >> Jun 18 14:47:28 pulpserver rq: message, payload = await >> self._protocol.read() # type: ignore # noqa >> Jun 18 14:47:28 pulpserver rq: File >> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >> line 591, in read >> Jun 18 14:47:28 pulpserver rq: await self._waiter >> Jun 18 14:47:28 pulpserver rq: >> aiohttp.client_exceptions.ServerDisconnectedError >> >> >> The aarch64 baseos repo completed successfully. >> >> >> >> >> From: dal...@redhat.com At: 06/18/21 10:55:56 UTC-4:00 >> To: dkli...@redhat.com >> Cc: Bin Li (BLOOMBERG/ 120 PARK ) <bli...@bloomberg.net>, >> pulp-list@redhat.com >> Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo >> >> replace "but only with the errors" with "but only with the normal types >> of errors" >> >> On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dal...@redhat.com> wrote: >> >>> @Bin Li <bli...@bloomberg.net> >>> Could you post the error that was logged? I think this traceback is >>> missing some of the relevant details. >>> >>> I can tell you what I expect that problem *might* be, because we've been >>> hearing about it a lot recently. Again, this is just a guess, without more >>> information. >>> >>> We have found that some content delivery networks (including the Red Hat >>> one) start returning seemingly irrelevant error messages when they get >>> 'angry' about too much traffic coming from one source. Rather than >>> returning HTTP 429 or some other standard response, sometimes it will just >>> terminate the connection, or give you SSL errors, or return an incomplete >>> payload. This might be some type of DDoS prevention / flood control >>> mechanism, I'm not entirely sure. The mitigation is to either reduce the >>> "download_concurrency" on your remote from 10 to 5, or using "on_demand" >>> syncs rather than "immediate". Both of those are just to lessen the load on >>> the CDN and make it less likely to start dropping connections. >>> >>> But the "proper" fix is being worked on for pulpcore 3.14. We want to >>> automatically retry downloads that fail using a backoff algorithm to >>> prevent the entire sync from failing. We were already doing this, but only >>> with the errors, rather than the weird ones that seem to come up in >>> practice. And we're making the number of retry attempts configurable. >>> >>> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkli...@redhat.com> >>> wrote: >>> >>>> I don't see anything special about these repos. Are you still >>>> experiencing this problem? >>>> >>>> >>>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) < >>>> bli...@bloomberg.net> wrote: >>>> >>>>> Tried to download aarch64 repo and got the errors below: >>>>> >>>>> "description": "", >>>>> "traceback": " File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", >>>>> line 886, in perform_job\n rv = job.perform()\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>>>> line 664, in perform\n self._result = self._execute()\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", >>>>> line 266, in synchronize\n dv.create()\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", >>>>> line 148, in create\n loop.run_until_complete(pipeline)\n File >>>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in >>>>> run_until_complete\n return future.result()\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>>>> line 43, in __call__\n await self.run()\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>>>> line 152, in run\n pb.done += task.result() # download_count\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>>>> line 178, in _handle_content_unit\n await >>>>> asyncio.gather(*downloaders_for_content)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\", >>>>> line 88, in download\n download_result = await >>>>> downloader.run(extra_data=self.extra_data)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\", >>>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\", >>>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy, >>>>> auth=self.auth) as response:\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>>>> line 1012, in __aenter__\n self._resp = await self._coro\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>>>> line 504, in _request\n await resp.start(conn)\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\", >>>>> line 847, in start\n message, payload = await self._protocol.read() # >>>>> type: >>>>> ignore # noqa\n File >>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\", >>>>> line 591, in read\n await self._waiter\n >>>>> >>>>> The same error happened on both remote >>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and >>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os >>>>> >>>>> Please advise. >>>>> >>>>> Thanks >>>>> _______________________________________________ >>>>> Pulp-list mailing list >>>>> Pulp-list@redhat.com >>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>> >>>> _______________________________________________ >>>> Pulp-list mailing list >>>> Pulp-list@redhat.com >>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>> >>> >>
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-list
