replace "but only with the errors" with "but only with the normal types of errors"
On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <dal...@redhat.com> wrote: > @Bin Li <bli...@bloomberg.net> > Could you post the error that was logged? I think this traceback is > missing some of the relevant details. > > I can tell you what I expect that problem *might* be, because we've been > hearing about it a lot recently. Again, this is just a guess, without more > information. > > We have found that some content delivery networks (including the Red Hat > one) start returning seemingly irrelevant error messages when they get > 'angry' about too much traffic coming from one source. Rather than > returning HTTP 429 or some other standard response, sometimes it will just > terminate the connection, or give you SSL errors, or return an incomplete > payload. This might be some type of DDoS prevention / flood control > mechanism, I'm not entirely sure. The mitigation is to either reduce the > "download_concurrency" on your remote from 10 to 5, or using "on_demand" > syncs rather than "immediate". Both of those are just to lessen the load on > the CDN and make it less likely to start dropping connections. > > But the "proper" fix is being worked on for pulpcore 3.14. We want to > automatically retry downloads that fail using a backoff algorithm to > prevent the entire sync from failing. We were already doing this, but only > with the errors, rather than the weird ones that seem to come up in > practice. And we're making the number of retry attempts configurable. > > On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <dkli...@redhat.com> wrote: > >> I don't see anything special about these repos. Are you still >> experiencing this problem? >> >> >> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) < >> bli...@bloomberg.net> wrote: >> >>> Tried to download aarch64 repo and got the errors below: >>> >>> "description": "", >>> "traceback": " File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", >>> line 886, in perform_job\n rv = job.perform()\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>> line 664, in perform\n self._result = self._execute()\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", >>> line 266, in synchronize\n dv.create()\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", >>> line 148, in create\n loop.run_until_complete(pipeline)\n File >>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, in >>> run_until_complete\n return future.result()\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>> line 43, in __call__\n await self.run()\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>> line 152, in run\n pb.done += task.result() # download_count\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>> line 178, in _handle_content_unit\n await >>> asyncio.gather(*downloaders_for_content)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\", >>> line 88, in download\n download_result = await >>> downloader.run(extra_data=self.extra_data)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\", >>> line 227, in run\n return await self._run(extra_data=extra_data)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\", >>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy, >>> auth=self.auth) as response:\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>> line 1012, in __aenter__\n self._resp = await self._coro\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>> line 504, in _request\n await resp.start(conn)\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\", >>> line 847, in start\n message, payload = await self._protocol.read() # type: >>> ignore # noqa\n File >>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\", >>> line 591, in read\n await self._waiter\n >>> >>> The same error happened on both remote >>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and >>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os >>> >>> Please advise. >>> >>> Thanks >>> _______________________________________________ >>> Pulp-list mailing list >>> Pulp-list@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-list >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://listman.redhat.com/mailman/listinfo/pulp-list > >
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-list
