It says "Timeout on reading data from socket" so either this is a bug, or the remote server is not delivering you data over the socket connection, perhaps due to some rate limiting. The remote has a download_concurrency option [0] so perhaps reducing that will cause the server to give you all of the data. You can also open an issue with reproducer details and we can try to reproduce it.
[0]: https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create On Thu, Jun 27, 2019 at 10:48 AM Bin Li (BLOOMBERG/ 120 PARK) < [email protected]> wrote: > I am now able to sync the redhat repo through proxy but the task failed > eventually on a timeout after sync most of contents. I tried sync both > rhel7 and rhel6 os repo and both gave the same error. Any idea what could > cause this? > > > # ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/ > HTTP/1.1 200 OK > Allow: GET, PATCH, DELETE, HEAD, OPTIONS > Connection: close > Content-Length: 3512 > Content-Type: application/json > Date: Thu, 27 Jun 2019 14:38:12 GMT > Server: gunicorn/19.9.0 > Vary: Accept, Cookie > X-Frame-Options: SAMEORIGIN > > { > "_created": "2019-06-26T20:35:38.514762Z", > "_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/", > "created_resources": [], > "error": { > "code": null, > "description": "Timeout on reading data from socket", > "traceback": " File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, > in perform_job\n rv = job.perform()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in > perform\n self._result = self._execute()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in > _execute\n return self.func(*self.args, **self.kwargs)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", > line 68, in synchronize\n dv.create()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", > line 169, in create\n loop.run_until_complete(pipeline)\n File > \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in > run_until_complete\n return future.result()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", > line 209, in create_pipeline\n await asyncio.gather(*futures)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", > line 43, in __call__\n await self.run()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", > line 132, in run\n pb.done += task.result() # download_count\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", > line 155, in _handle_content_unit\n await > asyncio.gather(*downloaders_for_content)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", > line 78, in download\n download_result = await > downloader.run(extra_data=self.extra_data)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", > line 212, in run\n return await self._run(extra_data=extra_data)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line > 131, in retry\n ret = await target(*args, **kwargs)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", > line 184, in _run\n to_return = await self._handle_response(response)\n > File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", > line 158, in _handle_response\n chunk = await > response.content.read(1048576) # 1 megabyte\n File > \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", > line 369, in read\n await self._wait('read')\n File > \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", > line 297, in _wait\n await waiter\n" > }, > "finished_at": "2019-06-26T21:57:33.499085Z", > "name": "pulp_rpm.app.tasks.synchronizing.synchronize", > "non_fatal_errors": [], > "parent": null, > "progress_reports": [ > { > "done": 3757, > "message": "Downloading Artifacts", > "state": "failed", > "suffix": null, > "total": 3757 > }, > { > "done": 5, > "message": "Downloading Metadata Files", > "state": "canceled", > "suffix": null, > "total": 5 > }, > { > "done": 11350, > "message": "Associating Content", > "state": "canceled", > "suffix": null, > "total": 11350 > }, > { > "done": 3582, > "message": "Parsed Erratum", > "state": "running", > "suffix": null, > "total": 3582 > }, > { > "done": 8201, > "message": "Parsed Packages", > "state": "running", > "suffix": null, > "total": 24392 > } > ], > "spawned_tasks": [], > "started_at": "2019-06-26T20:35:38.650635Z", > "state": "failed", > "worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/" > } > > > > From: [email protected] At: 06/23/19 07:14:50 > To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> > Cc: [email protected] > Subject: Re: [Pulp-list] create remote to redhat on pulp 3 > > The problem is that the proxy setting is not being respected by Pulp. I > filed an issue about this[0] and also made a patch to fix it[1]. This patch > should be merged in the next 24 hours. > > If you cannot access cdn.redhat.com without a proxy, the error you got > makes sense. I was only able to reproduce the error message when I added an > incorrect entry for cdn.redhat.com in /etc/hosts. > > > [0] https://pulp.plan.io/issues/5011 > [1] https://github.com/pulp/pulpcore-plugin/pull/107 > > On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK) < > [email protected]> wrote: > >> Thanks. I am now getting a different error. Please advise >> >> "_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/", >> "created_resources": [], >> "error": { >> "code": null, >> "description": "Cannot connect to host cdn.redhat.com:443 ssl:None >> [Connect call failed ('23.65.16.251', 443)]", >> "traceback": " File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, >> in perform_job\n rv = job.perform()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in >> perform\n self._result = self._execute()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in >> _execute\n return self.func(*self.args, **self.kwargs)\n File >> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in >> synchronize\n dv.create()\n File >> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", >> line 169, in create\n loop.run_until_complete(pipeline)\n File >> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in >> run_until_complete\n return future.result()\n File >> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in >> create_pipeline\n await asyncio.gather(*futures)\n File >> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in >> __call__\n await self.run()\n File >> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n >> result = await downloader.run()\n File >> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in >> run\n return await self._run(extra_data=extra_data)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line >> 131, in retry\n ret = await target(*args, **kwargs)\n File >> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in >> _run\n async with self.session.get(self.url) as response:\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line >> 1005, in __aenter__\n self._resp = await self._coro\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line >> 476, in _request\n timeout=real_timeout\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >> line 522, in connect\n proto = await self._create_connection(req, traces, >> timeout)\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >> line 854, in _create_connection\n req, traces, timeout)\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >> line 992, in _create_direct_connection\n raise last_exc\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >> line 974, in _create_direct_connection\n req=req, >> client_error=client_error)\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >> line 931, in _wrap_create_connection\n raise >> client_error(req.connection_key, exc) from exc\n" >> }, >> >> >> I was able to connect using client cert through proxy >> # curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key >> /etc/pki/entitlement/8521692907269500331-key.pem --cert >> /etc/pki/entitlement/8521692907269500331.pem >> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/ >> * About to connect() to proxy ... port ... (#0) >> * Trying ... >> * Connected to ... () port ... (#0) >> * Establish HTTP proxy tunnel to cdn.redhat.com:443 >> > CONNECT cdn.redhat.com:443 HTTP/1.1 >> > Host: cdn.redhat.com:443 >> > User-Agent: curl/7.29.0 >> > Proxy-Connection: Keep-Alive >> > >> < HTTP/1.1 200 Connection established >> HTTP/1.1 200 Connection established >> < >> >> * Proxy replied OK to CONNECT request >> * Initializing NSS with certpath: sql:/etc/pki/nssdb >> >> >> From: [email protected] At: 06/20/19 16:34:40 >> To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> >> Cc: [email protected] >> Subject: Re: [Pulp-list] create remote to redhat on pulp 3 >> >> You need to make sure that the following services are running: >> pulp-resource-manager, pulp-worker@0, pulp-worker@1 (at least 1 worker >> is needed). >> >> On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK) < >> [email protected]> wrote: >> >>> I installed the latest version of component. No more errors when try to >>> sync the repo but the task is in waiting state forever. Any idea what I >>> should check? >>> >>> "versions": [ >>> { >>> "component": "pulpcore", >>> "version": "3.0.0rc3.dev0" >>> }, >>> { >>> "component": "pulpcore-plugin", >>> "version": "0.1.0rc3.dev0" >>> }, >>> { >>> "component": "pulp_rpm", >>> "version": "3.0.0b4.dev0" >>> } >>> >>> >>> >>> From: [email protected] At: 06/18/19 13:42:01 >>> To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> >>> Cc: [email protected] >>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3 >>> >>> This issue has now been resolved and a fix is available on master branch >>> of pulpcore-plugin[0]. >>> >>> >>> [0] https://github.com/pulp/pulpcore-plugin/tree/master/ >>> >>> On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <[email protected]> >>> wrote: >>> >>>> There is an open issue about this[0]. I recommend clicking the watch >>>> button so you get notifications about it. We should be fixing this soon. >>>> >>>> [0] https://pulp.plan.io/issues/4825 >>>> >>>> On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK) < >>>> [email protected]> wrote: >>>> >>>>> >>>>> I was not able to sync a protected repo from redhat. The host running >>>>> pulp 3 subscribes to redhat directly. >>>>> >>>>> I first tried to use the following to create remote: >>>>> { >>>>> "name": "rhel-x86_64-server-7", >>>>> "url": " >>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";, >>>>> "ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem", >>>>> "ssl_client_certificate": "/etc/pki/consumer/cert.pem", >>>>> "ssl_client_key": "/etc/pki/consumer/key.pem", >>>>> "ssl_validation": true, >>>>> "proxy_url": "http://proxy:80";, >>>>> "download_concurrency": 1, >>>>> "policy": "immediate" >>>>> } >>>>> >>>>> Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244 >>>>> rq[19389]: pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no >>>>> start line (_ssl.c:3626)" when I tried to sync with http POST >>>>> :24817${REMOTE_HREF}sync/ repository=$REPO_HREF >>>>> >>>>> Then I tried to converted the pem file to a json format >>>>> { >>>>> "name": "rhel-x86_64-server-7", >>>>> "url": " >>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";, >>>>> "ssl_ca_certificate": "-----BEGIN >>>>> CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg >>>>> ... >>>>> "ssl_client_certificate": "-----BEGIN >>>>> CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ >>>>> ... >>>>> "ssl_client_key": "-----BEGIN RSA PRIVATE >>>>> KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee >>>>> ... >>>>> "ssl_validation": true, >>>>> "proxy_url": "http://proxy:80";, >>>>> "download_concurrency": 1, >>>>> "policy": "immediate" >>>>> } >>>>> >>>>> I got this error when I tried to sync repo with the remote: >>>>> Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM lib >>>>> (_ssl.c:3503) >>>>> >>>>> >>>>> What is the correct json input for creating a remote to redhat? How do >>>>> I troubleshoot the sync issue? Thanks for your help. >>>>> _______________________________________________ >>>>> Pulp-list mailing list >>>>> [email protected] >>>>> https://www.redhat.com/mailman/listinfo/pulp-list >>>> >>>> >>> >> > _______________________________________________ > Pulp-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
