Please file an issue at https://pulp.plan.io/issues/new/ and we will try to reproduce the issue.
On Thu, Jun 27, 2019, 5:50 PM Bin Li (BLOOMBERG/ 120 PARK) < [email protected]> wrote: > Tried again for both rhel7 and rhel6 repo. The download_concurrency is > always set to "1". We tried to sync rhel6 repo first and task was > completed. Rerun sync on rhel7 repo failed again. Our network team didn't > find any issues on the network and proxy. > > This is our remote setting: > { > "_created": "2019-06-21T16:10:05.493082Z", > "_href": > "/pulp/api/v3/remotes/rpm/rpm/b308a912-1dc9-46cb-b340-450a96e5ee93/", > "_last_updated": "2019-06-21T16:10:05.493107Z", > "_type": "rpm.rpm", > "download_concurrency": 1, > "name": "rhel-x86_64-server-7", > "policy": "immediate", > "proxy_url": "http://proxyhost.bloomberg.com:8000";, > "ssl_ca_certificate": > "c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21", > "ssl_client_certificate": > "c5c5dfac3e16e93b394af137ebf8f0c5eaa37277699e72dc8042a53f6133b999", > "ssl_client_key": > "3f8bde4c2baf94196a9e40d0834f86a2890b3df08053eef9283b3c136aae487f", > "ssl_validation": true, > "url": " > https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/";, > "validate": true > } > > Task status: > > Allow: GET, PATCH, DELETE, HEAD, OPTIONS > Connection: close > Content-Length: 3515 > Content-Type: application/json > Date: Thu, 27 Jun 2019 21:32:27 GMT > Server: gunicorn/19.9.0 > Vary: Accept, Cookie > X-Frame-Options: SAMEORIGIN > > { > "_created": "2019-06-27T18:15:28.484561Z", > "_href": "/pulp/api/v3/tasks/72da60f0-1beb-4447-b8d2-718a40b5fccc/", > "created_resources": [], > "error": { > "code": null, > "description": "Timeout on reading data from socket", > "traceback": " File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, > in perform_job\n rv = job.perform()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in > perform\n self._result = self._execute()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in > _execute\n return self.func(*self.args, **self.kwargs)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", > line 68, in synchronize\n dv.create()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", > line 169, in create\n loop.run_until_complete(pipeline)\n File > \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in > run_until_complete\n return future.result()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", > line 209, in create_pipeline\n await asyncio.gather(*futures)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", > line 43, in __call__\n await self.run()\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", > line 132, in run\n pb.done += task.result() # download_count\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", > line 155, in _handle_content_unit\n await > asyncio.gather(*downloaders_for_content)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", > line 78, in download\n download_result = await > downloader.run(extra_data=self.extra_data)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", > line 212, in run\n return await self._run(extra_data=extra_data)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line > 131, in retry\n ret = await target(*args, **kwargs)\n File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", > line 184, in _run\n to_return = await self._handle_response(response)\n > File > \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", > line 158, in _handle_response\n chunk = await > response.content.read(1048576) # 1 megabyte\n File > \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", > line 369, in read\n await self._wait('read')\n File > \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", > line 297, in _wait\n await waiter\n" > }, > "finished_at": "2019-06-27T21:19:18.675004Z", > "name": "pulp_rpm.app.tasks.synchronizing.synchronize", > "non_fatal_errors": [], > "parent": null, > "progress_reports": [ > { > "done": 20201, > "message": "Parsed Packages", > "state": "running", > "suffix": null, > "total": 24397 > }, > { > "done": 11966, > "message": "Downloading Artifacts", > "state": "failed", > "suffix": null, > "total": 11966 > }, > { > "done": 5, > "message": "Downloading Metadata Files", > "state": "canceled", > "suffix": null, > "total": 5 > }, > { > "done": 23300, > "message": "Associating Content", > "state": "canceled", > "suffix": null, > "total": 23300 > }, > { > "done": 3583, > "message": "Parsed Erratum", > "state": "running", > "suffix": null, > "total": 3583 > } > ], > "spawned_tasks": [], > "started_at": "2019-06-27T18:15:28.618096Z", > "state": "failed", > "worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/" > } > > From: [email protected] At: 06/27/19 10:54:05 > To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> > Cc: [email protected], [email protected] > Subject: Re: [Pulp-list] create remote to redhat on pulp 3 > > It says "Timeout on reading data from socket" so either this is a bug, or > the remote server is not delivering you data over the socket connection, > perhaps due to some rate limiting. The remote has a download_concurrency > option [0] so perhaps reducing that will cause the server to give you all > of the data. You can also open an issue with reproducer details and we can > try to reproduce it. > > [0]: > https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create > > On Thu, Jun 27, 2019 at 10:48 AM Bin Li (BLOOMBERG/ 120 PARK) < > [email protected]> wrote: > >> I am now able to sync the redhat repo through proxy but the task failed >> eventually on a timeout after sync most of contents. I tried sync both >> rhel7 and rhel6 os repo and both gave the same error. Any idea what could >> cause this? >> >> >> # ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/ >> HTTP/1.1 200 OK >> Allow: GET, PATCH, DELETE, HEAD, OPTIONS >> Connection: close >> Content-Length: 3512 >> Content-Type: application/json >> Date: Thu, 27 Jun 2019 14:38:12 GMT >> Server: gunicorn/19.9.0 >> Vary: Accept, Cookie >> X-Frame-Options: SAMEORIGIN >> >> { >> "_created": "2019-06-26T20:35:38.514762Z", >> "_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/", >> "created_resources": [], >> "error": { >> "code": null, >> "description": "Timeout on reading data from socket", >> "traceback": " File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, >> in perform_job\n rv = job.perform()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in >> perform\n self._result = self._execute()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in >> _execute\n return self.func(*self.args, **self.kwargs)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", >> line 68, in synchronize\n dv.create()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", >> line 169, in create\n loop.run_until_complete(pipeline)\n File >> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in >> run_until_complete\n return future.result()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", >> line 209, in create_pipeline\n await asyncio.gather(*futures)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", >> line 43, in __call__\n await self.run()\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >> line 132, in run\n pb.done += task.result() # download_count\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >> line 155, in _handle_content_unit\n await >> asyncio.gather(*downloaders_for_content)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", >> line 78, in download\n download_result = await >> downloader.run(extra_data=self.extra_data)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\", >> line 212, in run\n return await self._run(extra_data=extra_data)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line >> 131, in retry\n ret = await target(*args, **kwargs)\n File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", >> line 184, in _run\n to_return = await self._handle_response(response)\n >> File >> \"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\", >> line 158, in _handle_response\n chunk = await >> response.content.read(1048576) # 1 megabyte\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", >> line 369, in read\n await self._wait('read')\n File >> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", >> line 297, in _wait\n await waiter\n" >> }, >> "finished_at": "2019-06-26T21:57:33.499085Z", >> "name": "pulp_rpm.app.tasks.synchronizing.synchronize", >> "non_fatal_errors": [], >> "parent": null, >> "progress_reports": [ >> { >> "done": 3757, >> "message": "Downloading Artifacts", >> "state": "failed", >> "suffix": null, >> "total": 3757 >> }, >> { >> "done": 5, >> "message": "Downloading Metadata Files", >> "state": "canceled", >> "suffix": null, >> "total": 5 >> }, >> { >> "done": 11350, >> "message": "Associating Content", >> "state": "canceled", >> "suffix": null, >> "total": 11350 >> }, >> { >> "done": 3582, >> "message": "Parsed Erratum", >> "state": "running", >> "suffix": null, >> "total": 3582 >> }, >> { >> "done": 8201, >> "message": "Parsed Packages", >> "state": "running", >> "suffix": null, >> "total": 24392 >> } >> ], >> "spawned_tasks": [], >> "started_at": "2019-06-26T20:35:38.650635Z", >> "state": "failed", >> "worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/" >> } >> >> >> >> From: [email protected] At: 06/23/19 07:14:50 >> To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> >> Cc: [email protected] >> Subject: Re: [Pulp-list] create remote to redhat on pulp 3 >> >> The problem is that the proxy setting is not being respected by Pulp. I >> filed an issue about this[0] and also made a patch to fix it[1]. This patch >> should be merged in the next 24 hours. >> >> If you cannot access cdn.redhat.com without a proxy, the error you got >> makes sense. I was only able to reproduce the error message when I added an >> incorrect entry for cdn.redhat.com in /etc/hosts. >> >> >> [0] https://pulp.plan.io/issues/5011 >> [1] https://github.com/pulp/pulpcore-plugin/pull/107 >> >> On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK) < >> [email protected]> wrote: >> >>> Thanks. I am now getting a different error. Please advise >>> >>> "_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/", >>> "created_resources": [], >>> "error": { >>> "code": null, >>> "description": "Cannot connect to host cdn.redhat.com:443 ssl:None >>> [Connect call failed ('23.65.16.251', 443)]", >>> "traceback": " File >>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, >>> in perform_job\n rv = job.perform()\n File >>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in >>> perform\n self._result = self._execute()\n File >>> \"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in >>> _execute\n return self.func(*self.args, **self.kwargs)\n File >>> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in >>> synchronize\n dv.create()\n File >>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", >>> line 169, in create\n loop.run_until_complete(pipeline)\n File >>> \"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in >>> run_until_complete\n return future.result()\n File >>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in >>> create_pipeline\n await asyncio.gather(*futures)\n File >>> \"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in >>> __call__\n await self.run()\n File >>> \"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n >>> result = await downloader.run()\n File >>> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in >>> run\n return await self._run(extra_data=extra_data)\n File >>> \"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line >>> 131, in retry\n ret = await target(*args, **kwargs)\n File >>> \"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in >>> _run\n async with self.session.get(self.url) as response:\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line >>> 1005, in __aenter__\n self._resp = await self._coro\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line >>> 476, in _request\n timeout=real_timeout\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >>> line 522, in connect\n proto = await self._create_connection(req, traces, >>> timeout)\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >>> line 854, in _create_connection\n req, traces, timeout)\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >>> line 992, in _create_direct_connection\n raise last_exc\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >>> line 974, in _create_direct_connection\n req=req, >>> client_error=client_error)\n File >>> \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", >>> line 931, in _wrap_create_connection\n raise >>> client_error(req.connection_key, exc) from exc\n" >>> }, >>> >>> >>> I was able to connect using client cert through proxy >>> # curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key >>> /etc/pki/entitlement/8521692907269500331-key.pem --cert >>> /etc/pki/entitlement/8521692907269500331.pem >>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/ >>> * About to connect() to proxy ... port ... (#0) >>> * Trying ... >>> * Connected to ... () port ... (#0) >>> * Establish HTTP proxy tunnel to cdn.redhat.com:443 >>> > CONNECT cdn.redhat.com:443 HTTP/1.1 >>> > Host: cdn.redhat.com:443 >>> > User-Agent: curl/7.29.0 >>> > Proxy-Connection: Keep-Alive >>> > >>> < HTTP/1.1 200 Connection established >>> HTTP/1.1 200 Connection established >>> < >>> >>> * Proxy replied OK to CONNECT request >>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >>> >>> >>> From: [email protected] At: 06/20/19 16:34:40 >>> To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> >>> Cc: [email protected] >>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3 >>> >>> You need to make sure that the following services are running: >>> pulp-resource-manager, pulp-worker@0, pulp-worker@1 (at least 1 worker >>> is needed). >>> >>> On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK) < >>> [email protected]> wrote: >>> >>>> I installed the latest version of component. No more errors when try to >>>> sync the repo but the task is in waiting state forever. Any idea what I >>>> should check? >>>> >>>> "versions": [ >>>> { >>>> "component": "pulpcore", >>>> "version": "3.0.0rc3.dev0" >>>> }, >>>> { >>>> "component": "pulpcore-plugin", >>>> "version": "0.1.0rc3.dev0" >>>> }, >>>> { >>>> "component": "pulp_rpm", >>>> "version": "3.0.0b4.dev0" >>>> } >>>> >>>> >>>> >>>> From: [email protected] At: 06/18/19 13:42:01 >>>> To: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]> >>>> Cc: [email protected] >>>> Subject: Re: [Pulp-list] create remote to redhat on pulp 3 >>>> >>>> This issue has now been resolved and a fix is available on master >>>> branch of pulpcore-plugin[0]. >>>> >>>> >>>> [0] https://github.com/pulp/pulpcore-plugin/tree/master/ >>>> >>>> On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <[email protected]> >>>> wrote: >>>> >>>>> There is an open issue about this[0]. I recommend clicking the watch >>>>> button so you get notifications about it. We should be fixing this soon. >>>>> >>>>> [0] https://pulp.plan.io/issues/4825 >>>>> >>>>> On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK) < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> I was not able to sync a protected repo from redhat. The host running >>>>>> pulp 3 subscribes to redhat directly. >>>>>> >>>>>> I first tried to use the following to create remote: >>>>>> { >>>>>> "name": "rhel-x86_64-server-7", >>>>>> "url": " >>>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";, >>>>>> "ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem", >>>>>> "ssl_client_certificate": "/etc/pki/consumer/cert.pem", >>>>>> "ssl_client_key": "/etc/pki/consumer/key.pem", >>>>>> "ssl_validation": true, >>>>>> "proxy_url": "http://proxy:80";, >>>>>> "download_concurrency": 1, >>>>>> "policy": "immediate" >>>>>> } >>>>>> >>>>>> Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244 >>>>>> rq[19389]: pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no >>>>>> start line (_ssl.c:3626)" when I tried to sync with http POST >>>>>> :24817${REMOTE_HREF}sync/ repository=$REPO_HREF >>>>>> >>>>>> Then I tried to converted the pem file to a json format >>>>>> { >>>>>> "name": "rhel-x86_64-server-7", >>>>>> "url": " >>>>>> https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";, >>>>>> "ssl_ca_certificate": "-----BEGIN >>>>>> CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg >>>>>> ... >>>>>> "ssl_client_certificate": "-----BEGIN >>>>>> CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ >>>>>> ... >>>>>> "ssl_client_key": "-----BEGIN RSA PRIVATE >>>>>> KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee >>>>>> ... >>>>>> "ssl_validation": true, >>>>>> "proxy_url": "http://proxy:80";, >>>>>> "download_concurrency": 1, >>>>>> "policy": "immediate" >>>>>> } >>>>>> >>>>>> I got this error when I tried to sync repo with the remote: >>>>>> Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM >>>>>> lib (_ssl.c:3503) >>>>>> >>>>>> >>>>>> What is the correct json input for creating a remote to redhat? How >>>>>> do I troubleshoot the sync issue? Thanks for your help. >>>>>> _______________________________________________ >>>>>> Pulp-list mailing list >>>>>> [email protected] >>>>>> https://www.redhat.com/mailman/listinfo/pulp-list >>>>> >>>>> >>>> >>> >> _______________________________________________ >> Pulp-list mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/pulp-list > > >
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
