Hello pulp-list, I have just observed a repeatable pulp sync failure of an ISO repo, where curl managed to fetch the PULP_MANIFEST successfully.
The pulp sync failure reason appears to be the following log message: May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR: Failed to download https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST: ('Connection aborted.', error(110, 'Connection timed out')). The timeout occurs 2 min after the previous log message. The following list summarises all the gory details that can be found at the end of this email. * pulp-admin iso repo create * pulp-admin iso repo sync * pulp.log containing failure * successful curl command * pulp rpm versions I am hoping that someone will be able to help me to sync the repo successfully, or reproduce the problem and initiate a fix. Thanks, Ben Stanley. =========== pulp-admin iso repo create =================================== [root@bumblebee conf.d]# pulp-admin iso repo create \ --repo-id=rhel-7-server-x86_64-iso \ --feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso \ --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \ --feed-cert=/etc/pki/entitlement/$keyid.pem \ --feed-key=/etc/pki/entitlement/$keyid-key.pem \ --validate true \ --serve-http true Successfully created repository [rhel-7-server-x86_64-iso] ========================================================================== =========== pulp-admin iso repo sync run ================================= [root@bumblebee conf.d]# date Wed May 13 17:35:56 AEST 2015 [root@bumblebee conf.d]# pulp-admin iso repo sync run --repo-id=rhel-7-server-x86_64-iso +----------------------------------------------------------------------+ Synchronizing Repository [rhel-7-server-x86_64-iso] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading the Pulp Manifest... Downloading the Pulp Manifest failed: {} Task Failed Importer indicated a failed response [root@bumblebee conf.d]# date Wed May 13 17:38:23 AEST 2015 [root@bumblebee conf.d]# ========================================================================== =========== /var/log/pulp ================================================ May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91] May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644] May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3] May 13 17:36:00 bumblebee pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 13 17:36:00 bumblebee pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91] succeeded in 0.0365472619887s: None May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) ('Connection aborted.', error(110, 'Connection timed out')) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) Traceback (most recent call last): May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 190, in _fetch May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) response = session.get(request.url, headers=headers) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 469, in get May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) return self.request('GET', url, **kwargs) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 457, in request May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) resp = self.send(prep, **send_kwargs) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 569, in send May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) r = adapter.send(request, **kwargs) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 407, in send May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) raise ConnectionError(err, request=request) May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR: (17541-47008) ConnectionError: ('Connection aborted.', error(110, 'Connection timed out')) May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR: Failed to download https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST: ('Connection aborted.', error(110, 'Connection timed out')). May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) Task pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644] raised unexpected: PulpExecutionException('Importer indicated a failed response',) May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) Traceback (most recent call last): May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) R = retval = fun(*args, **kwargs) May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__ May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) return super(Task, self).__call__(*args, **kwargs) May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) return self.run(*args, **kwargs) May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) raise PulpExecutionException(_('Importer indicated a failed response')) May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936) PulpExecutionException: Importer indicated a failed response May 13 17:38:12 bumblebee pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3] succeeded in 0.00555812800303s: None ========================================================================== =========== successful curl command ====================================== [root@bumblebee ~]# curl -v \ --proxy $http_proxy \ https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST \ --insecure \ --cert /etc/pki/entitlement/$keyid.pem \ --key /etc/pki/entitlement/$keyid-key.pem * About to connect() to proxy XXX.XXX.XXX.XXX port YY (#0) * Trying XXX.XXX.XXX.XXX... * Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port YY (#0) * Establish HTTP proxy tunnel to cdn.redhat.com:443 * Proxy auth using Basic with user 'USER' > CONNECT cdn.redhat.com:443 HTTP/1.1 > Host: cdn.redhat.com:443 > Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXX > User-Agent: curl/7.29.0 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection established < * Proxy replied OK to CONNECT request * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * NSS: client certificate from file * subject: CN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX * start date: Dec 09 05:00:00 2013 GMT * expire date: Dec 09 04:59:59 2016 GMT * common name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX * issuer: [email protected],CN=Red Hat Candlepin Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA * Server certificate: * subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red Hat,L=Raleigh,ST=North Carolina,C=US * start date: May 14 19:48:02 2014 GMT * expire date: May 11 19:48:02 2024 GMT * common name: cdn.redhat.com * issuer: [email protected],CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US > GET /content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST HTTP/1.1 > User-Agent: curl/7.29.0 > Host: cdn.redhat.com > Accept: */* > < HTTP/1.1 200 OK < Server: Apache < ETag: "dde1e965a48e56cba8aac0b597f0c7a6:1426504103" < Last-Modified: Mon, 16 Mar 2015 11:08:16 GMT < Accept-Ranges: bytes < Content-Length: 424 < Content-Type: text/plain < X-Akamai-Request-ID: 5c57dff5.19737949 < Date: Wed, 13 May 2015 07:14:12 GMT < X-Cache: TCP_MISS from a104-71-131-23.deploy.akamaitechnologies.com (AkamaiGHost/7.2.0-15182023) (-) < Connection: keep-alive < EJ-HOST: edgejavaapp5.prod.a4.vary.redhat.com < RedHat-PassSync-1.1.6-i386.msi,0cf9b1e6998f756effc051a193c764e6f9976ac00b506db1381f8649404ec534,3059712 RedHat-PassSync-1.1.6-x86_64.msi,5430ac6ccb26340db4da82235e0bbd70c4a21f7ad7e7cd6d953159ea2f1c80df,3399680 rhel-server-7.1-x86_64-boot.iso,b6c57194130025cecc036676d43235508f4031d99041051ea1c61e259bd95903,358612992 rhel-server-7.1-x86_64-dvd.iso,3685468ec6cdcb70dfc85ebbc164da427dc2d762644c3c2ee1520f4f661c15ce,3890216960 * Connection #0 to host 10.226.8.34 left intact [root@bumblebee ~]# ========================================================================== =========== pulp rpm versions ============================================ [root@bumblebee conf.d]# rpm -qa | grep pulp pulp-puppet-admin-extensions-2.6.1-1.el7.noarch python-pulp-client-lib-2.6.1-1.el7.noarch pulp-selinux-2.6.1-1.el7.noarch pulp-rpm-plugins-2.6.1-1.el7.noarch python-isodate-0.5.0-4.pulp.el7.noarch pulp-server-2.6.1-1.el7.noarch python-pulp-rpm-common-2.6.1-1.el7.noarch pulp-rpm-admin-extensions-2.6.1-1.el7.noarch python-kombu-3.0.24-5.pulp.el7.noarch python-pulp-puppet-common-2.6.1-1.el7.noarch pulp-puppet-plugins-2.6.1-1.el7.noarch pulp-admin-client-2.6.1-1.el7.noarch python-pulp-common-2.6.1-1.el7.noarch python-pulp-bindings-2.6.1-1.el7.noarch ========================================================================== _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
