I'm guessing it's a broken symlink? Try removing it. David
On Thu, Apr 15, 2021 at 8:53 AM Eric VS <vs.e...@gmail.com> wrote: > No the one that the pulpcore-api service isn’t started and when trying to > start manually I get that error for the jsquery not found > > Kind regards > > > On Thu, 15 Apr 2021 at 07:50 David Davis <davidda...@redhat.com> wrote: > >> When you say you run the playbook again and got the same error, which >> error do you mean? The "No such file or directory: >> '/var/lib/pulp/assets/rest_framework/js/jquery-3.4.1.min.js'" one? >> >> >> David >> >> >> On Thu, Apr 15, 2021 at 4:43 AM Eric VS <vs.e...@gmail.com> wrote: >> >>> Hey Matthias, >>> >>> I ran the pulpcore-manager collectstatic with success, then the playbook >>> again but got the same error. This is the full list of commands for >>> pulpcore-manager, any others I would need to run? >>> >>> Last login: Thu Apr 15 16:23:00 CST 2021 on pts/0 >>> [root@tw-stg-man01 ~]# export PULP_SETTINGS=/etc/pulp/settings.py && >>> /usr/local/lib/pulp/bin/pulpcore-manager >>> >>> Type 'pulpcore-manager help <subcommand>' for help on a specific >>> subcommand. >>> >>> Available subcommands: >>> >>> [app] >>> handle-artifact-checksums >>> reset-admin-password >>> stage-profile-summary >>> >>> [auth] >>> changepassword >>> createsuperuser >>> >>> [contenttypes] >>> remove_stale_contenttypes >>> >>> [django] >>> check >>> compilemessages >>> createcachetable >>> dbshell >>> diffsettings >>> dumpdata >>> flush >>> inspectdb >>> loaddata >>> makemessages >>> makemigrations >>> migrate >>> sendtestemail >>> shell >>> showmigrations >>> sqlflush >>> sqlmigrate >>> sqlsequencereset >>> squashmigrations >>> startapp >>> startproject >>> test >>> testserver >>> >>> [drf_spectacular] >>> spectacular >>> >>> [guardian] >>> clean_orphan_obj_perms >>> >>> [rest_framework] >>> generateschema >>> >>> [sessions] >>> clearsessions >>> >>> [staticfiles] >>> collectstatic >>> findstatic >>> runserver >>> >>> *Eric Van Steenbergen* >>> >>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>> *Skype: ericvs2014* >>> *LinkedIn: Eric Van Steenbergen >>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>> >>> >>> On Thu, Apr 15, 2021 at 3:13 AM Matthias Dellweg <mdell...@redhat.com> >>> wrote: >>> >>>> Yeah, i can see how that happened. There may be some triggers from the >>>> previous run that aren't triggered now, because "nothing changed" according >>>> to ansible. >>>> This one should be: pulpcore-manager collect-static (or very similar) >>>> When you just run pulpcore-manager you get a list where you should be >>>> able to spot that subcommand. >>>> >>>> On Thu, Apr 15, 2021 at 10:06 AM Eric VS <vs.e...@gmail.com> wrote: >>>> >>>>> Hi Matthias, >>>>> >>>>> I did try to run that command but thought it was failing due to the >>>>> last error. >>>>> Unfortunately, when I just tried running the playbook again I ran into >>>>> the next error :-) >>>>> >>>>> TASK [pulp.pulp_installer.pulp_health_check : Checking Pulp services] >>>>> *********************************************************************************************************************************************************** >>>>> skipping: [localhost] => (item={'key': 'pulpcore-worker@2.service', >>>>> 'value': {'status': 'unknown', 'source': 'systemd', 'state': 'running', >>>>> 'name': 'pulpcore-worker@2.service'}}) >>>>> skipping: [localhost] => (item={'key': >>>>> 'pulpcore-resource-manager.service', 'value': {'status': 'enabled', >>>>> 'source': 'systemd', 'state': 'running', 'name': >>>>> 'pulpcore-resource-manager.service'}}) >>>>> skipping: [localhost] => (item={'key': 'pulpcore-worker@.service', >>>>> 'value': {'status': 'enabled', 'source': 'systemd', 'state': 'unknown', >>>>> 'name': 'pulpcore-worker@.service'}}) >>>>> skipping: [localhost] => (item={'key': 'pulpcore-worker@1.service', >>>>> 'value': {'status': 'unknown', 'source': 'systemd', 'state': 'running', >>>>> 'name': 'pulpcore-worker@1.service'}}) >>>>> failed: [localhost] (item={'key': 'pulpcore-api.service', 'value': >>>>> {'status': 'enabled', 'source': 'systemd', 'state': 'stopped', 'name': >>>>> 'pulpcore-api.service'}}) => {"ansible_loop_var": "item", "changed": >>>>> false, >>>>> "item": {"key": "pulpcore-api.service", "value": {"name": >>>>> "pulpcore-api.service", "source": "systemd", "state": "stopped", "status": >>>>> "enabled"}}, "msg": "pulpcore-api.service state: stopped"} >>>>> skipping: [localhost] => (item={'key': 'pulpcore-content.service', >>>>> 'value': {'status': 'enabled', 'source': 'systemd', 'state': 'running', >>>>> 'name': 'pulpcore-content.service'}}) >>>>> >>>>> PLAY RECAP >>>>> ********************************************************************************************************************************************************************************************************************** >>>>> localhost : ok=115 changed=19 unreachable=0 >>>>> failed=1 skipped=50 rescued=0 ignored=0 >>>>> >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo systemctl start >>>>> pulpcore-api >>>>> >>>>> 2 ↵ >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) ^start^status >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo systemctl status >>>>> pulpcore-api >>>>> ● pulpcore-api.service - Pulp WSGI Server >>>>> Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; >>>>> enabled; vendor preset: disabled) >>>>> Active: activating (auto-restart) (Result: exit-code) since Thu >>>>> 2021-04-15 16:03:12 CST; 1s ago >>>>> Process: 4875 ExecStart=/usr/local/lib/pulp/bin/gunicorn >>>>> pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --workers 1 >>>>> --timeout 90 --access-logfile - --access-logformat pulp >>>>> [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s >>>>> "%(f)s" "%(a)s" (code=exited, status=3) >>>>> Main PID: 4875 (code=exited, status=3) >>>>> >>>>> Apr 15 16:03:12 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>>>> pulpcore-api.service entered failed state. >>>>> Apr 15 16:03:12 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service failed. >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo journalctl -u >>>>> pulpcore-api >>>>> >>>>> 3 ↵ >>>>> -- Logs begin at Mon 2020-07-20 11:44:39 CST, end at Thu 2021-04-15 >>>>> 16:03:38 CST. -- >>>>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Started Pulp WSGI Server. >>>>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:09 +0800] [815] [INFO] Starting gunicorn 20.0.4 >>>>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:09 +0800] [815] [INFO] Listening at: >>>>> http://127.0.0.1:24817 (815) >>>>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:09 +0800] [815] [INFO] Using worker: sync >>>>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:09 +0800] [923] [INFO] Booting worker with pid: 923 >>>>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Stopping Pulp WSGI Server... >>>>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:27 +0800] [815] [INFO] Handling signal: term >>>>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 02:24:27 +0000] [923] [INFO] Worker exiting (pid: 923) >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>>>> [2020-08-05 10:24:28 +0800] [815] [INFO] Shutting down: Master >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Stopped Pulp WSGI Server. >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Started Pulp WSGI Server. >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 10:24:28 +0800] [3794] [INFO] Starting >>>>> gunicorn >>>>> 20.0.4 >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 10:24:28 +0800] [3794] [INFO] Listening at: >>>>> http://127.0.0.1:24817 (3794) >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 10:24:28 +0800] [3794] [INFO] Using worker: >>>>> sync >>>>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 10:24:28 +0800] [3873] [INFO] Booting worker >>>>> with pid: 3873 >>>>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 11:16:03 +0800] [3794] [INFO] Handling signal: >>>>> term >>>>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Stopping Pulp WSGI Server... >>>>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 03:16:03 +0000] [3873] [INFO] Worker exiting >>>>> (pid: 3873) >>>>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[3794]: [2020-08-05 11:16:03 +0800] [3794] [INFO] Shutting down: >>>>> Master >>>>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Stopped Pulp WSGI Server. >>>>> -- Reboot -- >>>>> Aug 05 11:19:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>>>> Started Pulp WSGI Server. >>>>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: [2020-08-05 11:19:05 +0800] [1858] [INFO] Starting >>>>> gunicorn >>>>> 20.0.4 >>>>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: [2020-08-05 11:19:05 +0800] [1858] [INFO] Listening at: >>>>> http://127.0.0.1:24817 (1858) >>>>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: [2020-08-05 11:19:05 +0800] [1858] [INFO] Using worker: >>>>> sync >>>>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: [2020-08-05 11:19:05 +0800] [2329] [INFO] Booting worker >>>>> with pid: 2329 >>>>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: pulp: django.request:WARNING: Not Found: / >>>>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: 127.0.0.1 - - [05/Aug/2020:03:38:18 +0000] "GET / >>>>> HTTP/1.0" >>>>> 404 77 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) >>>>> Gecko/20100101 Firefox/81.0 >>>>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: pulp: django.request:WARNING: Not Found: /favicon.ico >>>>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net >>>>> gunicorn[1858]: 127.0.0.1 - - [05/Aug/2020:03:38:18 +0000] "GET >>>>> /favicon.ico HTTP/1.0" 404 77 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X >>>>> 10.15; rv:81.0) Gecko/20100101 F >>>>> Aug 07 13:40:58 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:40:58 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>>>> Aug 07 13:41:07 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:41:07 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>>>> Aug 07 13:41:43 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:41:43 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>>>> Aug 07 13:44:48 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:44:48 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>>>> Aug 07 13:44:59 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:44:59 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>>>> Aug 07 13:48:11 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [07/Aug/2020:05:48:11 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>>>> Aug 10 10:34:07 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [10/Aug/2020:02:34:07 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>>>> Aug 10 12:55:34 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - - [10/Aug/2020:04:55:34 +0000] "GET /pulp/api/v3/status/ >>>>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>>>> Aug 10 13:01:47 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> pulp: django.request:WARNING: Not Found: / >>>>> Aug 10 13:01:47 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:01:47 +0000] "GET / HTTP/1.1" 404 77 "-" >>>>> "HTTPie/0.9.4" >>>>> Aug 10 13:02:01 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> pulp: django.request:WARNING: Not Found: / >>>>> Aug 10 13:02:01 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:02:01 +0000] "GET / HTTP/1.1" 404 77 "-" >>>>> "HTTPie/0.9.4" >>>>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "POST >>>>> /pulp/api/v3/repositories/rpm/rpm/ HTTP/1.1" 201 441 "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "GET >>>>> /pulp/api/v3/repositories/rpm/rpm/46d2f3e3-904e-448c-ae36-8d0662788f8e/ >>>>> HTTP/1.1" 200 441 "-" "HTTPie/0.9. >>>>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "POST >>>>> /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1" 201 462 "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>>>> /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1" 200 514 "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>>>> /pulp/api/v3/remotes/rpm/rpm/a4e0901e-5091-4a3e-ad6d-41f1f0c88d1e/ >>>>> HTTP/1.1" 200 462 "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "POST >>>>> /pulp/api/v3/repositories/rpm/rpm/46d2f3e3-904e-448c-ae36-8d0662788f8e/sync/ >>>>> HTTP/1.1" 202 67 "-" "HTTPie >>>>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 606 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:19 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:19 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1094 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:20 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:20 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:22 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:22 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:23 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:23 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:24 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:24 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:26 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:26 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:27 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:27 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:28 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:28 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:30 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:30 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:31 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:31 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> Aug 10 13:03:33 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>>>> 127.0.0.1 - admin [10/Aug/2020:05:03:33 +0000] "GET >>>>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 >>>>> 1201 >>>>> "-" "HTTPie/0.9.4" >>>>> ...skipping... >>>>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service: main process exited, code=exited, >>>>> status=3/NOTIMPLEMENTED >>>>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>>>> pulpcore-api.service entered failed state. >>>>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service failed. >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service holdoff time over, scheduling restart. >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> Stopped Pulp WSGI Server. >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> Started Pulp WSGI Server. >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Starting gunicorn 20.1.0 >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Listening at: >>>>> http://127.0.0.1:24817 (4938) >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Using worker: sync >>>>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:37 +0800] [4941] [INFO] Booting worker with pid: 4941 >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 08:03:39 +0000] [4941] [ERROR] Exception in worker process >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> Traceback (most recent call last): >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/arbiter.py", >>>>> line 589, in spawn_worker >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> worker.init_process() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/workers/base.py", >>>>> line 134, in init_process >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.load_wsgi() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/workers/base.py", >>>>> line 146, in load_wsgi >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.wsgi = self.app.wsgi() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/base.py", >>>>> line 67, in wsgi >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.callable = self.load() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/wsgiapp.py", >>>>> line 58, in load >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> return self.load_wsgiapp() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/wsgiapp.py", >>>>> line 48, in load_wsgiapp >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> return util.import_app(self.app_uri) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/util.py", >>>>> line 359, in import_app >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> mod = importlib.import_module(module) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in >>>>> import_module >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> return _bootstrap._gcd_import(name[level:], package, level) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap>", line 994, in _gcd_import >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap>", line 971, in _find_and_load >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap_external>", line 678, in exec_module >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File "<frozen importlib._bootstrap>", line 219, in >>>>> _call_with_frames_removed >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/wsgi.py", >>>>> line 17, in <module> >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> application = get_wsgi_application() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/wsgi.py", >>>>> line 13, in get_wsgi_application >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> return WSGIHandler() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/wsgi.py", >>>>> line 135, in __init__ >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.load_middleware() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", >>>>> line 37, in load_middleware >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> mw_instance = middleware(handler) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/middleware.py", >>>>> line 49, in __init__ >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.add_files(self.static_root, prefix=self.static_prefix) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>>>> line 113, in add_files >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> self.update_files_dictionary(root, prefix) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>>>> line 120, in update_files_dictionary >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> stat_cache = dict(scantree(root)) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>>>> line 271, in scantree >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> yield from scantree(entry.path) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>>>> line 271, in scantree >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> yield from scantree(entry.path) >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>>>> line 273, in scantree >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> yield entry.path, entry.stat() >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> FileNotFoundError: [Errno 2] No such file or directory: >>>>> '/var/lib/pulp/assets/rest_framework/js/jquery-3.4.1.min.js' >>>>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 08:03:39 +0000] [4941] [INFO] Worker exiting (pid: 4941) >>>>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:40 +0800] [4938] [INFO] Shutting down: Master >>>>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>>>> [2021-04-15 16:03:40 +0800] [4938] [INFO] Reason: Worker failed to boot. >>>>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service: main process exited, code=exited, >>>>> status=3/NOTIMPLEMENTED >>>>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>>>> pulpcore-api.service entered failed state. >>>>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>>>> pulpcore-api.service failed. >>>>> >>>>> Kind regards, >>>>> >>>>> *Eric Van Steenbergen* >>>>> >>>>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>>>> *Skype: ericvs2014* >>>>> *LinkedIn: Eric Van Steenbergen >>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>> >>>>> >>>>> On Thu, Apr 15, 2021 at 2:56 AM Matthias Dellweg <mdell...@redhat.com> >>>>> wrote: >>>>> >>>>>> >>>>>> There is the clue: Run 'pulpcore-manager handle-artifact-checksums' >>>>>> first to unset forbidden checksums. >>>>>> Pulp dropped to support md5 and sha1 by default. And there is a >>>>>> manual step that should bring the database in the right place. >>>>>> Try rerunning the ansible playbook afterwards. >>>>>> >>>>>> However i thought this cleanup routine should have been called >>>>>> automatically (as a migration) in the upgrade process. Can anyone spot >>>>>> why >>>>>> it didn't? >>>>>> >>>>>> On Thu, Apr 15, 2021 at 1:48 AM Eric VS <vs.e...@gmail.com> wrote: >>>>>> >>>>>>> Hey Matthias, >>>>>>> >>>>>>> The only plugin I am currently using, at least until I'm familiar >>>>>>> enough with Pulp, is the RPM plugin. That's our primary use case for >>>>>>> now to >>>>>>> make sure we are in control of what version of packages we are using >>>>>>> across >>>>>>> our environments and avoid discrepancies between servers. >>>>>>> >>>>>>> I have never added or removed any other plugin. After also upgrading >>>>>>> the Postgres ansible-galaxy role I don't get the version mismatch error >>>>>>> any >>>>>>> more but now get the below error and 'command to run'. Running the >>>>>>> command >>>>>>> fails however also.... >>>>>>> >>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static >>>>>>> content] >>>>>>> ****************************************************************************************************************************************************** >>>>>>> fatal: [localhost]: FAILED! => {"changed": true, "cmd": >>>>>>> ["/usr/local/lib/pulp/bin/pulpcore-manager", "collectstatic", >>>>>>> "--noinput", >>>>>>> "--link"], "delta": "0:00:02.481870", "end": "2021-04-15 >>>>>>> 07:34:39.205489", >>>>>>> "msg": "non-zero return code", "rc": 1, "start": "2021-04-15 >>>>>>> 07:34:36.723619", "stderr": "Traceback (most recent call last):\n File >>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>\n >>>>>>> sys.exit(manage())\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>> line 11, in manage\n execute_from_command_line(sys.argv)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 381, in execute_from_command_line\n utility.execute()\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 357, in execute\n django.setup()\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py\", >>>>>>> line 24, in setup\n apps.populate(settings.INSTALLED_APPS)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py\", >>>>>>> line 114, in populate\n app_config.import_models()\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py\", >>>>>>> line 211, in import_models\n self.models_module = >>>>>>> import_module(models_module_name)\n File >>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>> import_module\n return _bootstrap._gcd_import(name[level:], package, >>>>>>> level)\n File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>> _gcd_import\n File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>> _find_and_load\n File \"<frozen importlib._bootstrap>\", line 955, in >>>>>>> _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", line >>>>>>> 665, in _load_unlocked\n File \"<frozen >>>>>>> importlib._bootstrap_external>\", >>>>>>> line 678, in exec_module\n File \"<frozen importlib._bootstrap>\", line >>>>>>> 219, in _call_with_frames_removed\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py\", >>>>>>> line 22, in <module>\n from .exporter import ( # noqa\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py\", >>>>>>> line 15, in <module>\n from .task import CreatedResource, Task\n >>>>>>> >>>>>>> File\"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py\", >>>>>>> line 14, in <module>\n from pulpcore.app.settings import WORKER_TTL\n >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 366, in <module>\n raise e\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 344, in <module>\n >>>>>>> ).format(checksum)\ndjango.core.exceptions.ImproperlyConfigured: There >>>>>>> have been identified artifacts with forbidden checksum 'sha1'. Run >>>>>>> 'pulpcore-manager handle-artifact-checksums' first to unset forbidden >>>>>>> checksums.", "stderr_lines": ["Traceback (most recent call last):", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>", " >>>>>>> sys.exit(manage())", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>> line 11, in manage", " execute_from_command_line(sys.argv)", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 381, in execute_from_command_line", " utility.execute()", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 357, in execute", " django.setup()", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py\", >>>>>>> line 24, in setup", " apps.populate(settings.INSTALLED_APPS)", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py\", >>>>>>> line 114, in populate", " app_config.import_models()", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py\", >>>>>>> line 211, in import_models", " self.models_module = >>>>>>> import_module(models_module_name)", " File >>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>> import_module", " return _bootstrap._gcd_import(name[level:], >>>>>>> package, >>>>>>> level)", " File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>> _gcd_import", " File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>> _find_and_load", " File \"<frozen importlib._bootstrap>\", line 955, in >>>>>>> _find_and_load_unlocked", " File \"<frozen importlib._bootstrap>\", >>>>>>> line >>>>>>> 665, in _load_unlocked", " File \"<frozen >>>>>>> importlib._bootstrap_external>\", line 678, in exec_module", " File >>>>>>> \"<frozen importlib._bootstrap>\", line 219, in >>>>>>> _call_with_frames_removed", >>>>>>> " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py\", >>>>>>> line 22, in <module>", " from .exporter import ( # noqa", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py\", >>>>>>> line 15, in <module>", " from .task import CreatedResource, Task", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py\", >>>>>>> line 14, in <module>", " from pulpcore.app.settings import >>>>>>> WORKER_TTL", >>>>>>> " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 366, in <module>", " raise e", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 344, in <module>", " ).format(checksum)", >>>>>>> "django.core.exceptions.ImproperlyConfigured: There have been identified >>>>>>> artifacts withforbidden checksum 'sha1'. Run 'pulpcore-manager >>>>>>> handle-artifact-checksums' first to unset forbidden checksums."], >>>>>>> "stdout": >>>>>>> "", "stdout_lines": []} >>>>>>> >>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restart all Pulp >>>>>>> services] >>>>>>> *************************************************************************************************************************************************** >>>>>>> >>>>>>> NO MORE HOSTS LEFT >>>>>>> ************************************************************************************************************************************************************************************************************** >>>>>>> >>>>>>> PLAY RECAP >>>>>>> ********************************************************************************************************************************************************************************************************************** >>>>>>> localhost : ok=87 changed=3 unreachable=0 >>>>>>> failed=1 skipped=48 rescued=0 ignored=0 >>>>>>> >>>>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) export >>>>>>> PULP_SETTINGS=/etc/pulp/settings.py && >>>>>>> /usr/local/lib/pulp/bin/pulpcore-manager handle-artifact-checksums >>>>>>> 2 ↵ >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", >>>>>>> line 283, in <module> >>>>>>> CONTENT_ORIGIN >>>>>>> NameError: name 'CONTENT_ORIGIN' is not defined >>>>>>> >>>>>>> During handling of the above exception, another exception occurred: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/local/lib/pulp/bin/pulpcore-manager", line 8, in >>>>>>> <module> >>>>>>> sys.exit(manage()) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py", >>>>>>> line 11, in manage >>>>>>> execute_from_command_line(sys.argv) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>>>> line 381, in execute_from_command_line >>>>>>> utility.execute() >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>>>> line 357, in execute >>>>>>> django.setup() >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py", >>>>>>> line 24, in setup >>>>>>> apps.populate(settings.INSTALLED_APPS) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py", >>>>>>> line 114, in populate >>>>>>> app_config.import_models() >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py", >>>>>>> line 211, in import_models >>>>>>> self.models_module = import_module(models_module_name) >>>>>>> File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in >>>>>>> import_module >>>>>>> return _bootstrap._gcd_import(name[level:], package, level) >>>>>>> File "<frozen importlib._bootstrap>", line 994, in _gcd_import >>>>>>> File "<frozen importlib._bootstrap>", line 971, in _find_and_load >>>>>>> File "<frozen importlib._bootstrap>", line 955, in >>>>>>> _find_and_load_unlocked >>>>>>> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked >>>>>>> File "<frozen importlib._bootstrap_external>", line 678, in >>>>>>> exec_module >>>>>>> File "<frozen importlib._bootstrap>", line 219, in >>>>>>> _call_with_frames_removed >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py", >>>>>>> line 22, in <module> >>>>>>> from .exporter import ( # noqa >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py", >>>>>>> line 15, in <module> >>>>>>> from .task import CreatedResource, Task >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py", >>>>>>> line 14, in <module> >>>>>>> from pulpcore.app.settings import WORKER_TTL >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", >>>>>>> line 287, in <module> >>>>>>> "CONTENT_ORIGIN is a required setting but it was not configured. >>>>>>> This may be caused " >>>>>>> django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a >>>>>>> required setting but it was not configured. This may be caused by >>>>>>> invalid >>>>>>> read permissions of the settings file. Note that CONTENT_ORIGIN is set >>>>>>> by >>>>>>> the installer automatically. >>>>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) >>>>>>> >>>>>>> I finally had some time to dig deeper and played around with the >>>>>>> permissions on the file to no result, until I ran the pulpcore-manager >>>>>>> command as root. Then this popped up as error indicating there is some >>>>>>> integrity error: >>>>>>> >>>>>>> [root@tw-stg-man01 ~]# export PULP_SETTINGS=/etc/pulp/settings.py >>>>>>> && /usr/local/lib/pulp/bin/pulpcore-manager handle-artifact-checksums >>>>>>> Removing forbidden checksum sha1 from database >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>>>> line 84, in _execute >>>>>>> return self.cursor.execute(sql, params) >>>>>>> psycopg2.errors.NotNullViolation: null value in column "sha1" >>>>>>> violates not-null constraint >>>>>>> DETAIL: Failing row contains (1883479d-9e2d-45e1-971f-37246fc3c358, >>>>>>> 2020-08-13 05:50:33.543549+00, 2020-08-13 05:50:33.543572+00, >>>>>>> artifact/52/e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac..., >>>>>>> 286400, 51b3d1a3ff20a0f88ccb084eca86073c, null, >>>>>>> bd036226130d705e95bf4ad9acaa5d8d6d4df7460943062c08154bf1, >>>>>>> 52e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac0f16a27789, >>>>>>> 53964efdef715dd8f8ff96588e7f1ab184a3747d39bb3c9c28fc339659014274..., >>>>>>> 0b0d8a7d5d5816705016a5a59489f032fddc31afc438a482549091b730ca4f6b...). >>>>>>> >>>>>>> >>>>>>> The above exception was the direct cause of the following exception: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/local/lib/pulp/bin/pulpcore-manager", line 8, in >>>>>>> <module> >>>>>>> sys.exit(manage()) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py", >>>>>>> line 11, in manage >>>>>>> execute_from_command_line(sys.argv) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>>>> line 381, in execute_from_command_line >>>>>>> utility.execute() >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>>>> line 375, in execute >>>>>>> self.fetch_command(subcommand).run_from_argv(self.argv) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/base.py", >>>>>>> line 323, in run_from_argv >>>>>>> self.execute(*args, **cmd_options) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/base.py", >>>>>>> line 364, in execute >>>>>>> output = self.handle(*args, **options) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", >>>>>>> line 212, in handle >>>>>>> artifacts_qs.update(**update_params) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", >>>>>>> line 741, in update >>>>>>> rows = query.get_compiler(self.db).execute_sql(CURSOR) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", >>>>>>> line 1471, in execute_sql >>>>>>> cursor = super().execute_sql(result_type) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", >>>>>>> line 1142, in execute_sql >>>>>>> cursor.execute(sql, params) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>>>> line 67, in execute >>>>>>> return self._execute_with_wrappers(sql, params, many=False, >>>>>>> executor=self._execute) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>>>> line 76, in _execute_with_wrappers >>>>>>> return executor(sql, params, many, context) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>>>> line 84, in _execute >>>>>>> return self.cursor.execute(sql, params) >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py", >>>>>>> line 89, in __exit__ >>>>>>> raise dj_exc_value.with_traceback(traceback) from exc_value >>>>>>> File >>>>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>>>> line 84, in _execute >>>>>>> return self.cursor.execute(sql, params) >>>>>>> django.db.utils.IntegrityError: null value in column "sha1" violates >>>>>>> not-null constraint >>>>>>> DETAIL: Failing row contains (1883479d-9e2d-45e1-971f-37246fc3c358, >>>>>>> 2020-08-13 05:50:33.543549+00, 2020-08-13 05:50:33.543572+00, >>>>>>> artifact/52/e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac..., >>>>>>> 286400, 51b3d1a3ff20a0f88ccb084eca86073c, null, >>>>>>> bd036226130d705e95bf4ad9acaa5d8d6d4df7460943062c08154bf1, >>>>>>> 52e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac0f16a27789, >>>>>>> 53964efdef715dd8f8ff96588e7f1ab184a3747d39bb3c9c28fc339659014274..., >>>>>>> 0b0d8a7d5d5816705016a5a59489f032fddc31afc438a482549091b730ca4f6b...). >>>>>>> >>>>>>> I could just delete the whole bunch and reinstall but that wouldn't >>>>>>> teach me anything useful I guess so I'm sticking to my guns hoping you >>>>>>> guys >>>>>>> can help me out in understanding the root cause. >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> *Eric Van Steenbergen* >>>>>>> >>>>>>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>>>>>> *Skype: ericvs2014* >>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 14, 2021 at 3:53 AM Matthias Dellweg < >>>>>>> mdell...@redhat.com> wrote: >>>>>>> >>>>>>>> In your playbook, where there is the list of plugins to install, >>>>>>>> did you ever remove one from that list? >>>>>>>> Also can you try adding "upgrade: true" to all plugins in that list? >>>>>>>> >>>>>>>> On Wed, Apr 14, 2021 at 5:27 AM Eric VS <vs.e...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Me again :-). >>>>>>>>> >>>>>>>>> I had a Pulp installation from several weeks ago (on our staging >>>>>>>>> environment) and wanted to align it today with my main Pulp >>>>>>>>> installation >>>>>>>>> running 3.12.0. >>>>>>>>> The staging one was still running on 3.5.0, installed using the >>>>>>>>> Ansible Pulp Installer so I followed the upgrade process for the >>>>>>>>> collection >>>>>>>>> and ran the playbook. This is the error I got, due to version >>>>>>>>> mismatch but >>>>>>>>> not sure on how to fix this one. >>>>>>>>> >>>>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restore SELinux >>>>>>>>> contexts on Pulp dirs that may exist] >>>>>>>>> ************************************************************************************************************************ >>>>>>>>> changed: [localhost] >>>>>>>>> >>>>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static >>>>>>>>> content] >>>>>>>>> ****************************************************************************************************************************************************** >>>>>>>>> fatal: [localhost]: FAILED! => {"changed": true, "cmd": >>>>>>>>> ["/usr/local/lib/pulp/bin/pulpcore-manager", "collectstatic", >>>>>>>>> "--noinput", >>>>>>>>> "--link"], "delta": "0:00:00.297250", "end": "2021-04-14 >>>>>>>>> 11:20:11.886134", >>>>>>>>> "msg": "non-zero return code", "rc": 1, "start": "2021-04-14 >>>>>>>>> 11:20:11.588884", "stderr": "Traceback (most recent call last):\n >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>\n >>>>>>>>> sys.exit(manage())\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>>>> line 11, in manage\n execute_from_command_line(sys.argv)\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>>>> line 381, in execute_from_command_line\n utility.execute()\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>>>> line 325, in execute\n settings.INSTALLED_APPS\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 79, in __getattr__\n self._setup(name)\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 66, in _setup\n self._wrapped = Settings(settings_module)\n >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 157, in __init__\n mod = >>>>>>>>> importlib.import_module(self.SETTINGS_MODULE)\n File >>>>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>>>> import_module\n return _bootstrap._gcd_import(name[level:], >>>>>>>>> package, >>>>>>>>> level)\n File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>>>> _gcd_import\n File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>>>> _find_and_load\n File \"<frozen importlib._bootstrap>\", line 955, in >>>>>>>>> _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", >>>>>>>>> line >>>>>>>>> 665, in _load_unlocked\n File \"<frozen >>>>>>>>> importlib._bootstrap_external>\", >>>>>>>>> line 678, in exec_module\n File \"<frozen importlib._bootstrap>\", >>>>>>>>> line >>>>>>>>> 219, in _call_with_frames_removed\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>>>> line 83, in <module>\n plugin_app_config = entry_point.load()\n >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 2317, in load\n self.require(*args, **kwargs)\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 2340, in require\n items = working_set.resolve(reqs, env, >>>>>>>>> installer, extras=self.extras)\n File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 779, in resolve\n raise VersionConflict(dist, >>>>>>>>> req).with_context(dependent_req)\npkg_resources.VersionConflict: >>>>>>>>> (pulpcore >>>>>>>>> 3.12.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), >>>>>>>>> Requirement.parse('pulpcore<3.6,>=3.4'))", "stderr_lines": ["Traceback >>>>>>>>> (most recent call last):", " File >>>>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>", " >>>>>>>>> sys.exit(manage())", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>>>> line 11, in manage", " execute_from_command_line(sys.argv)", " >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>>>> line 381, in execute_from_command_line", " utility.execute()", " >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>>>> line 325, in execute", " settings.INSTALLED_APPS", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 79, in __getattr__", " self._setup(name)", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 66, in _setup", " self._wrapped = Settings(settings_module)", >>>>>>>>> " >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>>>> line 157, in __init__", " mod = >>>>>>>>> importlib.import_module(self.SETTINGS_MODULE)", "File >>>>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>>>> import_module", " return _bootstrap._gcd_import(name[level:], >>>>>>>>> package, >>>>>>>>> level)", " File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>>>> _gcd_import", " File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>>>> _find_and_load", " File \"<frozen importlib._bootstrap>\", line 955, >>>>>>>>> in >>>>>>>>> _find_and_load_unlocked", " File \"<frozen importlib._bootstrap>\", >>>>>>>>> line >>>>>>>>> 665, in _load_unlocked", "File \"<frozen >>>>>>>>> importlib._bootstrap_external>\", >>>>>>>>> line 678, in exec_module", " File \"<frozen importlib._bootstrap>\", >>>>>>>>> line >>>>>>>>> 219, in _call_with_frames_removed", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>>>> line 83, in <module>", " plugin_app_config = entry_point.load()", " >>>>>>>>> File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 2317, in load", " self.require(*args, **kwargs)", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 2340, in require", " items = working_set.resolve(reqs, env, >>>>>>>>> installer, extras=self.extras)", " File >>>>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>>>> line 779, in resolve", " raise VersionConflict(dist, >>>>>>>>> req).with_context(dependent_req)", "pkg_resources.VersionConflict: >>>>>>>>> (pulpcore 3.12.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), >>>>>>>>> Requirement.parse('pulpcore<3.6,>=3.4'))"], "stdout": "", >>>>>>>>> "stdout_lines": >>>>>>>>> []} >>>>>>>>> >>>>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restart all >>>>>>>>> Pulp services] >>>>>>>>> *************************************************************************************************************************************************** >>>>>>>>> >>>>>>>>> RUNNING HANDLER [geerlingguy.postgresql : restart postgresql] >>>>>>>>> ******************************************************************************************************************************************************************* >>>>>>>>> >>>>>>>>> NO MORE HOSTS LEFT >>>>>>>>> ************************************************************************************************************************************************************************************************************** >>>>>>>>> >>>>>>>>> PLAY RECAP >>>>>>>>> ********************************************************************************************************************************************************************************************************************** >>>>>>>>> localhost : ok=91 changed=33 unreachable=0 >>>>>>>>> failed=1 skipped=46 rescued=0 ignored=0 >>>>>>>>> >>>>>>>>> Any help is greatly appreciated. >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> *Eric Van Steenbergen* >>>>>>>>> >>>>>>>>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>>>>>>>> *Skype: ericvs2014* >>>>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>>>> _______________________________________________ >>>>>>>>> Pulp-list mailing list >>>>>>>>> Pulp-list@redhat.com >>>>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>>>>>> >>>>>>>> _______________________________________________ >>> Pulp-list mailing list >>> Pulp-list@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-list >> >> -- > *Eric Van Steenbergen* > > *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* > *Skype: ericvs2014* > *LinkedIn: Eric Van Steenbergen > <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-list