** Description changed: [ Impact ] Starting in OpenStack 2023.1 (Antelope), compressing static assets with django-compressor fails around 5-10% of the time (see also the Debian bug [1]). This is done during the Horizon postinst and prevents the package from installing when the race occurs. This affects Noble & Oracular, and UCA Antelope through Epoxy. Hitting a parse error in pyscss: ``` Looking for 'dashboard/scss/serial_console.scss' in storage Traceback (most recent call last): File "/usr/share/openstack-dashboard/manage.py", line 25, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 296, in handle self.handle_inner(**options) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 319, in handle_inner offline_manifest, block_count, results = self.compress(engine, extensions, verbosity, follow_links, log) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 227, in compress raise errors[0] django.core.management.base.CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR ``` Debian had little luck in finding the root cause and uploaded a workaround that simply retries failed compressions 5 times, dramatically reducing the liklihood that the failure breaks installation [2]. django-compressor introduced threaded compression in 3.0 [3], which was released in UCA Zed and Ubuntu Noble. I opened [4] in django-compressor to troubleshoot. The investigation there points to pyscss failing with correct input from django-compressor, indicating that the root cause is likely in pyscss. Horizon (Flamingo+) has switched from pyscss to libsass [5]. This bug is not present in Ubuntu Plucky containing that commit. The bug does not affect pyscss 1.3.7-5build1 or earlier. Since the bug is no longer present in devel, I'd like to SRU the Debian workaround to the affected Ubuntu & UCA packages without knowing the root cause of the race in pyscss. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069097 [2] https://salsa.debian.org/openstack-team/services/horizon/-/commit/9802ded2ab24cab767d0c91dfcabea117ec5fd9d [3] https://github.com/django-compressor/django-compressor/pull/912 [4] https://github.com/django-compressor/django-compressor/issues/1291 [5] https://opendev.org/openstack/horizon/commit/283219ab5ef4344e3808cd61b4cc0821886acffb [ Test Plan ] Install horizon and an affected version of python3-django-compressor (>=3.0). The Debian bug indicates that this fails between 5-10% of the time. Running `dpkg-reconfigure openstack-dashboard` in a loop on ~10 containers at once usually reproduces it in a couple of minutes, at most 30 minutes. The bug can be considered fixed if the loop runs for 1 hour with no occurances. I use the following scripts against my LXD cluster: ``` #!/bin/bash # # /usr/local/bin/horizon-reconfigure.sh in each test container set -ex while true; do dpkg-reconfigure openstack-dashboard done ``` Start and check if running: ``` lxc exec -t "${inst}" -- nohup bash -c "horizon-reconfigure.sh &>> /var/log/dpkg-horizon &" lxc exec "${inst}" -- ps -aux | grep horizon-reconfigure ``` For reference, compress can be run with the following to get more output and a traceback on failure: ``` python3 /usr/share/openstack-dashboard/manage.py compress -v 3 --traceback ``` [ Where problems could occur ] * The workaround does not fix the race; the package may still fail to install if the race occurs 5 times in a row. * If the change is wrong/broken, it risks regressions during openstack- dashboard install/reconfigure. This is covered by `dpkg-reconfigure` in the test plan. [ Other info ] * As this change affects upgrades between OpenStack versions via the UCA, I'm opening MPs against all affected versions, even those that are - past EOL (Zed+). + past EOL (Antelope+). * The patch I've proposed drops the css change from the Debian commit as it is unrelated to the issue described here. [ Original Description ] We have seen that the puppet integration job with Ubuntu 22.04 and UCA Bobcat consistently fails. Looking into the error it seems something is wrong with installation of openstack-dashboard package. 2023-12-01 07:13:16 +0000 Puppet (err): Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openstack-dashboard' returned 100: Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: openstack-dashboard-common python3-asgiref python3-csscompressor python3-django python3-django-appconf python3-django-compressor python3-django-debreach python3-django-horizon python3-django-openstack-auth python3-django-pyscss python3-pint python3-pyscss python3-rcssmin python3-rjsmin Suggested packages: bpython3 geoip-database-contrib gettext ipython3 libgdal20 libsqlite3-mod-spatialite python-django-doc python3-flup python3-mysqldb python3-pil python3-selenium python3-sqlite python-django-appconf-doc python3-calmjs python-django-debreach-doc The following NEW packages will be installed: openstack-dashboard openstack-dashboard-common python3-asgiref python3-csscompressor python3-django python3-django-appconf python3-django-compressor python3-django-debreach python3-django-horizon python3-django-openstack-auth python3-django-pyscss python3-pint python3-pyscss python3-rcssmin python3-rjsmin 0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded. Need to get 14.2 MB of archives. After this operation, 92.3 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! python3-rcssmin python3-rjsmin python3-django-compressor python3-pyscss openstack-dashboard-common python3-django-openstack-auth python3-django-horizon openstack-dashboard Authentication warning overridden. Get:1 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rcssmin amd64 1:1.1.0-2~cloud0 [17.5 kB] Get:2 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-asgiref all 3.5.0-1 [23.9 kB] Get:3 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rjsmin amd64 1.2.0+dfsg1-2build2~cloud0 [16.6 kB] Get:4 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-csscompressor all 0.9.5-2 [57.3 kB] Get:5 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-compressor all 4.0-2ubuntu1~cloud0 [71.8 kB] Get:6 https://mirror.gra1.ovh.opendev.org/ubuntu jammy-updates/main amd64 python3-django all 2:3.2.12-2ubuntu1.9 [3020 kB] Get:7 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-pyscss amd64 1.4.0-3~cloud0 [97.9 kB] Get:8 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-appconf all 1.0.3-1 [7620 B] Get:9 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard-common all 4:23.3.0-0ubuntu1.1~cloud0 [27.0 kB] Get:10 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-debreach all 2.0.1-2 [5482 B] Get:11 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-openstack-auth all 4:23.3.0-0ubuntu1.1~cloud0 [54.9 kB] Get:12 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-pyscss all 2.0.2-10 [9320 B] Get:13 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-horizon all 4:23.3.0-0ubuntu1.1~cloud0 [10.6 MB] Get:14 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-pint all 0.18-1 [156 kB] Get:15 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard all 4:23.3.0-0ubuntu1.1~cloud0 [20.2 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 14.2 MB in 7s (2004 kB/s) Selecting previously unselected package python3-asgiref. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 87302 files and directories currently installed.) Preparing to unpack .../00-python3-asgiref_3.5.0-1_all.deb ... Unpacking python3-asgiref (3.5.0-1) ... Selecting previously unselected package python3-csscompressor. Preparing to unpack .../01-python3-csscompressor_0.9.5-2_all.deb ... Unpacking python3-csscompressor (0.9.5-2) ... Selecting previously unselected package python3-django. Preparing to unpack .../02-python3-django_2%3a3.2.12-2ubuntu1.9_all.deb ... Unpacking python3-django (2:3.2.12-2ubuntu1.9) ... Selecting previously unselected package python3-django-appconf. Preparing to unpack .../03-python3-django-appconf_1.0.3-1_all.deb ... Unpacking python3-django-appconf (1.0.3-1) ... Selecting previously unselected package python3-rcssmin. Preparing to unpack .../04-python3-rcssmin_1%3a1.1.0-2~cloud0_amd64.deb ... Unpacking python3-rcssmin (1:1.1.0-2~cloud0) ... Selecting previously unselected package python3-rjsmin. Preparing to unpack .../05-python3-rjsmin_1.2.0+dfsg1-2build2~cloud0_amd64.deb ... Unpacking python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ... Selecting previously unselected package python3-django-compressor. Preparing to unpack .../06-python3-django-compressor_4.0-2ubuntu1~cloud0_all.deb ... Unpacking python3-django-compressor (4.0-2ubuntu1~cloud0) ... Selecting previously unselected package python3-django-debreach. Preparing to unpack .../07-python3-django-debreach_2.0.1-2_all.deb ... Unpacking python3-django-debreach (2.0.1-2) ... Selecting previously unselected package python3-pyscss. Preparing to unpack .../08-python3-pyscss_1.4.0-3~cloud0_amd64.deb ... Unpacking python3-pyscss (1.4.0-3~cloud0) ... Selecting previously unselected package python3-django-pyscss. Preparing to unpack .../09-python3-django-pyscss_2.0.2-10_all.deb ... Unpacking python3-django-pyscss (2.0.2-10) ... Selecting previously unselected package python3-pint. Preparing to unpack .../10-python3-pint_0.18-1_all.deb ... Unpacking python3-pint (0.18-1) ... Selecting previously unselected package openstack-dashboard-common. Preparing to unpack .../11-openstack-dashboard-common_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package python3-django-openstack-auth. Preparing to unpack .../12-python3-django-openstack-auth_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package python3-django-horizon. Preparing to unpack .../13-python3-django-horizon_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package openstack-dashboard. Preparing to unpack .../14-openstack-dashboard_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ... Setting up python3-pyscss (1.4.0-3~cloud0) ... Setting up python3-rcssmin (1:1.1.0-2~cloud0) ... Setting up python3-csscompressor (0.9.5-2) ... Setting up python3-pint (0.18-1) ... Setting up openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-asgiref (3.5.0-1) ... Setting up python3-django (2:3.2.12-2ubuntu1.9) ... Setting up python3-django-appconf (1.0.3-1) ... Setting up python3-django-debreach (2.0.1-2) ... Setting up python3-django-compressor (4.0-2ubuntu1~cloud0) ... Setting up python3-django-pyscss (2.0.2-10) ... Setting up python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ... update-alternatives: using /usr/lib/python3/dist-packages/openstack_dashboard to provide /usr/share/openstack-dashboard/openstack_dashboard (openstack_dashboard) in auto mode Setting up openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ... Adding system user `horizon' (UID 133) ... Adding new user `horizon' (UID 133) with group `horizon' ... Not creating home directory `/var/lib/openstack-dashboard'. Collecting and compressing static assets... CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR dpkg: error processing package openstack-dashboard (--configure): installed openstack-dashboard package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: openstack-dashboard E: Sub-process /usr/bin/dpkg returned an error code (1) ...
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2045394 Title: Installation of openstack-dashboard fails during asset compression To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2045394/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
