Debdiff for mod_wsgi patch - 2:28.0.0-0ubuntu1.1. Supercedes the May 29 debdiff Launchpad won't let me delete.
** Description changed: [Impact] - This release sports mostly bug-fixes and we would like to make sure all of our users have access to these improvements. - - The update contains the following package updates: - - neutron 2:28.0.0-0ubuntu1.1 + On Ubuntu resolute (and the Gazpacho cloud archive), the neutron-api + package serves the Neutron API under Apache mod_wsgi, but Neutron 28.0.0 + upstream assumes uWSGI is the only WSGI loader. Without the uwsgi module, + no API worker identifies itself as the first worker, so ML2 default + network segment ranges are never seeded -- tenant network creation on + ML2/OVN with the geneve type driver fails immediately with + NoNetworkAvailable. In addition, workers do not share a server start + time, so the OVN hash-ring cleanup races workers into deleting each + other's freshly registered rows. + + Every fresh or upgraded deployment using the packaged mod_wsgi + configuration is affected; basic workflows such as + "openstack network create" are non-functional. + + The update contains the following package updates: + - neutron 2:28.0.0-0ubuntu1.1 [Test Case] - This upload extends the existing neutron-api autopkgtest (debian/tests/neutron-api) so it directly exercises the mod_wsgi fix. The test deploys the Neutron API under Apache mod_wsgi (the package's default, WSGIDaemonProcess processes=4) with the geneve type driver and the network_segment_range service plugin enabled, then asserts the behaviour that was broken: - - a default network segment range is seeded exactly once; - the OVN hash ring has one node per mod_wsgi worker, all sharing a single start time (created_at); - that start time advances across an apache2 restart. - On the package currently in the release pocket these assertions fail (segment ranges are never seeded and the hash-ring workers race into deleting each other's rows); on the fixed package they pass. A verifier therefore only needs to run the neutron-api autopkgtest against -proposed and confirm it passes, e.g.: - - autopkgtest --test-name=neutron-api neutron --apt-pocket=proposed - - -- lxd autopkgtest/ubuntu/<series>/amd64 - - The following SRU process was followed: - https://documentation.ubuntu.com/sru/en/latest/reference/exception-OpenStack-Updates - - In order to avoid regression of existing consumers, the OpenStack team will run their continuous integration test against the packages that are in -proposed. A successful run of all available tests will be required before the - proposed packages can be let into -updates. - - The OpenStack team will be in charge of attaching the output summary of - the executed tests. The OpenStack team members will not mark - ‘verification-done’ until this has happened. + 1. Autopkgtest + + This upload extends the existing neutron-api autopkgtest + (debian/tests/neutron-api) so it directly exercises the mod_wsgi fix. + The test deploys the Neutron API under Apache mod_wsgi (the package's + default, WSGIDaemonProcess processes=4) with the geneve type driver and + the network_segment_range service plugin enabled, then asserts the + behaviour that was broken: + + * at least one default network segment range is seeded; + * the OVN hash ring has one node per mod_wsgi worker, all sharing a + single start time (created_at); + * that start time advances across an apache2 restart, with no stale + rows or primary-key collisions. + + On the package currently in the release pocket these assertions fail + (segment ranges are never seeded and the hash-ring workers race into + deleting each other's rows); on the fixed package they pass. Run the + test against -proposed with the proposed pocket pinned to this source + package only: + + autopkgtest --test-name=neutron-api neutron \ + --apt-pocket=proposed=src:neutron \ + -- lxd autopkgtest/ubuntu/resolute/amd64 + + 2. Manual end-to-end verification + + # 1. Install the broken (release-pocket) packages plus OVN and MySQL + apt update + apt install -y neutron-api neutron-common ovn-central mysql-server crudini + + # 2. Expose the OVN NB/SB databases on localhost TCP + ovn-nbctl set-connection ptcp:6641:127.0.0.1 + ovn-sbctl set-connection ptcp:6642:127.0.0.1 + + # 3. Database + mysql -u root <<'EOF' + CREATE DATABASE neutron; + CREATE USER 'neutron'@'localhost' IDENTIFIED BY 'changeme'; + GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost'; + EOF + + # 4. Neutron config: ML2/OVN with geneve tenant networks, no keystone + crudini --set /etc/neutron/neutron.conf database connection \ + mysql+pymysql://neutron:changeme@localhost/neutron + crudini --set /etc/neutron/neutron.conf DEFAULT auth_strategy noauth + crudini --set /etc/neutron/neutron.conf DEFAULT service_plugins network_segment_range + + ML2=/etc/neutron/plugins/ml2/ml2_conf.ini + crudini --set $ML2 ml2 type_drivers local,geneve + crudini --set $ML2 ml2 tenant_network_types geneve + crudini --set $ML2 ml2 mechanism_drivers ovn + crudini --set $ML2 ml2_type_geneve vni_ranges 1:65536 + crudini --set $ML2 ml2_type_geneve max_header_size 38 + crudini --set $ML2 ovn ovn_nb_connection tcp:127.0.0.1:6641 + crudini --set $ML2 ovn ovn_sb_connection tcp:127.0.0.1:6642 + + # 5. Schema, then start the API under Apache mod_wsgi + neutron-db-manage upgrade head + systemctl restart neutron-api neutron-rpc-server + + Then the before/after check. With noauth you can use plain curl (or + the openstack client with OS_AUTH_TYPE=none + OS_ENDPOINT=http://localhost:9696): + + curl -s -X POST http://localhost:9696/v2.0/networks \ + -H 'Content-Type: application/json' \ + -d '{"network": {"name": "private"}}' + + - Before (2:28.0.0-0ubuntu1): HTTP 503 with NoNetworkAvailable — "Unable to create the network. No tenant network is available for allocation." You may need + a few requests first to warm the workers, same as the autopkgtest. + - Upgrade: enable -proposed pinned to neutron (or your PPA for a pre-SRU rehearsal), apt install the neutron packages, systemctl restart apache2. + - After (2:28.0.0-0ubuntu1.1): HTTP 201 with the network JSON showing "provider:network_type": "geneve" and a VNI from the seeded default range.. + + 3. OpenStack CI + + The following SRU process was followed: + https://documentation.ubuntu.com/sru/en/latest/reference/exception-OpenStack-Updates + + In order to avoid regression of existing consumers, the OpenStack team + will run their continuous integration test against the packages that + are in -proposed. A successful run of all available tests will be + required before the proposed packages can be let into -updates. + + The OpenStack team will be in charge of attaching the output summary of + the executed tests. The OpenStack team members will not mark + 'verification-done' until this has happened. [Regression Potential] - In order to mitigate the regression potential, the results of the aforementioned tests are attached to this bug. - - The neutron change (LP: #2150285) is gated behind the mod_wsgi Python - module being importable, so uWSGI, CLI, and unit-test code paths are - unchanged; regression exposure is limited to Apache mod_wsgi API - deployments. The added behaviour (per-worker ID election and a shared, - restart-advancing start time) was verified end-to-end on resolute under - mod_wsgi: default segment ranges are seeded once, and OVN hash-ring - workers share one start time that advances across an apache2 restart. + The change is gated behind the mod_wsgi Python module being importable, + so uWSGI deployments, CLI tools, and unit tests execute unchanged code. + Regression exposure is therefore limited to Apache mod_wsgi API + deployments -- which are currently non-functional for tenant network + creation, bounding the downside. + + The shim adds file I/O under /var/lib/neutron/lock at worker startup: + each worker takes a numbered flock to elect a stable 1-indexed worker + ID, and a flock-guarded sentinel file records a start time shared by + all workers of one server generation. If the directory cannot be + created or written (the daemon runs as the neutron user, which owns + /var/lib/neutron), the helpers return None and behaviour degrades to + today's upstream behaviour rather than crashing. Worker ID slots are + bounded at 64; the packaged configuration uses 4. + + Untested edge cases: Apache graceful reload (the start time + intentionally does not advance within one server generation; cleanly + exiting workers deregister their own hash-ring rows) and mod_wsgi + worker recycling (a recycled worker releases its flock on exit and its + replacement reclaims a free slot). + + The autopkgtest results and OpenStack CI results are attached to this + bug. [Discussion] - neutron 2:28.0.0-0ubuntu1.1 carries a single downstream-only fix (LP: #2150285) that keeps the Neutron API operational under Apache mod_wsgi for one more cycle. Upstream supports only uWSGI, which exposes a worker ordinal and a shared server start time; under mod_wsgi these are absent, so default network segment ranges were never seeded and OVN hash-ring workers raced into deleting each other's rows. The patch supplies both via lock/sentinel files under /var/lib/neutron/lock and relaxes the ML2 first-worker guard as an idempotent safety net. It is marked Forwarded: not-needed and is intended to be dropped once the API is switched to uWSGI-only. - + neutron 2:28.0.0-0ubuntu1.1 carries a single downstream-only fix + (LP: #2150285) that keeps the Neutron API operational under Apache + mod_wsgi for one more cycle. Upstream supports only uWSGI, which + exposes a worker ordinal and a shared server start time via the uwsgi + module; under mod_wsgi these are absent. The patch supplies equivalents + via flock files and a parent-PID-keyed sentinel file under + /var/lib/neutron/lock, active only when the mod_wsgi module is + importable, and ships unit tests for the shim. No upstream code paths + outside neutron/common/wsgi_utils.py are modified. The patch is marked + Forwarded: not-needed and is intended to be dropped once upstream + supports non-uWSGI loaders or the package switches the API to + uWSGI-only. -------------------------------------------------------------------------------- Original Bug Report Below -------------------------------------------------------------------------------- In Ubuntu Resolute, the neutron-api package deploys the Neutron API using Apache mod_wsgi. With Neutron 2026.1 / OpenStack Gazpacho, this can prevent ML2 default network segment ranges from being initialized, causing tenant network creation to fail with NoNetworkAvailable. Ubuntu packaging currently installs neutron-api as an Apache/mod_wsgi application: - debian/control: neutron-api depends on apache2 | httpd and libapache2-mod-wsgi-py3 - debian/neutron-api.conf: uses WSGIDaemonProcess and WSGIScriptAlias - debian/neutron-api.wsgi: imports from neutron.wsgi.api import application However, Neutron’s ML2 network segment range initialization now gates initialization on the uWSGI worker ID: if wsgi_utils.get_api_worker_id() != wsgi_utils.FIRST_WORKER_ID: return wsgi_utils.get_api_worker_id() imports the uwsgi module and calls uwsgi.worker_id(). Under Apache mod_wsgi, the uwsgi module is not available, so this function returns None. As a result: None != 1 and initialize_network_segment_range_support() returns without initializing the default network segment ranges. In ML2/OVN deployments this leaves the allocation state for tenant network segment types, such as Geneve, unpopulated. Tenant network creation then fails with NoNetworkAvailable. Impact Fresh or upgraded Ubuntu/Sunbeam deployments using the packaged Apache/mod_wsgi neutron-api can bring up the API successfully but fail functional tenant network creation. This is not just a CI issue. It affects a normal tenant workflow: openstack network create private or equivalent tenant network creation through the Neutron API. Expected Result Ubuntu’s packaged neutron-api deployment should initialize ML2 default network segment ranges and allow tenant networks to be created. Actual Result Default network segment ranges are not initialized under Apache mod_wsgi, and tenant network creation fails with NoNetworkAvailable. Root Cause Neutron 2026.1 assumes uWSGI runtime metadata in the ML2 segment range initialization path. Ubuntu’s package intentionally deploys neutron-api via Apache mod_wsgi, where import uwsgi fails and get_api_worker_id() returns None. The relevant code is: # neutron/common/wsgi_utils.py def get_api_worker_id() -> int | None: try: import uwsgi return uwsgi.worker_id() except (ImportError, ModuleNotFoundError): return None # neutron/plugins/ml2/managers.py def initialize_network_segment_range_support(self, start_time): if wsgi_utils.get_api_worker_id() != wsgi_utils.FIRST_WORKER_ID: return Proposed Fix Allow non-uWSGI loaders to run segment range initialization while preserving existing uWSGI behavior: worker_id = wsgi_utils.get_api_worker_id() if worker_id is not None and worker_id != wsgi_utils.FIRST_WORKER_ID: return This keeps the existing behavior for uWSGI: - worker 1 initializes - workers other than 1 skip and fixes Apache/mod_wsgi: - missing worker ID no longer causes required initialization to be skipped ** Patch added: "lp2150285_28.0.0-0ubuntu1.1.debdiff" https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/2150285/+attachment/5976507/+files/lp2150285_28.0.0-0ubuntu1.1.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2150285 Title: Neutron 2026.1 with mod_wsgi packaging skips ML2 segment range initialization, causing tenant network creation failures To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/gazpacho/+bug/2150285/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
