mikebridge opened a new issue, #45362: URL: https://github.com/apache/superset/issues/45362
### What's happening Since 2026-10-12 the `Python-Unit` → `unit-tests (current)` job intermittently dies at the very end of the suite. The step reaches `[ 99%]`, then the log ends with `The runner has received a shutdown signal` / `The operation was canceled`, and the job lingers ~26–28 minutes before being cancelled. All tests that ran had passed. From the Actions API: 50 such kills against ~500 successful runs between Oct 12 and Oct 15, including `master` (e.g. runs 38502343489 and 38563726398 on Oct 14). Zero before Oct 12. Some branches were hit repeatedly. Example PR runs: https://github.com/apache/superset/actions/runs/38619149658, https://github.com/apache/superset/actions/runs/38752136752 (both attempts). ### Root cause Each xdist worker's memory grows by ~10 MB per test module and reaches 5–7 GB per worker by the end of the suite (857 modules). Two workers plus the controller don't fit on the 16 GB hosted runner. The unit `app` fixture is module-scoped (`tests/unit_tests/conftest.py`) and builds a full `SupersetApp` per module. Those apps are never freed (228 live apps after 110 modules, measured with `gc`), because two module-level singletons keep references to every app ever created: 1. `SupersetAppInitializer.configure_celery()` subclasses whatever `celery_app.Task` currently is and replaces it. Each init wraps the previous wrapper, and each `AppContextTask` closes over its app: https://github.com/apache/superset/blob/e0e728ef7644451a8b7162a7130175fdadef1571/superset/initialization/__init__.py#L145-L171 2. `init_views()` adds menu items whose lambdas reference the initializer/app; the fixture resets `appbuilder.baseviews` but not `appbuilder.menu`, so menu items accumulate per app. The defect is old (the Celery wrapping is from 2021, the module-scoped fixture from 2022). It started failing now because the suite grew quickly (about 776 → 856 modules and 21.5k → 27.5k tests between Oct 3 and Oct 15), and the leak costs per module. Large transient tests add peaks on top (e.g. `test_validation_extraction_is_bounded[100_000-*]` allocates 1.5–2.6 GB for over a minute). Locally, making the `app` fixture session-scoped took retained growth from ~10 MB per module to about zero. ### Fix I'm working on a PR (will link it here): - session-scope the unit `app` fixture (or tear it down explicitly); - make `configure_celery` subclass the original `celery_app.Task` instead of the current one, so re-initialization replaces rather than chains (no change for a single init in production); - reset `appbuilder.menu` in the fixture; - a regression test that initializing twice doesn't grow the task class chain and that the first app can be garbage-collected. Stopgaps if needed before that lands: split the unit job across 2–3 runners, or reduce the largest parametrized case in `test_validation_extraction_is_bounded`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
