rusackas opened a new pull request, #45756: URL: https://github.com/apache/superset/pull/45756
### SUMMARY The Python integration suite runs serially: `scripts/python_tests.sh` calls plain `pytest`, which takes 20 to 28 minutes per database leg (MySQL is the slowest) on a 4 vCPU runner that is mostly idle. Unlike sharding across more runners, which only moves the same CPU work around and adds a setup cost per shard, running the suite on pytest-xdist workers inside the one job should cut the runner minutes themselves, not just the wall clock. This is a draft spike to find out what the suite needs to run in parallel: - `PYTEST_WORKERS` (the three database jobs set it to 4) makes `scripts/python_tests.sh` run `pytest -n <N> --dist loadfile`, so each test file stays on one worker and module-scoped fixtures keep working. - Each xdist worker gets its own metadata database, examples database and Redis database range. The test config derives them from `PYTEST_XDIST_WORKER` (`superset` becomes `superset_gw0`, `superset.db` becomes `superset-gw0.db`, Redis dbs 2/3/4 shift by three per worker), so nothing in the test code changes. - The script migrates and seeds one database per worker (in parallel) before pytest starts; `setup-postgres` and `setup-mysql` in `bashlib.sh` create the per-worker databases, and `celery-worker` starts one Celery worker per xdist worker on that worker's broker database. - A plain `pytest` run, or `PYTEST_WORKERS=1`, keeps today's behaviour exactly. What I expect this run to surface: tests that depend on state another test file left behind, and MySQL tests that touch the shared `sqllab_test_db` and `admin_database` databases (on Postgres those are schemas inside each per-worker database, on MySQL they are still shared). I'll fix or isolate whatever shows up and post the runner-minute numbers against the serial runs before taking this out of draft. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A ### TESTING INSTRUCTIONS CI: the three database legs of the `Python-Integration` workflow run with four workers. Compare their job durations with the serial runs on `master`. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
