potiuk commented on code in PR #68814: URL: https://github.com/apache/airflow/pull/68814#discussion_r3478023958
########## dev/breeze/doc/ci/04_selective_checks.md: ########## @@ -573,6 +573,21 @@ That's why we do not base our `full tests needed` decision on changes in depende from the `provider.yaml` files, but on `generated/provider_dependencies.json` and `pyproject.toml` files being modified. This can be overridden by setting `full tests needed` label in the PR. +[2] Note on how `providers-test-types-list-as-strings-in-json` is split. + +The provider DB tests run as parallel test-type *groups* (`breeze testing providers-tests +--run-in-parallel`), but each group runs **without xdist** — its tests run serially. To keep all +parallel slots busy, the big "all other providers" group is **not** emitted as one +`Providers[-amazon,celery,google]` monolith (which would be the serial bottleneck while the other slots +idle); instead every provider except the few isolated big ones (`amazon`, `celery`, `google`) is split +into `NUMBER_OF_PARALLEL_PROVIDER_SLICES` balanced chunks (`Providers[a,b,c] Providers[d,e,f] …`). The Review Comment: Actually we already do more optimisation here - and I think this one is really not better- empirically. We are already separating out the "amazon", "google" and "celery" as separate "standalone" test type to run in parallel to achieve most of parallelism - and this works well already <img width="1512" height="187" alt="Screenshot 2026-06-25 at 18 08 40" src="https://github.com/user-attachments/assets/169612e9-4f68-4491-83fe-d5f8487f21ff"; /> And .. looking at actual results - the build runs **slower** with this distribution. Big part of it is that there is a non-zero overhead on settiing up the docker container and collecting tests - so more groups means more overhead - and even if we utilise mutliple processors better, we will not achieve much on two processors. I think we should come back to this when we get bigger machines where we can experiment a bit with the split when we have more processors, but currently - I think that one is a bit of a dead end Currently it's ~ 11 m before and 14-15 m after, so we run **slower**... Closing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
