kaxil opened a new pull request, #73478: URL: https://github.com/apache/airflow/pull/73478
These tests exercised retry and polling code with the production backoff left in place, so each one spent 5 to 30 seconds sleeping. In the `Non-DB tests: providers` job on a recent main run, 22 of the 100 slowest tests were pure waits of this kind, about three minutes of wall clock per run. The provider compatibility jobs run each test type single-process, so the same waits land directly on their critical path. | Test | Before | Cause | |---|---|---| | snowflake `test_snowflake_sql_api.py`, 2 retry-exhaustion tests | 15s each | tenacity `wait_exponential`, 1+2+4+8s | | amazon `test_glue.py`, 2 retry-exhaustion tests | 15s each | same | | dbt `test_dbt.py::test_get_job_details_retry_with_retryable_errors[aiohttp_429]` | 15s | `retry_delay=0.1` only sets the floor of `wait_exponential`, so 5 attempts still wait 15s | | google `test_dataprep.py`, 2 `get_job_group_status` tests | 15s | patched `run_job_group.retry.sleep` while calling `get_job_group_status` | | apache.spark `test_poll_k8s_driver_succeeds` | 20s | 20s minimum poll interval | | amazon `test_ecs.py`, 2 tests with awslogs configured | 10s each | operator sleeps 10s before describing the task to find a container name | | microsoft.azure `test_powerbi.py` refresh-exception test | 10s | tenacity backoff around the status fetch | | cncf.kubernetes `test_await_xcom_sidecar_container_timeout` | 10s | real 1s poll up to a 10s timeout | | amazon `test_ec2.py`, 4 operator tests | 7 to 8.6s | hook polls with a 15s interval; moto flips state on the next describe | | amazon `test_batch_client.py`, `test_batch_waiters.py` | 6s each | thread mocks given `side_effect=lambda: time.sleep(2)` | | keycloak routes fixture | 5.8 to 8.6s setup | `create_app()` called twice per test | | microsoft.azure `test_wasb.py`, 2 tests | 5.5s each | test waits `POKE_INTERVAL + 0.5` with a 5s interval | Where the hook already exposes its retry config (`api_retry_args` on the Snowflake and Glue hooks, `retry_args` on the dbt hook) the tests pass `wait_none()`, so the attempt count under test is unchanged and only the wait goes. Elsewhere the sleep is patched, matching what neighbouring tests in the same files already do. Every test still asserts the same call counts and outcomes. After the change all of these run in under 2 seconds, and the EC2 tests in 1 to 2 seconds. Left alone: the two EKS `[200]` pagination tests at 23s each create 200 resources through moto rather than sleeping, and the mongo tests start a real container. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
