anishgirianish opened a new pull request, #62343:
URL: https://github.com/apache/airflow/pull/62343
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
## Summary
Follows the direction proposed by @potiuk in #59643 to move connection
testing off the API server and onto workers.
Connection testing has been disabled by default since Airflow 2.7.0 because
executing user-supplied driver code (ODBC/JDBC) on the API server poses
security risks, and workers typically have network access to external systems
that API servers don't — making API-server results misleading.
This PR adds async connection testing that dispatches tests to workers via
the ExecutorCallback infrastructure:
- **`POST /connections/test-async`** : queues a connection test, returns a
crypto-random polling token (`secrets.token_urlsafe`). Connection must already
be saved; only `connection_id` is sent, workers fetch credentials
via `get_connection_from_secrets`.
- **`GET /connections/test-async/{token}`** : polls for results. Token
knowledge serves as authorization.
- **`PATCH /execution/connection-tests/{id}`** : execution API endpoint for
workers to report results back.
- **`ConnectionTest` model + migration** : state tracking (`PENDING` →
`QUEUED` → `RUNNING` → `SUCCESS`/`FAILED`), callback FK, and `db clean` support
via `created_at`.
The scheduler dispatch (picking up QUEUED ExecutorCallbacks and sending them
to executors) depends on #61153. Once that merges `ExecuteCallback.make()` will
be decoupled from `DagRun`, connection tests will execute end-to-end.
## References
- Dev mailing list discussion: [[DISCUSS] Move connection testing to
workers](https://www.mail archive.com/[email protected]/msg21118.html)
- Builds on Callback infrastructure from #54796
- Depends on executor dispatch from #61153 for end-to-end flow
## Test plan
- [x] Unit tests for `ConnectionTest` model (token generation, state
transitions)
- [x] Unit tests for `run_connection_test` worker function (success,
failure, exception)
- [x] Unit tests for core API endpoints (POST 202, GET polling, 403
disabled, 404 not found)
- [x] Unit tests for execution API endpoint (PATCH 204, 404, 409 conflict,
422 validation)
- [x] Manual testing against PostgreSQL via Breeze (both core API and
execution API)
- [x] Migration verified (`0105_3_2_0_add_connection_test_table`)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
