hkc-8010 opened a new pull request, #74684:
URL: https://github.com/apache/airflow/pull/74684
`DbtCloudHook(timeout_seconds=...)` and proxies set on the dbt Cloud
connection have had no effect on the sync API calls since
apache-airflow-providers-http 5.3.2. Requests go out with `timeout=None` and no
proxy, so a dbt Cloud response that never arrives blocks the task forever and
`retry_limit` never fires. We saw a `DbtCloudRunJobOperator` task hang for
about 9 hours on one status request before deferring. Details and a py-spy dump
are in #74678.
Two changes in http 5.3.2 combined to cause this. #51893 made
`HttpHook.run()` pass `self.merged_extra` to `run_and_check()` instead of the
caller's `extra_options`. #51746 made `run_and_check()` take proxies from
`session.proxies`. Both values are set in `HttpHook.get_conn()`, and
`DbtCloudHook.get_conn()` overrides it without calling super. The hook was
still building `extra_options={"timeout": ..., "proxies": ...}` correctly;
nothing read it.
The fix sets `self.merged_extra` and `session.proxies` from the passed
`extra_options` in `DbtCloudHook.get_conn()`. It also gives `get_conn` the same
signature as `HttpHook.get_conn`. I didn't call
`_configure_session_from_extra()` here: it would also copy any leftover dbt
connection extras into the request headers, which would change behaviour for
existing connections. Everything else `run_and_check` reads (stream, verify,
cert, redirects, response checks) comes out the same as it did on http 5.3.1,
because a plain `Session()` already has those defaults. On http versions before
5.3.2, `merged_extra` is ignored and `session.proxies` is used, so the change
is safe there too. The deferrable path uses aiohttp and was never affected.
Tests: the existing timeout tests mock `run_with_advanced_retry`, so they
stop before `HttpHook.run` and could not see this. The new test goes through
`requests_mock` and asserts the timeout and proxy the transport actually
received. It fails on `main` with `-None +60`. Removing either of the two new
lines makes it fail on its own assertion.
`providers/dbt/cloud/tests/unit/dbt/cloud/` passes under Breeze on Python 3.11
(289 passed).
closes: #74678
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
