hkc-8010 opened a new issue, #74678:
URL: https://github.com/apache/airflow/issues/74678
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.1.x (also reproduced on main at eee6e2a4a5)
### What happened and how to reproduce it?
`DbtCloudHook(timeout_seconds=...)` has no effect on the sync API calls. The
request goes out with `timeout=None`, so if dbt Cloud never answers, the task
blocks forever on the socket read. Since nothing raises, `retry_limit` never
kicks in either. We hit this in production with `DbtCloudRunJobOperator`: a
task sat for about 9 hours on a single `GET /api/v2/accounts/<id>/runs/<id>/`
before deferring, even though `hook_params={"timeout_seconds": 60,
"retry_limit": 3}` was set. A py-spy dump of the stuck process showed the value
being lost between the two `HttpHook` methods:
```
run (http/hooks/http.py:419) extra_options: {"timeout": 60}
run_and_check (http/hooks/http.py:471) extra_options: {}
send_kwargs: {"timeout": None, ...}
_make_request (urllib3) read_timeout: None
```
Proxies from the dbt Cloud connection extra (`{"proxies": {...}}`) are
dropped the same way.
Cause: since #51893 (http 5.3.2), `HttpHook.run()` passes
`self.merged_extra` to `run_and_check()`, and `run_and_check()` reads proxies
from `session.proxies`. Both are only populated by `HttpHook.get_conn()`
(through `_configure_session_from_extra`). `DbtCloudHook.get_conn()` overrides
it without calling super, so `merged_extra` stays `{}` and the session has no
proxies. `DbtCloudHook` still builds `extra_options={"timeout": ..., "proxies":
...}` correctly; it is just never used.
The existing tests (`test_timeout_passed_to_run_and_get_response`,
`test_timeout_passed_to_paginate`, `test_timeout_with_proxies`) mock
`run_with_advanced_retry`, so they stop before `HttpHook.run` and could not
catch this.
To reproduce:
```python
import json
from unittest import mock
import requests
from airflow.models.connection import Connection
from airflow.providers.dbt.cloud.hooks.dbt import DbtCloudHook
conn = Connection(
conn_id="dbt", conn_type="dbt_cloud", login="1", password="tok",
extra=json.dumps({"proxies": {"https": "http://proxy.example:3128"}}),
)
hook = DbtCloudHook("dbt", timeout_seconds=60)
hook.__dict__["connection"] = conn
def fake_send(self, request, **kwargs):
print("timeout =", kwargs.get("timeout"), "proxies =",
kwargs.get("proxies"))
resp = requests.Response()
resp.status_code = 200
resp._content = b'{"data": {}}'
return resp
with mock.patch.object(requests.Session, "send", fake_send):
hook.get_job_run(run_id=1, account_id=1)
```
http 5.3.1: `timeout = 60 proxies = {'https': 'http://proxy.example:3128'}`
http 5.3.2+ and main: `timeout = None proxies = {}`
Against a TCP server that accepts and never replies, `timeout_seconds=2,
retry_limit=3` raises `ReadTimeout` after about 9s on http 5.3.1 and hangs
indefinitely on 5.3.2 / 6.2.0.
### What you think should happen instead?
`timeout_seconds` and connection `proxies` should reach
`requests.Session.send`, as they did with http 5.3.1. A timed-out request
should raise `requests.exceptions.ReadTimeout` and be retried per `retry_limit`
(`_retryable_error` already treats `Timeout` as retryable).
### Operating System
Linux (Debian-based Astro Runtime image); repro also runs on macOS
### Deployment
Astronomer
### Apache Airflow Provider(s)
dbt-cloud, http
### Versions of Apache Airflow Providers
apache-airflow-providers-dbt-cloud 4.9.3 (also 4.10.0 and main)
apache-airflow-providers-http 6.2.0 (any version from 5.3.2 on)
### Anything else?
The deferrable trigger path uses aiohttp with its own `ClientTimeout` and is
not affected. Only the sync calls are: trigger, the status check before
deferring, `wait_for_job_run_status`, and cancel.
I'd like to fix this in `DbtCloudHook.get_conn()` (set `merged_extra` and
`session.proxies` from the passed `extra_options`) rather than in `HttpHook`,
since other subclasses such as `DingdingHook` deliberately ignore
`extra_options`.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
