gtxu commented on code in PR #70960:
URL: https://github.com/apache/airflow/pull/70960#discussion_r4019694949
##########
providers/http/src/airflow/providers/http/hooks/http.py:
##########
@@ -52,6 +55,19 @@ def _url_from_endpoint(base_url: str | None, endpoint: str |
None) -> str:
return (base_url or "") + (endpoint or "")
+def _select_srv_target(answers: Iterable[SRV]) -> tuple[str, int]:
+ """Select a target host and port from resolved DNS SRV records."""
+ candidates_by_priority: dict[int, list[SRV]] = {}
+ for record in answers:
+ candidates_by_priority.setdefault(record.priority, []).append(record)
+ # RFC 2782 priority failover; weight is not honored, ties broken uniformly
at random.
+ candidates = candidates_by_priority[min(candidates_by_priority)]
Review Comment:
Thank you for the review!
I have reworked it and so failover happens within a single request instead
of depending on retries:
1. `_order_srv_targets` returns every target in RFC 2782 order: priority
ascending, then weighted random within a same priority. Weight is now honored.
2. `HttpHook.run` and `AsyncHttpSession.run` try each target in turn. They
move to the next one only when a connection can not be opened by error:
`NewConnectionError` / `ConnectTimeoutError`, or `aiohttp.ClientConnectorError`
/ `ConnectionTimeoutError`. Errors after the connection is open (for example a
server disconnect) and HTTP error responses are raised as before, so a
non-idempotent request is never sent twice. If every SRV target fails, the last
error is raised.
This works without `run_with_advanced_retry` as well, and the async hook's
retry loop now runs once per target.
Tests also cover the case for both hooks, the order of targets, errors that
must not trigger failover, and every target being unreachable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
