adrianrego opened a new issue, #37664: URL: https://github.com/apache/airflow/issues/37664
### Apache Airflow version 2.8.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When creating a deferrable HTTPSensor with no connection and full url for endpoint an invalid url is constructed: ``` HttpSensor(endpoint="https://someurl.com";, http_conn_id="", deferrable=True) ``` URL poked via HttpHookAsync is: `/http://someurl.com` This is due to the assumption that base_url is passed into the hook or pulled from an http connection: https://github.com/apache/airflow/blob/main/airflow/providers/http/hooks/http.py#L362 This is not an issue with HttpHook as it handles the base where base_url is None or endpoint is a complete url. https://github.com/apache/airflow/blob/main/airflow/providers/http/hooks/http.py#L264 Traceback: ``` [2024-02-22 19:19:47,225] Trigger failed: Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 529, in cleanup_finished_triggers result = details["task"].result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/airflow/jobs/triggerer_job_runner.py", line 601, in run_trigger async for event in trigger.run(): File "/opt/conda/lib/python3.11/site-packages/airflow/providers/http/triggers/http.py", line 183, in run await hook.run( File "/opt/conda/lib/python3.11/site-packages/airflow/providers/http/hooks/http.py", line 383, in run response = await request_func( ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/aiohttp/client.py", line 545, in _request req = self._request_class( ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 326, in __init__ self.update_host(url) File "/opt/conda/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 400, in update_host raise InvalidURL(url) aiohttp.client_exceptions.InvalidURL: /https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20240222/conus/hrrr.t18z.wrfsfcf25.grib2 ``` ### What you think should happen instead? HttpHookAsync should construct the url in a similar fashion to HttpHook and support a complete url via the `endpoint` kwarg. The original Astronomer HttpHookAsync also had similar behavior: https://github.com/astronomer/astronomer-providers/blob/1.18.4/astronomer/providers/http/hooks/http.py#L103 ### How to reproduce ``` HttpSensor(endpoint="https://someurl.com";, http_conn_id="", deferrable=True) ``` ### Operating System All ### Versions of Apache Airflow Providers apache-airflow-providers-http==4.9.1 ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
