dlesco opened a new issue, #37291: URL: https://github.com/apache/airflow/issues/37291
### Apache Airflow Provider(s) http ### Versions of Apache Airflow Providers apache-airflow-providers-http 4.5.2 ### Apache Airflow version 2.6.3 ### Operating System "Ubuntu 20.04.6 LTS" ### Deployment Google Cloud Composer ### Deployment details _No response_ ### What happened For SimpleHttpOperator, if you set deferrable=True (or set in airflow.cfg `[operators]default_deferrable=True`), then how your `data` parameter is handled in `POST` requests changes. I noticed that when we tried using an internal REST service; when the user-agent was python-requests, we got a 201 response; when the user-agent was aiohttp, we got a 400 response. Reading the code in: airflow/providers/http/hooks/http.py the issue is in the API differences of how HttpHook uses requests.Request to post data, versus how HttpAsyncHook uses aiohttp session.post to post data. With requests, it's using the `data` parameter, which is defined as: > data – the body to attach to the request. If a dictionary or list of tuples [(key, value)] is provided, form-encoding will take place. With aiohttp, it's using the `json` parameter for POST requests, which is defined as: > Any json compatible python object (optional). json and data parameters could not be used at the same time. The application we were using was sending json-encoded bytes as the data parameter to the operator. So: - in deferrable=False mode, it sends the json-encoded bytes - in deferrable=True mode, the json-encode bytes will be wrapped inside a json string This would be why we would get a 400 Bad Request response from the REST server, the client sent json-encoded-string of json data, not json data. If the application instead gave a Python dict as the SimpleHttpOperator data parameter: - in deferrable=False mode, it'd be form-encoded - in deferrable=True mode, it'd be json-encoded I haven't setup an environment where I can test all of this, but this is what I see from reading the code. ### What you think should happen instead I don't know how SimpleHttpOperator can be fixed now without breaking the API of SimpleHttpOperator and it's hooks. If people tested their SimpleHttpOperator code on Airflow 2.6.3, and got it working with deferrable set to True, going back now and fixing the Async API to match the Sync API would break their tested code. ### How to reproduce You can test both versions of the POST data APIs by using SimpleHttpOperator with both options of deferrable=False and deferrable=True. ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
