Vamsi-klu opened a new pull request, #69748:
URL: https://github.com/apache/airflow/pull/69748
## Problem
HttpOperator(deferrable=True) defers immediately without making request in
worker. Actual POST happens inside HttpTrigger.run() in triggerer. When
triggerer restarts, trigger re-instantiated from serialize() and run() called
again -> duplicate POST.
Default method POST makes out-of-box vulnerable. Silent data duplication.
Code:
- Operator execute_async defers HttpTrigger with method=self.method
- Trigger run() does aiohttp request in triggerer
- BaseTrigger docs say triggers must assume persistence, run() may be
re-executed after restart
## What I did
Short-term warning (reviewer-friendly, per issue suggestion):
- Add IDEMPOTENT_METHODS = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE",
"TRACE"} per RFC 7231
- In HttpOperator.execute_async(): if method.upper() not in
IDEMPOTENT_METHODS, emit UserWarning referencing #67945, guiding to idempotent
methods or HttpSensor/EventSensor polling
- Same warning in paginate_async path
- In HttpTrigger.__init__(): same warning, catches direct trigger usage;
HttpEventTrigger inherits via super
- Suppress duplicate warning from trigger when operator already warned (via
catch_warnings filter)
Long-term fix (documented as follow-up, not implemented):
- Execute non-idempotent requests in worker, only poll in trigger (Airbyte
pattern)
- Or deprecate deferrable for HttpOperator, recommend HttpSensor
## Impact
- Surfaces silent data duplication risk to users
- Default POST now warns out-of-box (good, surfaces risk)
- No breaking change, only warning
- Guides users to safe patterns
## Testing
- uv run --project providers-http pytest
providers/http/tests/unit/http/operators/test_http.py -k async → 4 passed (now
warns for POST, expected)
- triggers → 11 passed, full http package 103 passed 1 skipped
- Manual:
- POST/PATCH warns with "may send duplicate"
- GET/PUT/DELETE/HEAD/OPTIONS/TRACE no warn
Fixes: #67945
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Muse Spark 1.1
Generated-by: Muse Spark 1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
