ashutosh264 commented on issue #68908: URL: https://github.com/apache/airflow/issues/68908#issuecomment-4802638735
Building on @jonathanleek's analysis and @RNHTTR's workaround suggestion, the use case is clear, but before someone opens a PR, it would help to decide on the scope with a maintainer. **What already exists:** `CloudRunExecuteJobOperator(deferrable=True)` covers **Cloud Run Jobs** (LRO polling via `CloudRunJobFinishedTrigger`). This issue is about **HTTP Cloud Functions / HTTP Cloud Run services**, which is a different execution model. **Why scope matters:** The two usual deferrable patterns don't map cleanly here: 1. **Poll a status handle** - works for Jobs/Dataflow/etc., but HTTP function invoke has no GCP execution ID or status endpoint to poll. 2. **Hold the HTTP connection open** (`HttpTrigger` pattern) - only works when the function blocks until work is done, it doesn't cover fire-and-forget async work where the HTTP response returns before processing finishes (the scenario described in the issue). **Proposed scoping questions for maintainers:** - **Option A - Sync-only:** Add `deferrable=True` to `CloudFunctionInvokeFunctionOperator` (or a deferrable HTTP-to-URL operator) for long-running *synchronous* calls. Document explicitly that async fire-and-forget workloads are out of scope. - **Option B - Status-URL polling:** New operator/trigger that invokes the function, then polls a user-supplied `status_url` (or similar) until completion. Requires agreeing on the completion contract (response shape, auth, terminal states). - **Option C - Document workaround:** As @RNHTTR suggested, point users at `HttpOperator` + deferrable `HttpSensor` (HTTP provider), or invoke + deferrable BigQuery/GCS sensor for custom status stores - without adding a new Google provider primitive. Happy to help implement once we agree which option (or subset) is in scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
