I like the idea! We built something similar where a task would send an email with a report attached (Excel, of course). The user had to either approve the report or flag an issue. If they didn't respond in 2 hours, it escalated to a backup, then to the department lead. If no one responded, the DAG failed. It was pretty effective for business-critical approvals, and we had clear timeouts and fallback logic.
That said, I think this pattern works best when it's used sparingly and with good defaults. Echoing Alexander, human input is inherently unreliable, but sometimes you really do it, whether it's for validating a data quality issue, resolving a content flag, or signing off on something before moving forward. I don't think that breaks the Airflow model as long as: - You can define what happens if no one responds (timeout, fail, escalate, etc.) - There's an API so you're not forcing people into the UI - There's a centralized view of all "waiting for input" tasks to avoid hunting them down It shouldn't become a crutch for manual processes, but I'd love to see first-class support for this kind of interaction. It definitely comes up in the wild. On Tue, May 20, 2025 at 3:46 PM Alexander Shorin <kxe...@apache.org> wrote: > While this is an understandable idea, I think it's wrong to involve humans > to take a part into Airflow workflow. > Run with parameters. Read the logs. Retry. > Blocking the whole pipeline could not be acceptable: humans are not stable > resource for input, they may be sick or on vacation and who will be > operator for this branch? > Better strategy to collect metrics and logs and do post-review of tasks > results. > > > -- > ,,,^..^,,, > > On Tue, May 20, 2025 at 10:15 PM Vikram Koka <vik...@astronomer.io.invalid > > > wrote: > > > Hey everyone, > > > > This is a small technical change, but conceptually it is significant and > > therefore bringing it to the devlist. > > > > As we have been having conversations with Airflow users, who are using > > Airflow for Gen AI applications, there are a couple of features that have > > been brought up as being desirable. As context, based on the most recent > > Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI > > use cases. > > > > One of the desired features is to have "human interaction" within the > > context of the overall orchestration flow. Here are the most common > flows: > > > > 1. Choose a branch in the workflow: This is like a "human branch > > operator", where the human gets to pick one of the paths to move > > forward. > > This is a common pattern for business use cases such as content > > moderation, > > when the automated sentiment analysis is unsure if the content is > > "appropriate" and requires human judgement. > > 2. Approve / Reject: This is a specialized form of the above, where > the > > human approves or rejects the output from the prior tasks, thereby > > triggering a different branch in the flow. > > 3. Provide input: The human is expected to provide textual input, > which > > can then be used for subsequent steps. This is intended to be used > > within > > LLM workflows, where a human supplied context / guidance may be > > critical. > > > > All of these above operations are expected to be performed through the > > Airflow UI. > > > > 4. A distinct variation of the above could be an API interaction for the > > above workflows without using the UI. > > > > > > Technical notes: > > a. While the above tasks are being run (or rather waiting for human > input), > > the DAG will be treated as a continued DAG run. > > > > b. The Airflow UI for these tasks would be a standard UI form similar to > > the Trigger DAG form. > > For flows (1) and (2) above, this would be populated based on the > follow-on > > task options. > > For flow (3) above, this would be a UI form similar to DAG params. > > > > c. Each of the above operations will be treated as a "task". > > The task state while waiting for human input will probably be a variation > > of the "deferred task state". TBD as we get closer to implementation. > > For flow (3), the input data will be passed using XCom to subsequent > tasks. > > > > > > As detailed above, the technical changes are fairly small. Any feedback > is > > welcome. > > > > Best regards, > > Vikram > > -- > > > > Vikram Koka > > Chief Strategy Officer > > Email: vik...@astronomer.io > > > > > > <https://www.astronomer.io/> > > >
