Hey everyone, This is a small technical change, but conceptually it is significant and therefore bringing it to the devlist.
As we have been having conversations with Airflow users, who are using Airflow for Gen AI applications, there are a couple of features that have been brought up as being desirable. As context, based on the most recent Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI use cases. One of the desired features is to have "human interaction" within the context of the overall orchestration flow. Here are the most common flows: 1. Choose a branch in the workflow: This is like a "human branch operator", where the human gets to pick one of the paths to move forward. This is a common pattern for business use cases such as content moderation, when the automated sentiment analysis is unsure if the content is "appropriate" and requires human judgement. 2. Approve / Reject: This is a specialized form of the above, where the human approves or rejects the output from the prior tasks, thereby triggering a different branch in the flow. 3. Provide input: The human is expected to provide textual input, which can then be used for subsequent steps. This is intended to be used within LLM workflows, where a human supplied context / guidance may be critical. All of these above operations are expected to be performed through the Airflow UI. 4. A distinct variation of the above could be an API interaction for the above workflows without using the UI. Technical notes: a. While the above tasks are being run (or rather waiting for human input), the DAG will be treated as a continued DAG run. b. The Airflow UI for these tasks would be a standard UI form similar to the Trigger DAG form. For flows (1) and (2) above, this would be populated based on the follow-on task options. For flow (3) above, this would be a UI form similar to DAG params. c. Each of the above operations will be treated as a "task". The task state while waiting for human input will probably be a variation of the "deferred task state". TBD as we get closer to implementation. For flow (3), the input data will be passed using XCom to subsequent tasks. As detailed above, the technical changes are fairly small. Any feedback is welcome. Best regards, Vikram -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
