dominik-d-stampli commented on issue #47398: URL: https://github.com/apache/airflow/issues/47398#issuecomment-2706350029
Hey, I'm trying to create an event driven architecture in Airflow. Maybe I misunderstood the [documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html#) and this [video](https://www.youtube.com/watch?v=bk9NStntfi0), but I thought that Airflow is able to trigger DAGs based on incoming dataset events. That it can decide if it wants to batch them into a single DAG execution, or run multiple DAG executions each with a subset of events. That way I could send events to a Dataset either from other DAGs or from external systems using Airflow API, and have Airflow process all those events utilizing available capacity. But I was surprised to see that pending dataset events don't always trigger downstream DAG, they just hang there and are picked up only after more events are sent. I'm not sure I understand what is the value of such an approach? Is this by design? Or is it an unexpected bug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
