Hey all, We are planning to use airflow for the following purpose. Could you guys please let me know if this is a viable use-case or should we use some other framework instead of Airflow?
We are using the Airflow REST API to hit airflow with lets say an Id. 1. The first task in the DAG will hit a service with that id. 2. The next task will create a subdag which in itself has two tasks. 3. A new instance of the subdag will be scheduled every-time the REST API is hit with a different ID as a parameter and different start and end times. The subdag calls a service to scrape some data. Thus the subdag is essentially being used as a scheduler because we want something that can be scrape data once everyday. Thank you for your help, Swapnesh
