It sounds like you would need to dynamically create tasks based on some initial input.
Quoting Maxime Beauchemin from another thread: We don't really support mutating a DAG's shape or structure based on source > parameters. Think about it: it would be hard to fit the current paradigm of > the UI like representing that DAG in the tree view. We like to think of > DAGs as pretty static or "slowly changing", similar to how a database > physical model evolves in the lifecycle of an application or a data > warehouse (at a similar rhythm). For those use cases (where input > parameters would change the shape of the DAG), we think of those as > different "singleton" DAGs that are expected to run a single time. To get > this to work, we create a "DAG factory" as a python scripts that outputs > many different DAG objects (with different dag_ids) and where > `schedule_interval='@once'` based on a config file or something equivalent > (db configuration, airflow.models.Variable object, ...). Assuming that you are able to dynamically create tasks and that you are using the celery executor, you could assign each task to a particular queue and then spin up workers when you feel that you need something to be run faster. On Wed, Jul 20, 2016 at 8:08 PM, Adinata <[email protected]> wrote: > Hi. I have this problem where I have a queried data report for let's say > 100.000 users. The next task will give it to multiple worker, and we might > want to change how many worker handle it (usually we have 10 worker but > let's say the management in hurry hence we increase the worker to 50). How > would I handle it in airflow? > > Usually, before we used airflow we breakdown the 100.000 users into 1000 > batches of 100 users, and push it to sqs (queue), and worker will pull it > from the queue. So if we increase the worker it doesn't matter because it's > only pulling from the sqs. It should be simple, but I haven't been able to > grasp airflow pattern yet. > -- > *Adinata* > TOKI 2009 > SMAN Plus Provinsi Riau 9th > 13509022 - Informatika ITB 2009 > Engineer - UrbanIndo.com >
