Hi. I have this problem where I have a queried data report for let's say 100.000 users. The next task will give it to multiple worker, and we might want to change how many worker handle it (usually we have 10 worker but let's say the management in hurry hence we increase the worker to 50). How would I handle it in airflow?
Usually, before we used airflow we breakdown the 100.000 users into 1000 batches of 100 users, and push it to sqs (queue), and worker will pull it from the queue. So if we increase the worker it doesn't matter because it's only pulling from the sqs. It should be simple, but I haven't been able to grasp airflow pattern yet. -- *Adinata* TOKI 2009 SMAN Plus Provinsi Riau 9th 13509022 - Informatika ITB 2009 Engineer - UrbanIndo.com
