Airflow isn't really designed for tasks to pass data to each other. There are limited capabilities for doing so but Airflow is primarily a workflow management tool, not a dataflow tool.
--George Leslie-Waksman (Clover Health) On Sun, Sep 4, 2016 at 6:01 AM Mohsen Bande <[email protected]> wrote: > it would be nice to have a split (python) operator, which takes a job that > returns a list and streams each item to the specified job. like this > pattern:http://www.workflowpatterns.com/patterns/control/basic/wcp2.php > also, a chunk operator that splits a list to smaller chunks is more general > solution > is there any try on this already? > any help would be highly appreciated >
