GitHub user potiuk edited a comment on the discussion: How to modify data before passing it from one task to the other
> Okay, thank you for the answer. This makes using airflow somewhat awkward for > me because now I have to create separate tasks just to add a value to an > object between two other tasks. But at least I understand that now. It's not awkward. This is how distributed system works and it's a basic assumption of how airflow works and it's rather simple when you realise that: a) you can only define pipelines in top level code b) you can only do real work in tasks because the tak is what gets executed on a distributed worker if that assumptions are not good for you .maybe you should consider some other tooling because Airflow is designed to allow you to define pipelines that cna be executed in a distributed way. Maybe you should look for something else if you are trying to do different thing? GitHub link: https://github.com/apache/airflow/discussions/49947#discussioncomment-13059510 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
