GitHub user potiuk added a comment to the discussion: Thread-Powered Local Executor
This approach has multiple disadvantages. not the least of it is that each task is not isolated from each other and there are multiple problems connected with tasks shared in the same interpreter. Generically solving all the issues connected with it is not really feasible. But if **you** have **your case** where **all your tasks** are run with the code that **your** implementation will make sure to avoid side-effects connected then it might work. But if you notice how many times **you** is mentioned there, you will see that yes - this is great idea for **your own executor** to implement. Which is perfectly fine. Airflow Allows you to implement your own executor and use it, and you are perfectly fine to do so. Whether this can be shared with other users, is debatable. It implies quite many limitations on how DAGs should be implemented by users using it and supporting all the edge cases is likely going to be costly. I think a viable way of approach it is definitely first **you** implement it for your case and test. And battle test it. And figure out limitations that you document. Then - you will see if the limitations are good enough and easy to explain and document, so that average DAG author and DAGs developed by them will run without troubles. THEN the next step is to open a discussion in the devlist - this is where important additions to the community that inevitably lead to maintenance effort from the community side should be discussed. Ideally with experience and findings of you implementing your solution and battle-testing it. For now - I will convert it to a discussion if you need more feedback - but if you get to a stage where thigns are battle-tested and you have enough findings - feel free to start a discussion in devlist GitHub link: https://github.com/apache/airflow/discussions/57699#discussioncomment-14848117 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
