potiuk commented on issue #57487: URL: https://github.com/apache/airflow/issues/57487#issuecomment-3476859533
This approach has multiple disadvantages. not the least of it is that each task is not isolated from each other and there are multiple problems connected with tasks shared in the same interpreter. Generically solving all the issues connected with it is not really feasible. But if **you** have **your case** where **all your tasks** are run with the code that **your** implementation will make sure to avoid side-effects connected then it might work. But if you notice how many times **you** is mentioned there, you will see that yes - this is great idea for **your own executor** to implement. Which is perfectly fine. Airflow Allows you to implement your own executor and use it, and you are perfectly fine to do so. Whether this can be shared with other users, is debatable. It implies quite many limitations on how DAGs should be implemented by users using it and supporting all the edge cases is likely going to be costly. I think a viable way of approach it is definitely first **you** implement it for your case and test. And battle test it. And figure out limitations that you document. Then - you will see if the limitations are good enough and easy to explain and document, so that average DAG author and DAGs developed by them will run without troubles. THEN the next step is to open a discussion in the devlist - this is where important additions to the community that inevitably lead to maintenance effort from the community side should be discussed. Ideally with experience and findings of you implementing your solution and battle-testing it. For now - I will convert it to a discussion if you need more feedback - but if you get to a stage where thigns are battle-tested and you have enough findings - feel free to start a discussion in devlist -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
