GitHub user potiuk added a comment to the discussion: Thread-Powered Local Executor
Also. I think you might have an alternative solution - use Celery workers. Celery can be configure to provide you with 4 (and even 5) ways of running the workers: https://docs.celeryq.dev/en/latest/userguide/concurrency/index.html > * prefork: The default option, ideal for CPU-bound tasks and most use cases. > It is robust and recommended unless there’s a specific need for another model. > * eventlet and gevent: Designed for IO-bound tasks, these models use > greenlets for high concurrency. Note that certain features, like > soft_timeout, are not available in these modes. These have detailed > documentation pages linked below. > * solo: Executes tasks sequentially in the main thread. > * threads: Utilizes threading for concurrency, available if the > concurrent.futures module is present. > * custom: Enables specifying a custom worker pool implementation through > environment variables. All you need is to setup a redis broker to distribute tasks in memory and have a celery worker with as many concurrent workers with whatever concurrency you choose. There is little point to implement those concurrency methods in the Local executor if you can get them this way - and it has also the advantage that the celery worker will be even better separated from scheduler and any "side-effect" because of side-effects between running tasks can be solved by just restarting celery GitHub link: https://github.com/apache/airflow/discussions/57699#discussioncomment-14861060 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
