GitHub user tokoko added a comment to the discussion: run dag-processor as a job, only once
@potiuk sorry for digging up an old discussion. What's you current stance on this? I think this will make a lot of sense once multi-team feature lands. We have been running our stitched-together version of multi-team internally (a central webserver/scheduler/db, plus each team with their own dag-processors and workers). One problem that we came across recently is that some of those "teams" (they are effectively more like projects, to be completely honest) have so small of a workload (a couple of dags) it can hardly justify having a considerable long-running deployments just for them. The solution we're currently experimenting with is to run dag-processor as a k8s job (`-n 1`) on each deploy and launch tasks in k8s executor rather than celery, avoiding the need for running deployments for either a dag processor or a worker. Of course this presupposes that dag code doesn't depend on anything that can be changed w/o a commit in the repo and you don't care about some of the callbacks that are still ran by dag-processor. So far we only came across an issue when scheduler deactivates our dags unless we set `dag_stale_not_seen_duration` config to a huge value. GitHub link: https://github.com/apache/airflow/discussions/36436#discussioncomment-13841768 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
