GitHub user pykenny edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
As for duplicated work on DAG processors, there were some discussion about how to avoid such situation, and I feel like it's still an open question for now: - https://github.com/apache/airflow/issues/32966 (Update) I think distributing workload between multiple DAG processors evenly without having an outside organizer (for instance, scheduler) would end up with a distributed system design problem (e.g. introducing consensus algorithm). Otherwise something like randomized sorting or jittering may help with preventing, but not avoiding the worst scenario. GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-12235348 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
