GitHub user pykenny edited a comment on the discussion: Need help understanding total number of Dags oscillating on UI
As for duplicated work on DAG processors, there were some discussion about how to avoid such situation, and I feel like it's still an open question for now: - https://github.com/apache/airflow/issues/32966 (Update) I think distributing workload between multiple DAG processors evenly without having an outside organizer (for instance, scheduler) would end up with a distributed system design problem (e.g. introducing consensus algorithm to decide when to re-distribute job load when one of the processors is down). Otherwise something like randomized sorting or jittering may help with preventing, but not avoiding the worst scenario to happen. GitHub link: https://github.com/apache/airflow/discussions/44495#discussioncomment-12235348 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
