GitHub user HuanjieGuo edited a discussion: Auto Routing trigger instances to different triggerer
Currently, the capacity is set to default 1000 in TriggererJobRunner. https://github.com/apache/airflow/blob/main/airflow/jobs/triggerer_job_runner.py#L264 When a batch of tasks are triggered at the same time, these trigger jobs may be sent to only one triggerer and other would do nothing. It would be better if we write a logic here to calculate how many trigger jobs a triggerer should get from DB here and let other triggerers have a chance to dispatch the trigger jobs. https://github.com/apache/airflow/blob/main/airflow/jobs/triggerer_job_runner.py#L398 My initial idea is to change it like this ``` def load_triggers(self): recommend_capacity = total_trigger_job_cnt / active_triggerer_count + 1 instance_capacity = math.min(self.capacity, recommend_capacity) Trigger.assign_unassigned(self.job.id, instance_capacity, self.health_check_threshold) ids = Trigger.ids_for_triggerer(self.job.id) self.trigger_runner.update_triggers(set(ids)) ``` @potiuk is this a point to contribute? I want to submit a PR for it. GitHub link: https://github.com/apache/airflow/discussions/44211 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
