wolfier commented on issue #40435: URL: https://github.com/apache/airflow/issues/40435#issuecomment-2274527378
> does each deferred task start a new process? That's correct. Each task execution will spawn a new process. A deferrable operator task will be executed twice. Once to submit the trigger to the triggerer and another time to process the trigger event. The process should be fairly short running though as you can see with the successful attempt. I am not sure how MWAA is setup but the triggerer and worker should be running in two different execution spaces. This means the number of concurrent triggers should not affect the worker. > Does the scheduler throttle / limit the number of deferred tasks that are running at the same time? The scheduler does not but the triggerer does have a [limit of triggers per triggerer](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-capacity). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
