gopidesupavan commented on PR #44369: URL: https://github.com/apache/airflow/pull/44369#issuecomment-2516445588
> > Over all LGTM :) > > One question , do we have to re think triggers `default_capacity` config , with the new event driven it shares the triggers capacity config. so is it okay to have a single config for regular triggers and event driver triggers? > > I think so and to me it makes sense. The config `default_capacity` is on the triggerer level and set the maximum number of triggers one triggerer can run. The triggerer runs triggers from deferrable operators but from event driven scheduling as well, therefore, to me it makes sense `default_capacity` covers both use cases. > > The only issue with this approach is, triggers from event driven scheduling can max out the triggerer (if many many triggers are used to update assets). Event driven scheduling would become a noisy neighbour for deferrable tasks. That'd mean that no deferrable task could run unless some of the triggers used to update the assets (all that defined in DAGs) are removed. > > Knowing that the default value of `default_capacity` is 1000, I dont know if this limit is hit often nor if we should treat them separately. What I can suggest is for now to have one common `default_capacity` config for all triggers. If some users, or someone with some data demonstrating we should have two `default_capacity` config (one for deferrable tasks and one for event driven scheduling), then we could do it. Sure, that makes sense. I've heard from a couple of people two or three instances where they run thousands of triggers in their data pipelines, and am sure these people will leverage this awesome event driven feature effectively once it up, I just wanted to bring up that point here. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
