gopidesupavan commented on issue #40528: URL: https://github.com/apache/airflow/issues/40528#issuecomment-2281245271
Thank you @RNHTTR assigning to me, was looking into this, My understand of this ask is. limit total deferrable tasks per dag? Thanks @raphaelauv putting this. To my understand of the triggers and deferrable state, when the any task defers, a new record/row will be created into the trigger tables and once task state is set DEFERRABLE, then the triggerer job picks these deferred tasks from the trigger table and executes in asyncio event loop. The max_active_task verification is done by scheduler job, so no interaction between the scheduler job and trigger job. I would think to limit number of max deferrable tasks by dag below approach. Create new state something like DEFERRED_QUEUE, and when the capacity available(<=max deferrable tasks). then move the state to DEFERRED, and trigger job runner should consider only deferred state tasks to run. Please suggest your thoughts :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
