github-actions[bot] opened a new pull request, #74951: URL: https://github.com/apache/airflow/pull/74951
* Add a test for asset-triggered runs with use_job_schedule off The option is documented as turning off the scheduler's use of cron intervals, with manually triggered runs still running. Asset events are not cron intervals, so a queued event should still produce a run when the option is off. Today it does not: the event stays queued forever. * Create asset-triggered Dag runs when use_job_schedule is off The option is documented as turning off the scheduler's use of cron intervals, but its check wrapped all automatic run creation, so asset events and partitioned-asset events stopped producing runs too and stayed queued. With the option off, a Dag that is due by its schedule never gets a run, so it stays due. Were such Dags still selected, they would take the per-loop slots and keep their row locks on every loop, and on PostgreSQL, which sorts the asset-triggered Dags' empty next_dagrun_create_after last, they could keep those Dags out of every batch. They are left out of the query instead. The best-practices page offered the option as a way to stop all new Dag runs during maintenance, which is no longer true for asset-triggered runs. * Document use_job_schedule in the config reference and pin its SQL filter The config.yml description still said the option turns off cron intervals and that only manually triggered Dags keep running. It now says the scheduler stops creating runs from timetables while manual and asset-triggered runs are still created. The include_scheduled predicate is the only enforcement of the option, so build it before the query and say so in the comment. The test now limits the batch to one Dag so a filter applied after the query would fail it on Postgres, where the asset Dag's NULL sorts last. * Document that use_job_schedule=False also stops asset-triggered Dag runs Reviewers want the option to keep stopping every run the scheduler creates by itself, because best-practices.rst recommends it for maintenance windows and the draining work follows the same rule. The config description predates assets and only mentioned cron intervals, so it described the behaviour wrongly. Revert the scheduler change and its tests, and fix the description instead. (cherry picked from commit 3f03b18920f3a8aadb601e979c7e18018b89c99d) Co-authored-by: Lucas Pimentel <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
