argibbs commented on code in PR #25489:
URL: https://github.com/apache/airflow/pull/25489#discussion_r936094863
##########
newsfragments/25489.bugfix.rst:
##########
@@ -0,0 +1 @@
+DAGProcessorManager queue made fully FIFO, split into two (priority and std),
and dags guaranteed to be read from disk periodically (fixes/improves SLA alert
reliability)
Review Comment:
I suspect "bugfix" undersells the change. As per PR description, will
happily change the classification to whatever I'm told is the right one to use
for a change like this.
##########
airflow/config_templates/config.yml:
##########
@@ -1913,6 +1913,21 @@
type: string
example: ~
default: "30"
+ - name: max_file_process_interval
+ description: |
+ Number of seconds since the previous parse time after which a DAG file
will be parsed
+ again. This is only useful if your system contains a lot of dags /
slas which update
+ exceedingly frequently, as this may prevent the system from scanning
the file system
+ for changes to your dag files. This operates on a best efforts basis;
if you have
+ many dags, and it takes 10 minutes to scan and parse them all, this
will not make
+ it parse any more often.
+ Setting this to <= 0 disables the behaviour, in case it's important to
you that those
+ frequently updating dags / slas always take priority at the cost of
delaying updates
+ from disk
+ version_added: 2.4.0
Review Comment:
I mean, I'm assuming that this PR will be "subject to discussion" to put it
mildly and has no chance of making 2.3.4. If it sails through, I'll happily
correct this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
