johnwarburton commented on issue #27399:
URL: https://github.com/apache/airflow/issues/27399#issuecomment-1316487782
Can confirm when trying to schedule a DAG exactly one with
`CronTriggerTimetable` airflow [v2.4.2] in **standalone mode** (where
`dag_dir_list_interval = 300` seems to be ignored)
```
import subprocess
from airflow import DAG
from airflow.decorators import dag, task
from airflow.timetables.trigger import CronTriggerTimetable
from datetime import datetime
@task
def mytask():
bash_command=f'/bin/echo run once'
EXEC_OUT=(subprocess.check_output(bash_command,
shell=True)).decode('utf-8')
@dag(
dag_id=f'run_once',
start_date=datetime(2022,1,1),
timetable=CronTriggerTimetable('03 18 16 11 *',
timezone='Australia/Sydney'),
catchup=False,
)
def run_once_dag():
mytask()
run_once_dag()
```
In the minute before 18:03, the DAG was processed 22 times
The first processing at 18:03, the `next_dagrun` was set to the same time
one year later, without executing the DAG
```
[2022-11-16T18:02:50.448+1100] {processor.py:154} INFO - Started process
(PID=11018) to work on /export/home/jwarburt/airflow-venv/dags/run-once.py
[2022-11-16T18:02:50.506+1100] {logging_mixin.py:120} INFO -
[2022-11-16T18:02:50.506+1100] {dag.py:3336} INFO - Setting next_dagrun for
run_once to 2022-11-16T07:03:00+00:00, run_after=2022-11-16T07:03:00+00:00
[2022-11-16T18:02:50.609+1100] {processor.py:176} INFO - Processing
/export/home/jwarburt/airflow-venv/dags/run-once.py took 0.164 seconds
[2022-11-16T18:02:53.836+1100] {processor.py:154} INFO - Started process
(PID=11026) to work on /export/home/jwarburt/airflow-venv/dags/run-once.py
[2022-11-16T18:02:53.894+1100] {logging_mixin.py:120} INFO -
[2022-11-16T18:02:53.894+1100] {dag.py:3336} INFO - Setting next_dagrun for
run_once to 2022-11-16T07:03:00+00:00, run_after=2022-11-16T07:03:00+00:00
[2022-11-16T18:02:54.004+1100] {processor.py:176} INFO - Processing
/export/home/jwarburt/airflow-venv/dags/run-once.py took 0.171 seconds
[2022-11-16T18:02:59.105+1100] {processor.py:154} INFO - Started process
(PID=11032) to work on /export/home/jwarburt/airflow-venv/dags/run-once.py
[2022-11-16T18:02:59.163+1100] {logging_mixin.py:120} INFO -
[2022-11-16T18:02:59.163+1100] {dag.py:3336} INFO - Setting next_dagrun for
run_once to 2022-11-16T07:03:00+00:00, run_after=2022-11-16T07:03:00+00:00
[2022-11-16T18:02:59.272+1100] {processor.py:176} INFO - Processing
/export/home/jwarburt/airflow-venv/dags/run-once.py took 0.171 seconds
[2022-11-16T18:03:00.699+1100] {processor.py:154} INFO - Started process
(PID=11034) to work on /export/home/jwarburt/airflow-venv/dags/run-once.py
[2022-11-16T18:03:00.845+1100] {logging_mixin.py:120} INFO -
[2022-11-16T18:03:00.845+1100] {dag.py:3336} INFO - Setting next_dagrun for
run_once to 2023-11-16T07:03:00+00:00, run_after=2023-11-16T07:03:00+00:00
[2022-11-16T18:03:00.865+1100] {processor.py:176} INFO - Processing
/export/home/jwarburt/airflow-venv/dags/run-once.py took 0.169 seconds
```
If I could slow down the processing interval, that might help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
