GitHub user TimofeyPil created a discussion: Apache Airflow Scheduler creates DAG Runs before their scheduled execution time — tasks stuck with "Execution date is in the future"
Hello everyone, I'm experiencing a serious and unexpected scheduling issue with Apache Airflow, and I would really appreciate some help identifying the root cause and finding a safe solution. The issue appeared on October 15, 2026, and has been observed in two separate Airflow environments. **The main problem is that Airflow creates scheduled DAG Runs before their expected execution time.** These DAG Runs immediately enter the `running` state, even though their scheduled execution time has not yet arrived. However, the tasks inside these DAG Runs do not execute. Instead, they remain in the `None` state because Airflow reports: **"Execution date is in the future."** Once the scheduled time arrives, the tasks execute normally and complete within seconds. This behavior is unexpected because the DAG Run may appear to be running for several minutes or even over an hour, while no actual task execution is taking place. All affected runs have `Run Type = scheduled`, not `manual`. ## 1. Exact error message In the Airflow UI, under Task Instance → Task Failed Dependencies, I observed: `Execution date 2026-10-15T13:20:00+00:00 is in the future (the current date is 2026-10-15T13:14:39+00:00).` At that moment: - The DAG Run was already in the `running` state. - The first task was in the `None` state. - No task code had started executing. - The Scheduler had apparently created the DAG Run before the execution date became valid. After the specified time was reached, the tasks executed successfully. I am trying to understand why the Scheduler creates these future DAG Runs in the first place. ## 2. Example with a frequently scheduled DAG One of the affected DAGs is configured to run every five minutes during a defined daily time window. Previously, each DAG Run completed within a few seconds. After thousands of successful historical runs, the following behavior suddenly appeared: 1. A scheduled DAG Run was created before its expected execution time. 2. The DAG Run immediately entered `running`. 3. Its tasks remained in `None`. 4. The dependency check reported `Execution date is in the future`. 5. Additional scheduled runs began accumulating. 6. Some DAG Runs appeared to have execution durations of more than an hour, despite their actual tasks taking only seconds. When the expected execution time arrived, the tasks normally completed successfully. ### Controlled experiment I also tested the behavior by pausing and unpausing the DAG. At approximately 16:41, I unpaused it. Airflow then: 1. Processed a scheduled run associated with a data interval ending at 16:40. 2. Immediately processed another run associated with a data interval ending at 16:45. 3. Subsequently created another future scheduled DAG Run that remained in `running` while its tasks were blocked. The same problem continued after reparsing the DAG. ## 3. Similar behavior in another Airflow environment The issue was also observed in another Airflow environment with different DAGs and scheduling frequencies. Several scheduled executions had completed normally earlier in the day. However, later executions were unexpectedly created ahead of their expected schedule. For example: - One scheduled run associated with 18:15 appeared at approximately 15:18. - Another run associated with 17:00 appeared at approximately 13:54. - Earlier runs of those workflows had executed normally. These times are based on observations in the Airflow UI. I have not yet fully verified the relationship between the displayed timezone, logical date, data interval, and actual run-after time. The fact that a similar issue appeared in two environments makes me concerned about a common infrastructure or scheduling-related cause. 4. Recent changes before the incident Before the issue appeared, I had been deploying updates to several DAGs through CI/CD. The changes primarily involved adding standardized task and DAG lifecycle logging using normal Apache Airflow callback mechanisms. The callbacks record events such as: - Task execution started - Task succeeded - Task failed - Task retry scheduled - DAG started, succeeded, or failed These callbacks are intended only for monitoring. They do not intentionally create DAG Runs, modify schedules, change task dependencies, or alter scheduling metadata. There were multiple deployments during the day. I am wondering whether a deployment could have coincided with active DAG execution or caused a restart of some Airflow components. However, **I have not confirmed whether any Scheduler or Worker restart actually occurred.** The logging changes and deployment activity are possible contributing factors, not confirmed causes. ## 5. Troubleshooting already performed I have tried: 1. Pausing and unpausing the affected DAGs. 2. Waiting until after the scheduled time before unpausing. 3. Marking obsolete DAG Runs as failed. 4. Using the Airflow UI's Reparse DAG action. 5. Reviewing task logs and dependency checks. 6. Reviewing the scheduling configuration. 7. Verifying that the actual tasks execute quickly and successfully once they become eligible. 8. Limiting one affected DAG to a single active DAG Run. 9. Adding a DAG Run timeout to one affected DAG. None of these changes resolved the premature creation of scheduled DAG Runs. Importantly, the same issue has occurred in DAGs without the newly added timeout. I have not deleted or manually modified records in the Airflow metadata database. ## 6. Potentially related Apache Airflow issue I found this GitHub issue: https://github.com/apache/airflow/issues/17481 It describes similar symptoms involving future execution dates and unexpected Scheduler behavior. The discussion suggests that clock synchronization between Airflow components and the metadata database may be relevant. I have not yet been able to verify system clock synchronization in my environments. ## 7. Questions I would really appreciate clarification on the following: **1. Why would the Airflow Scheduler create a scheduled DAG Run and immediately mark it as `running` before its execution date or expected run-after time?** **2. Is this a known Scheduler issue in certain Apache Airflow versions?** **3. Could system clock differences between the Airflow Scheduler, Workers, and metadata database server cause exactly this behavior?** **4. Could a CI/CD deployment or restart of Airflow components during an active DAG execution trigger this issue?** **5. Could logging callbacks indirectly contribute to scheduling problems through resource contention or increased Scheduler workload?** **6. How can I determine whether the Scheduler incorrectly calculated the next execution time or incorrectly created a DAG Run before its valid run-after time?** **7. Which specific Scheduler logs, metadata fields, or diagnostic commands would help identify the root cause?** **8. Is there a safe recovery method that does not require deleting DAG history, recreating DAGs, or manually modifying the Airflow metadata database?** 9. If the problem is caused by clock synchronization or an infrastructure restart, will correcting the environment automatically resolve future scheduling, or is an additional recovery procedure required?** ## 8. Environment details - Apache Airflow version: 2.10 - Metadata database type: postgre - Multiple Airflow environments affected - Cron-based scheduling - DAGs with different scheduling frequencies - CI/CD deployment process - Recent introduction of standardized lifecycle callbacks - Issue first observed: October 15, 2026 For security reasons, I have intentionally omitted internal workflow identifiers, infrastructure names, configuration files, application code, and connection details. I can provide sanitized screenshots showing DAG Run states, timestamps, and task dependency errors. ## 9. Expected behavior A scheduled DAG Run should be created and become eligible for execution only when the scheduling conditions have been satisfied. For example, if a DAG is expected to run at 16:45, it should not appear as an active `running` DAG Run several minutes earlier with tasks blocked because their execution date is in the future. 10. Actual behavior Airflow creates scheduled DAG Runs prematurely. The DAG Runs remain in `running` with no tasks executing. The tasks become eligible only when the expected time arrives, at which point they generally complete successfully within seconds. My primary goal is to understand why these future scheduled DAG Runs are being created and how to restore correct scheduling safely without affecting production workflows or historical execution records.** Any guidance would be greatly appreciated. Thank you! GitHub link: https://github.com/apache/airflow/discussions/74808 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
