aran3 opened a new issue #17977:
URL: https://github.com/apache/airflow/issues/17977
### Apache Airflow version
2.1.3 (latest released)
### Operating System
Red Hat Enterprise Linux Server 7.6 (Maipo)
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==2.0.0
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-sqlite==2.0.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### What happened
Running a dag that is manually triggered, and has only 1 step, sometimes the
scheduler marks the DAG as success without running the step at all. The step
status remains Null in the db and in UI.
### What you expected to happen
Scheduler should never mark Dag complete if it has "Null" state tasks.
Probably this task should be marked as not "Null" before this function even
runs, but I'm not sure about it.
### How to reproduce
I don't have a fool-proof way to reproduce - but I was able to reproduce it
when I brought down the scheduler for a long time and triggered the dag quickly
a few times.
It seems to happen more if dag is triggered multiple times quickly (in the
order of seconds).
### Anything else
Happens about 20-50% of the time for this dag as described above.
I reproduced it while running the scheduler with DEBUG, and the only
relevant logs are:
```
[2021-09-01 21:58:17,358] {retries.py:78} DEBUG - Running
SchedulerJob._create_dagruns_for_dags with retries. Try 1 of 3
[2021-09-01 21:58:17,365] {retries.py:78} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2021-09-01 21:58:17,376] {retries.py:78} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2021-09-01 21:58:17,402] {scheduler_job.py:1138} DEBUG - DAG feeds_dag not
changed structure, skipping dagrun.verify_integrity
[2021-09-01 21:58:17,405] {dagrun.py:480} DEBUG - number of tis tasks for
<DagRun feeds_dag @ 2021-09-01 20:15:07+00:00:
mtf_9001_20210901_120__2021-09-01T20:15:07+00:00, externally triggered: True>:
0 task(s)
[2021-09-01 21:58:17,406] {dagrun.py:434} INFO - Marking run <DagRun
feeds_dag @ 2021-09-01 20:15:07+00:00:
mtf_9001_20210901_120__2021-09-01T20:15:07+00:00, externally triggered: True>
successful
[2021-09-01 21:58:17,406] {dagrun.py:604} WARNING - Failed to record
duration of <DagRun feeds_dag @ 2021-09-01 20:15:07+00:00:
mtf_9001_20210901_120__2021-09-01T20:15:07+00:00, externally triggered: True>:
start_date is not set.
```
Notice **0 task(s)**, which I expect is caused by the task being in Null
state in DB and `DagRun.task_instance_scheduling_decisions` not checking for
Null-state tasks.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
