aprettyloner opened a new issue, #28872:
URL: https://github.com/apache/airflow/issues/28872
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
# Context
We performed an upgrade today from 2.2.3 to 2.4.0, with a chart upgrade from
8.4.1 to 8.6.1.
# Issue Encountered
The grid UI at `/dags/{dag_id}/grid` endpoint has very unusual display
behavior.
## 1. Missing recent dag runs in grid
This dag has run hourly in production since 2021-08-19. For some reason, the
grid endpoint is displaying dag runs from January 2022. Even when a dag run is
in progress, the grid default view still shows old data.
### Data Mismatch
#### Expected
We would expect to see in the **DAG Runs Summary** table:
- First Run Start: 2023-01-11, 02:00:09 UTC
- Last Run Start: 2023-01-12, 02:00:03 UTC
#### Observed
In the **DAG Runs Summary** table, we see:
- First Run Start: 2022-01-24, 04:00:05 UTC
- Last Run Start: 2022-01-25, 04:00:04 UTC
This doesn't align with the database for the same query (default 25 most
recent runs).
### Workaround
If and only if a dag is running, then it appears in the grid using the
`/dags/{dag_id}/grid?run_state=running` endpoint.
## 2. Peculiar ordering for dag runs in grid
This dag has been running daily since 2022-01-07. Unlike the above example,
the recent 2023 dag runs do appear in the grid. However, the ordering is very
peculiar. We would expect this week's runs to be all the way to the right.
Runs from 2023-01-07 to 2023-01-12 appear left of first run (2022-01-06, in
the failed state).
#### Expected
We would expect to see in the **DAG Runs Summary** table:
- First Run Start: 2022-12-19, 00:05:00 UTC
- Last Run Start: 2023-01-12, 00:05:00 UTC
#### Observed
As above, there is a mismatch in the **DAG Runs Summary** table.
- First Run Start: 2023-01-07, 00:05:01 UTC <---- Why would "first" be a
later date? 🤔 🤔 🤔 🤔 🤔 🤔
- Last Run Start: 2022-01-25, 00:05:04 UTC
This doesn't align with the database for the same query (default 25 most
recent runs).
### What you think should happen instead
# What possibly went wrong
There seems to be some incorrect date filtering/ordering against the
database.
# Observations
Interestingly, our stage instance for the same DAG shows recent runs
correctly and in a logical order. The deployments process, chart configuration,
and airflow version are identical across our environments.
## DAG History
The only discernable difference between behavior is when the dags were first
turned on.
### Not missing recent runs
Staging DAG has run since 2022-05-05
Prod DAG has run since 2022-01-07 - **Note that this one had the additional
left -> right ordering issue**
### Missing recent runs
Prod DAG has run since 2021-08-19
### How to reproduce
1. Spin up with 2.4.0 and run multiple dags with varying `start_date` and
with `catchup` enabled.
2. Inspect the `/dags/{dag_id}/grid` endpoint for running dags.
### Operating System
Debian GNU/Linux - 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon 5.1.0
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-cncf-kubernetes 4.3.0
apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.1.0
apache-airflow-providers-elasticsearch 4.2.0
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-google 8.3.0
apache-airflow-providers-grpc 3.0.0
apache-airflow-providers-hashicorp 3.1.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-microsoft-azure 4.2.0
apache-airflow-providers-mysql 3.2.0
apache-airflow-providers-odbc 3.1.1
apache-airflow-providers-pagerduty 3.1.0
apache-airflow-providers-postgres 5.2.1
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-salesforce 5.2.0
apache-airflow-providers-sendgrid 3.0.0
apache-airflow-providers-sftp 4.0.0
apache-airflow-providers-slack 5.1.0
apache-airflow-providers-sqlite 3.3.1
apache-airflow-providers-ssh 3.1.0
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployed on GKE via helm:v3.2.0.
### Anything else
I am willing to submit a PR but I need to dig into the codebase a bit more.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
