ldacey opened a new issue, #24970:
URL: https://github.com/apache/airflow/issues/24970
### Apache Airflow version
2.3.2 (latest released)
### What happened
I had a DAG with an end_date that I needed to manually run to capture some
additional files. Manually triggering the DAG did not run any tasks, so I
temporarily committed a change in the DAG code and edited my "adjust_end"
filter which is used to find files modified within a window of the
data_interval_end period:
```
user_defined_filters=dict(
adjust_start=lambda x: x.subtract(minutes=30),
adjust_end=lambda x: x.add(minutes=10775),
)
# data_interval_end gets adjusted
start="{{ params.start if params.start is not none else data_interval_end |
adjust_start }}",
end="{{ params.end if params.end is not none else data_interval_end |
adjust_end }}",
```
I then cleared the last task in that DAG and was able to successfully
download the extra files. However, my transformation dynamic task kept
processing only the **first** file. Ultimately, I deleted the DAG run instead
of just clearing it and the DAG finally processed all 6 files which were
downloaded.
### What you think should happen instead
When I cleared the tasks, the extract task passed an XCom containing a list
of the 6 files. The transformation dynamic task should have processed all 6 of
those files instead of just the original one file. I do not think deleting the
DAG run should be necessary - it has not been required historically
(non-dynamic mapped task DAGs).
### How to reproduce
Clear a DAG run which has been successful previously after making some
changes to the DAG (for example, edit a query window). The extract task will be
correct, but the dynamic task (in my case `transform`) will not pull the proper
XComs.
extract >> transform[ ] >> load
When I delete the DAG run (instead of just clearing it), the files are
processed as expected. For some reason clearing is not enough which is strange.
### Operating System
Ubuntu 20.04
### Versions of Apache Airflow Providers
I am using the Airflow 2.3.2 constraints with these providers:
```
apache-airflow-providers-celery
apache-airflow-providers-docker
apache-airflow-providers-ftp
apache-airflow-providers-http
apache-airflow-providers-microsoft-azure
apache-airflow-providers-mysql
apache-airflow-providers-postgres
apache-airflow-providers-odbc
apache-airflow-providers-redis
apache-airflow-providers-salesforce
apache-airflow-providers-sftp
apache-airflow-providers-ssh
apache-airflow-providers-google
```
### Deployment
Other Docker-based deployment
### Deployment details
Docker Swarm - should not be relevant though.
### Anything else
N/A
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
