GitHub user ziggekatten edited a discussion: Deleting old XCom entrys does not
work
Trying to clear old entries as some DAGs passes some larger json objects, and
the database is getting unneccessary large.
I have an cleanup task that runs, tells me the number of entries to be deleted,
but the delete action does not delete the entries.
My code from my dev environment (using minute to ensure I get some data to
delete):
```
@task
def cleanup_xcom():
with settings.Session() as session:
ts_limit = datetime.now(timezone.utc) - timedelta(minutes=1)
print(ts_limit)
cnt = session.query(XCom).filter(XCom.execution_date <=
ts_limit).count()
print(cnt)
session.query(XCom).filter(XCom.execution_date <= ts_limit
).delete(synchronize_session=False)
```
My log:
```
[2024-10-24, 07:47:26 UTC] {local_task_job_runner.py:120} ▶ Pre task execution
logs
[2024-10-24, 07:47:26 UTC] {logging_mixin.py:188} INFO - 2024-10-24
07:46:26.237197+00:00
[2024-10-24, 07:47:26 UTC] {logging_mixin.py:188} INFO - 462
[2024-10-24, 07:47:26 UTC] {python.py:237} INFO - Done. Returned value was: None
[2024-10-24, 07:47:26 UTC] {taskinstance.py:441} ▼ Post task execution logs
[2024-10-24, 07:47:26 UTC] {taskinstance.py:1206} INFO - Marking task as
SUCCESS. dag_id=xcom_nfs_cleanup, task_id=cleanup_xcom,
run_id=manual__2024-10-24T07:45:52.665417+00:00,
execution_date=20241024T074552, start_date=20241024T074726,
end_date=20241024T074726
[2024-10-24, 07:47:26 UTC] {local_task_job_runner.py:240} INFO - Task exited
with return code 0
[2024-10-24, 07:47:26 UTC] {taskinstance.py:3498} INFO - 0 downstream tasks
scheduled from follow-on schedule check
[2024-10-24, 07:47:26 UTC] {local_task_job_runner.py:222} ▲▲▲ Log group end
```
As seen, 462 entires should be deleted, but it does not do that. Running my dag
again just show that 462 is up for delete....
What do I miss here? Airflow 2.9.1
GitHub link: https://github.com/apache/airflow/discussions/43341
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
