[
https://issues.apache.org/jira/browse/AIRFLOW-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300909#comment-15300909
]
Chris Riccomini commented on AIRFLOW-108:
-----------------------------------------
I spoke with [[email protected]] about this a bit. One trick that I
didn't realize is that you can delete all of the task instances after their
DagRun is marked as success/failed. Once the DagRun is marked as such, if the
tasks are deleted, the scheduler won't try to re-run them because the DagRun is
already showing as a terminal state.
This is a bit hacky, but does work. I still think a retention policy that
allows us to delete TaskInstances *and* DagRuns would be useful, but due to the
trick described above, I think this JIRA is probably lower priority than it was
when I initially filed this ticket.
> Add data retention policy to Airflow
> ------------------------------------
>
> Key: AIRFLOW-108
> URL: https://issues.apache.org/jira/browse/AIRFLOW-108
> Project: Apache Airflow
> Issue Type: Wish
> Components: db, scheduler
> Reporter: Chris Riccomini
>
> Airflow's DB currently holds the entire history of all executions for all
> time. This is problematic as the DB grows. The UI starts to get slower, and
> the DB's disk usage grows. There is no bound to how large the DB will grow.
> It would be useful to add a feature in Airflow to do two things:
> # Delete old data from the DB
> # Mark some lower watermark, past which DAG executions are ignored
> For example, (2) would allow you to tell the scheduler "ignore all data prior
> to a year ago". And (1) would allow Airflow to delete all data prior to
> January 1, 2015.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
