[
https://issues.apache.org/jira/browse/AIRFLOW-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804144#comment-16804144
]
Ultrabug commented on AIRFLOW-1135:
-----------------------------------
Hi [~jackjack10]
I understand this because this is how you use XComs but some users (like me)
also use it to store (a lot of) states between runs which matter only for the
next one. So this is in my case transient data which I'm not interested in
afterwards.
You can see it as a way to avoid having to rely on an external database to
store ephemeral yet persistent-for-the-next-run data.
In this case yes I'm interested about the DAG metadata, but not by the data my
code uses/generate through XComs.
> Clearing XCom data from previous executions
> -------------------------------------------
>
> Key: AIRFLOW-1135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1135
> Project: Apache Airflow
> Issue Type: New Feature
> Components: db
> Reporter: Ultrabug
> Priority: Major
>
> It looks like XCom data from previous executions of tasks are never cleared
> from database and that there's no easy way to do so.
> Since XComs are created by execution_date of a task, when a task heavily uses
> XComs (like 20 000+) we can end up with a large xcom table on the database.
> Is there something I'm missing about cleaning up those previous XComs from
> database (since they're not used anymore) ?
> Would you accept the idea of having a new PR providing a helper to remove
> them ?
> Thanks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
