dstandish edited a comment on pull request #19505: URL: https://github.com/apache/airflow/pull/19505#issuecomment-985854128
> @dstandish So clear really isn't meant to be a way to clean up the data that was stored in S3 or any other external database I wouldn't say quite that... if you are making a custom backend and you don't want orphaned objects, you very well might want to override clear to have it do that. Admittedly BaseXCom might eventually want a refactor to be more friendly for this issue, and for other aspects of implementing custom xcom backends also. > its moreso meant to ensure each task starts with a clean slate and doesn't duplicate its own XComs? Airflow does have the behavior of clearing xcoms at execution start. > What if I wanted to ensure that my S3 is purged once the dag completes (maybe there is sensitive data that I dont want sitting in S3). Does overriding any of these functions help in that case? Yes of course you can do whatever you want to do in your custom xcom backend so long as it plays nicely enough with airflow 🙂 Maybe ping me on slack if you want to continue the conversation on this topic, which is separate from this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
