dstandish commented on pull request #19505: URL: https://github.com/apache/airflow/pull/19505#issuecomment-985829944
> @uranusjr needs to make a similar sort of change (where the signature for BaseOperatorLink changes) and I think he was proposing putting `version` class attribute instead of using sig.inspect. > > I guess the main difference there is the number of args wasn't changing, but sig.inspect is relatively slow. Thoughts TP? Should we use the same `Class.version` attribute approach here? Yeah sounds reasonable. Probably not the end of the world to have a little slowness here but it does sound like a clean and direct way to achieve the same goal. LMK @jmaldon1 > Slightly off-topic but curious to know if anyone can help me out. When is the clear method called in the DAG process? How are we avoiding the case that we clear the data from the external database, but the tasks fails and has to rerun? If the task reruns, won't it need to query for the data again, but if we deleted it already, then it wont be there anymore. Also what is the difference between delete and clear. When is delete called? Clear will be called at the start of the run, no matter whether it's the first attempt, second attempt, a retry, a backfill -- whatever: XCom is cleared at the start of the task. Delete vs clear, delete is just lower level and will delete a collection of xcom objects. Clear removes all xcom records for a specific task instance But now that you bring this up, yeah depending on how you implement your custom xcom backend the backing files might not be purged. E.g. in the example backend on the astronomer blog they wouldn't be; but the database would not know they are there so in effect they are not there, apart from showing up your s3 bill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
