dstandish commented on PR #39046: URL: https://github.com/apache/airflow/pull/39046#issuecomment-2065539169
so the problem @o-nikolas is that the we don't have a TIRun object, for example. we only have TI. this one object is all we have no matter how many retries or clears there are. so we are stuck with the problem of losing history of basically every kind when you are talking about the TI object. if your concern is what _actually_ happened, the only real source of truth for that is logs, and i would suggest just logging what executor it ran with (indeed it probably already does). i understand your frustration but just calling it like I see it as we should all do. note that i didn't "request changes" and block anything, and that's intentional cus if i'm outnumbered i go with the wisdom of the group. a totally valid option when we can't reach consensus I think is to just bring it to the dev list. i've done this in the past, and that's what i would recommend if you feel like we're not getting to the right result. i don't personally think of this as "it's broken now for everything else, why fix it for this one". from my perspective, i'm just saying, i don't think it makes sense to not allow the executor to be changed in between retries etc. unless there's a good reason, we should respect the code the user provides i think. i don't think "we want to use TI to tell the truth about historical runs" is a good reason because (1) it has this negative consequence of not respecting the user's current config and (2) TI is simply incapable of doing so in general (as mentioned above) and therefore, until Jed's vision with keeping TI history is realized, i think we must simply rely on logs as the source of truth for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
