uranusjr commented on issue #41707: URL: https://github.com/apache/airflow/issues/41707#issuecomment-2308186610
(Copying my message from elsewhere with minor edits for context) It’s better to use setup and teardown because they handle errors better. For the reported issue specifically, `pre_execute` is actually executed *every time a worker is started for the task*, not only *when `execute` is called*, despite the name. (This is also true for `on_execution_callback`, by the way.) Resuming a deferred task starts a worker and causes `pre_execute` to be called. This is kind of by design, but I guess there’s a case to be made it’s not a good design. If we are to change this, I would recommend: 1. Rename `pre_execute` and `on_execution_callback` to remove `execute` from the name to avoid misunderstandings. Document them well to describe the actual behaviour. 2. (Maybe?) Add a hook that’s actually only run when the first worker is started for a ti. 3. (Maybe??) Add hooks that are run when a task is deferred and resumed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
