josh-fell commented on PR #38001: URL: https://github.com/apache/airflow/pull/38001#issuecomment-2023850316
> @josh-fell As I think about adding a flag for retry from failure. I can't wrap my head around the logic for retry in a deferrable operator. Mind share your thoughts on how this could work if the operator is deferrable? @andyguwc Thanks for your patience here with my responses! How I think about this feature, from a UX perspective, is really handling which endpoint is used in the `trigger_job_run` method of the DbtCloudHook based on a DbtCloudRunJobOperator parameter. The key is the behavior change of _triggering_ the job rather than handling what to do based on its run status which is what the deferrable functionality checks. Also, let Airflow continue with its retry functionality; nothing new needed in the provider I think. From the documentation of this new endpoint, it seems as though it could be used for retries as well as regular job execution. >Use this endpoint to retry a failed run for a job from the point of failure, if the run failed. Otherwise trigger a new run. When this endpoint returns a successful response, a new run will be enqueued for the account. What this suggests is the dbt API handling the lookup of the last run for a given job and choosing whether or not to start a new instance of the job or not, which is great! So, I would think the implementation in Airflow would be adding a `retry_from_failure` parameter (or similarly named as you wish of course) to the DbtRunJobOperator, proprogate that value down to the `trigger_job_run` method in the hook, and then choose which endpoint to use based on that value. This way users simply just need to set a value in an existing operator and that's it. I could be misinterpreting the robustness of this new endpoint. If I am, the checking of the last job run status could be implemented in the hook and then carry on with the endpoint decision. I hope that helps! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
