hussein-awala commented on issue #32638: URL: https://github.com/apache/airflow/issues/32638#issuecomment-1659192636
> But triggers already have a timeout mechanism. The current mechanism is not an implementation for timeout (time-based), but it's based on an extra parameter called `max_attempts`, which tells the trigger how many attempts it should try before failing. For example, when we configure it with `max_attempts=300`, the maximum execution time for this trigger will be `300 * (sleep time + execution time)`. The execution time is not negligible, as in most cases, it involves a call to an external API, and we could experience some latency during this call. Additionally, the trigger could be adopted by another triggerer if we lose the one that runs the trigger. Since the consumed attempts are not synchronized with the Metadata, it will restart from the beginning with each run. What I'm trying to do is: - Run the task until deferring it. - Calculate the time when we should stop the task because of a timeout -> `start_date + timeout duration`. - Create the trigger and provide the timeout timestamp. - Run the trigger in a triggerer. - In the triggerer, there will be a loop that checks if one of the timeouts is already reached to cancel the asyncio task. - The cancellation exception will be caught and the task will be marked as failed because of a timeout. What are the benefits of this mechanism? - No need to implement it explicitly in each Trigger. - The task timeout is respected as the normal task. - No need to handle the timeout in the task, as it will be caught by Airflow jobs. What do you think about this solution? cc: @potiuk @pankajastro @syedahsn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
