github-actions[bot] opened a new pull request, #76028: URL: https://github.com/apache/airflow/pull/76028
When a task instance failed and was put up for retry, `prepare_db_for_next_try` copied every column of the failed attempt into the new one, so the waiting attempt showed the previous attempt's start date, end date, duration, hostname and pid in the API and UI. The retry deadline was computed from that copied end_date, so it could not simply be nulled out. We now store the deadline on the waiting attempt as `retry_at`. The Execution API deliberately avoids loading the serialized Dag, and loading it on every retry just to compute a delay would undo that. A retry reported by a worker therefore arrives without a deadline, and the scheduler computes and stores it on its first pass over the run. Until then, and for rows already waiting at upgrade time, the deadline is computed from the previous attempt's end date. Because the delay is fixed at that point, changing retry_delay in a new Dag version no longer retimes a retry that is already waiting. The column is ret-conned in tothe 3.4.0 migration that made each attempt its own task_instance row, since that is where a waiting retry became a separate row inheriting the failed attempt's fields. Anyone who has already migrated to RC1 will need to manually apply this, or recreate their DB: ```sql -- PostgreSQL ALTER TABLE task_instance ADD COLUMN retry_at TIMESTAMP WITH TIME ZONE; -- MySQL ALTER TABLE task_instance ADD COLUMN retry_at TIMESTAMP(6) NULL; -- SQLite ALTER TABLE task_instance ADD COLUMN retry_at DATETIME; ``` The scheduler has always skipped dependency evaluation for retries that are not yet due. A waiting retry has already passed its trigger rule, so an upstream that changes state during the wait is only acted on once the retry is due or the run checks for a deadlock, and the end state is the same. Doesn't really affect anything but people messing about with clearing upstream tasks etc, but the eventually consistent state remains the same before and after this change, it just might take more time to reach it `emit_state_change_metric` relied on the copied end_date to skip retries, so scheduled_duration and queued_duration are now also emitted for retries and cleared tasks. Each attempt has its own scheduled and queued times, so these samples are accurate. (cherry picked from commit 155560df6685ccb3db59f954faf927e80af8059c) Co-authored-by: Ash Berlin-Taylor <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
