Leondon9 commented on PR #73702: URL: https://github.com/apache/airflow/pull/73702#issuecomment-6229757648
> On my machine I'm seeing a problem arise with this PR when a deferred task is cleared. `clear` instructs the task to run again, and on 3.3+ the re-run looks for an existing Glue job from the same task. It finds the old one still running, and decides to just wait on that one. But clearing also triggers this PR's "on kill" step, which stops that same old Glue job. So the re-run ends up waiting on a stopped job and fails with `STOPPED`. On main, the re-run reattached and succeeded. > > Could the re-run start a fresh Glue job when `stop_job_run_on_kill=True`, instead of reattaching? Thanks for reproducing this, you're right. on_kill fires on any user action and can't tell a clear from mark failed/success, so the re-run reattached to the run it was stopping. Fixed in db0041cde4: with stop_job_run_on_kill=True in deferrable mode, a retried try no longer reattaches. It stops the previous run itself, waits for it to reach a terminal state (a STOPPING run still counts toward MaxConcurrentRuns), then submits a fresh one. If the run doesn't stop within 10 minutes, the try fails instead of starting a second run. durable and its default are unchanged, so stop_job_run_on_kill=False still reattaches as before. Would you mind re-running your clear scenario? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
