kaxil commented on issue #74660: URL: https://github.com/apache/airflow/issues/74660#issuecomment-6164487949
Thanks, that helps. The quota 403 fits what I described: the pod never ran, so a reschedule that doesn't spend a retry is the right outcome. Evicted and OOMKilled are different, because the pod did run. Eviction under node memory pressure is capacity, I agree. But an OOMKill from going over the container's own memory limit happens again on every attempt, so a reschedule just loops until the bound kicks in. Does your setup tell those apart, e.g. burstable pods getting killed by node pressure vs a real limit breach? There's also a gap on the KPO side. A failed pod currently comes back as a plain `AirflowException` with the termination message in the text, so a policy can only find `Evicted` or `OOMKilled` by string matching. I'd rather fix that in KPO with a typed exception that carries the pod's termination reason, so a `RetryRule` can match on it. That's a separate provider change from `RESCHEDULE` in the SDK. On the code pointer: no need for that one. I wrote the retry policy plumbing, so I know where this change goes. What I can't see from here is how it fails in your cluster, so details like the ones above are what help most. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
