kaxil commented on issue #74660:
URL: https://github.com/apache/airflow/issues/74660#issuecomment-6156272409
Thanks @stephen-bracken. Are you seeing pods stuck in `Pending`, or pod
creation rejected by the namespace quota? The answer changes what you need here.
For Pending pods KPO can already wait without burning retries:
```python
KubernetesPodOperator(
...,
deferrable=True,
schedule_timeout_seconds=4 * 60 * 60,
)
```
The wait happens in the triggerer so no worker slot is held, and the pod
starts when a node frees up rather than on your next 10 minute poll. Downside
is the Pending pods sit in the cluster and Kubernetes picks the order, not
Airflow priority weights, so if that's a problem for you let me know.
A full `ResourceQuota` is different: create returns a 403, KPO doesn't retry
it, and there's no pod to wait on. Today you'd have to subclass the operator to
raise `AirflowRescheduleException`, or set a lot of retries.
That's where I think `RetryAction.RESCHEDULE` belongs, and it's useful
beyond KPO for anything waiting on capacity. It shouldn't need an Execution API
change since any task can already go to `UP_FOR_RESCHEDULE`; the task runner
would send the same reschedule message sensors use. Try number and `retries`
stay untouched and `on_retry_callback` won't fire.
I'll make the reschedule bounded (by count or elapsed time) and fall back to
the normal retry/fail logic past that, since a rule matching a broad exception
would otherwise keep rescheduling real failures forever.
If you can paste the error from one of the failed attempts, that'd help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
