Hi Daniel et al,
yes and actually it is a bit strange... means even w/o interface change
a local feature could be implemented in the EdgeWorker by fetching
indirectly the task instance via
```
task_instance = self.queued_tasks[key][3] # TaskInstance in fourth element
--> Pool slots == task_instance.pool_slots
```
... which might be a "bit dirty"... and would be great to make this in
common for Airflow 3.
But being reminded that I was reading AIP-72 a long long time ago...
actually anyway a breaking change of executor interface is around the
corner which would actually provide the requested need. So most probably
we just need to "sit and wait" and once Airflow 3 is around the corner
we could re-discuss whether we should (w/o breaking interface change)
enable support for pool slots in some common executors.
Then... I assume we could close the discussion?
Jens
On 18.11.24 17:28, Daniel Standish wrote:
@jens
re
exactly this is the "minor" gap. In...
https://github.com/apache/airflow/blob/main/airflow/executors/base_executor.py#L522
... the Scheduler passes only the "executor_config" field and not the
full TaskInstance. So an executor does not have access to "pool_slots"
of the task. PR https://github.com/apache/airflow/pull/44016 is
The method `execute_async` is not actually called by the scheduler. It's
something the executor does internally in `_process_tasks`, and in
*that* method,
we seem to have the full ti:
https://github.com/apache/airflow/blob/298f1fd73b500a70b48267160ad0d6bb258fd205/airflow/executors/base_executor.py#L371
But that is weird if you look at the type of queued tasks 🤦
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
