aa1371 commented on a change in pull request #16829:
URL: https://github.com/apache/airflow/pull/16829#discussion_r673627247
##########
File path: airflow/executors/dask_executor.py
##########
@@ -81,7 +81,22 @@ def airflow_run():
if not self.client:
raise AirflowException(NOT_STARTED_MESSAGE)
- future = self.client.submit(airflow_run, pure=False)
+ resources = None
+ if queue is not None:
+ avail_resources = self.client.run_on_scheduler(
+ lambda dask_scheduler: dask_scheduler.resources
+ )
Review comment:
Cool, didn't know about the scheduler plugins, so thanks for that. Not
sure what I'm doing wrong, but doesn't quite seem to work though,
```
# file.py
class GetResourcesPlugin(SchedulerPlugin):
def __init__(self, scheduler):
self.scheduler = scheduler
self.scheduler.handlers["get-resources"] = self.get_resources
def get_resources(self):
return self.scheduler.resources
@click.command()
def dask_setup(scheduler):
plugin = GetResourcesPlugin(scheduler)
scheduler.add_plugin(plugin)
print(scheduler.handlers) # I see the desired handler added here
# print(scheduler.get_resources()) # get error since this method does
exist, so tried without it too
```
```
# start scheduler
$ dask-scheduler --preload file.py
```
```
c = Client('<scheduler address>')
c.scheduler.get_resources() # doesn't seem to work
```
Any ideas what I'm doing wrong here?
In any case, I think suggestion #1 makes the most sense though, since
knowing what worker resources you have available is relevant to how you would
submit tasks from the client, I think it would be a good idea to be able to
access this directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
