[jira] [Commented] (AIRFLOW-245) Access to task instance from custom Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341824#comment-15341824 ] Alexandr Nikitin commented on AIRFLOW-245: -- [~jlowin] Thank you for the reply. Yes, I have looked at the MesosExecutor. What I don't like is the dependency on Airflow. I don't want /cannot install Airflow on Mesos slaves. I want to use native APIs of Mesos/Marathon/etc. Also I don't want to introduce new Operators. E.g. I can run DockerOperator on local executor, I want to be able to run the same operator on Mesos via its API. > Access to task instance from custom Executor > > > Key: AIRFLOW-245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-245 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Reporter: Alexandr Nikitin > > I'm writing a custom executor that executes tasks on mesos and I want to have > access to task instances from it. So that I can reuse all existing operators > e.g. DockerOperator and access its fields like image, command, volumes and > transform them to mesos. > This can be done by changing `def execute_async(self, key, command, > queue=None):` in `BaseExecutor`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-245) Access to task instance from custom Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341783#comment-15341783 ] Jeremiah Lowin commented on AIRFLOW-245: The Scheduler is an object that decides what tasks to run; the Executor is an object that decides where to run those tasks. So the Scheduler hands a TaskInstance to the Executor and says "I want to run this task". That's why the Executor has access to the TaskInstance when tasks are being queued. The Executor then extracts the TI's "command" (for example: airflow run my_task) and its sole job is to decide how to run that command. execute_async() is where that decision is made. The important thing is that the thing being run is the string command, not the TI object, and the signature of execute_async should reflect that. The larger question here is why would an operator ever influence the executor? An Operator is a series of instructions that are designed to be run in a self-contained way, no matter what the executor is. An executor is a way of kicking off an Operator. There should be no dependency between them. To be explicit: an Operator must be able to run under any Executor. However, I am not familiar with Mesos so unfortunately I don't know exactly what you're trying to do -- but have you looked at the existing MesosExecutor in airflow.contrib.executors? Is it insufficient for your objectives? If your operator requires special handling, then it sounds like maybe you want a MesosOperator that runs your docker command? > Access to task instance from custom Executor > > > Key: AIRFLOW-245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-245 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Reporter: Alexandr Nikitin > > I'm writing a custom executor that executes tasks on mesos and I want to have > access to task instances from it. So that I can reuse all existing operators > e.g. DockerOperator and access its fields like image, command, volumes and > transform them to mesos. > This can be done by changing `def execute_async(self, key, command, > queue=None):` in `BaseExecutor`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-245) Access to task instance from custom Executor
[ https://issues.apache.org/jira/browse/AIRFLOW-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339888#comment-15339888 ] Jeremiah Lowin commented on AIRFLOW-245: I also posted this on PR 1596 but perhaps it's more relevant here: Something important to keep in mind -- the current version of execute_async() just requires the task key (a bunch of strings) and the command that runs the task (another string, something like airflow run task). That's because we make no assumptions about how the execution is implemented, or on what machine/environment, or even when it will be run. Passing a task object to the execute_async() method is contrary to that assumption. In particular, there is no guarantee that task objects are serializable. The key should give enough information to load the task inside the executor by querying the database for the matching taskinstance -- I think that would be the far preferable way to implement the desired behavior in a custom executor. > Access to task instance from custom Executor > > > Key: AIRFLOW-245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-245 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Reporter: Alexandr Nikitin > > I'm writing a custom executor that executes tasks on mesos and I want to have > access to task instances from it. So that I can reuse all existing operators > e.g. DockerOperator and access its fields like image, command, volumes and > transform them to mesos. > This can be done by changing `def execute_async(self, key, command, > queue=None):` in `BaseExecutor`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
