Paymahn Moghadasian created AIRFLOW-2214:

             Summary: Import error when interpreting DAG with customer operator
                 Key: AIRFLOW-2214
             Project: Apache Airflow
          Issue Type: Bug
          Components: DAG, operators, plugins
    Affects Versions: 1.9.0
            Reporter: Paymahn Moghadasian

The [airflow docs|(] suggest 
that a basic sanity check for a DAG file is to interpret it. ie:

$ python ~/path/to/my/

I've found this to be useful. However, now I've created a plugin, 
`MordorOperator` under `$AIRFLOW_HOME/plugins`:

from airflow.plugins_manager import AirflowPlugin
from airflow.utils.decorators import apply_defaults
from airflow.operators import BaseOperator
from airflow.exceptions import AirflowException

class MordorOperator(BaseOperator):
    JOB_QUEUE_MAPPING = {"testing": "testing"}

    def __init__(self, job, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def execute(self, context):
        print("Executing mordoroperator")

class MordorPlugin(AirflowPlugin):
    name = "MordorPlugin"
    operators = [MordorOperator]

I can import the plugin and see it work in a sample DAG:

from airflow import DAG
from airflow.operators import MordorOperator
from datetime import datetime

dag = DAG('mordor_dag', description='DAG with a single task',, catchup=False)

mordor = MordorOperator(job="testing", task_id='run_single_task', dag=dag)

However, when I try to interpret this file I get failures which I suspect I 
shouldn't get since the plugin successfully runs. My suspicion is that this is 
because there's some dynamic code gen happening at runtime which isn't 
available when a DAG is interpreted by itself. I also find that PyCharm can't 
perform any autocompletion when importing the plugin.

(venv)  3:54PM /Users/paymahn/solvvy/scheduler mordor.operator ✱
 ❮❮❮ python dags/
section/key [core/airflow-home] not found in config
Traceback (most recent call last):
  File "dags/", line 2, in <module>
    from airflow.operators import MordorOperator
ImportError: cannot import name 'MordorOperator'

How can a DAG using a plugin be sanity tested? Is there another mechanism to 
import the custom operator which will allow interpreting the DAG from the 

This message was sent by Atlassian JIRA

Reply via email to