Dennis O'Brien created AIRFLOW-664:
--------------------------------------

             Summary: airflow built from master unstable since change to parse 
DAG files in child process
                 Key: AIRFLOW-664
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-664
             Project: Apache Airflow
          Issue Type: Bug
          Components: scheduler
            Reporter: Dennis O'Brien


Prior to commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf I was able to pip 
build, install, and run airflow from the github commit.  E.g.,

{code}
pip install -e 
git://github.com/apache/incubator-airflow.git@835bcb6234956445d819ff39a5790f635c201c5d#egg=airflow[celery,crypto,hive,jdbc,ldap,password,postgres,s3,vertica]
{code}

Since this commit, airflow is not functional.
* The on/off button to enable a button does not appear.
* Next to each DAG an info icon appears with the hover text "This DAG seems to 
be existing only locally. The master scheduler doesn't seem to be aware of its 
existence."
* DAGs cannot be run from the command line.

I am able via the command line to list the dags and list the tasks for a dag, 
but trigger_dag does not actually cause any tasks to run (but no error messages 
either).
{code}
airflow@41e0e6920223:~$ airflow trigger_dag example_bash_operator
[2016-12-01 19:24:21,928] {__init__.py:50} INFO - Using executor CeleryExecutor
[2016-12-01 19:24:22,218] {models.py:162} INFO - Filling up the DagBag from 
/usr/local/airflow/dags
[2016-12-01 19:24:22,341] {cli.py:180} INFO - Created <DagRun 
example_bash_operator @ 2016-12-01 19:24:22: 
manual__2016-12-01T19:24:22.295709, externally triggered: True>
{code}

This commit was #1636 "[AIRFLOW-160] Parse DAG files through child processes".  
I haven't identified what part of this commit is to blame.  Or, what part of my 
configuration/setup is incompatible with this change.

My setup:
* Docker and docker-compose with...
* Ubuntu 16.04
* Python 3.5.2 (via Miniconda distribution)
* Separate container instances for scheduler, webserver, flower, and worker.
* Docker container running mysql 5.7
* Docker container running redis 3.2
* Using CeleryExecutor with redis broker
* Building and installing from master using `pip install -e git://...`

I have a github project that uses Docker and docker-compose to build and test 
these components.  
[https://github.com/dennisobrien/airflow-docker-compose-example] .
* pip install line in 
[Dockerfile|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/Dockerfile#L58]
* [airflow.cfg 
template|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/config/airflow.cfg.tpl]
* 
[docker-entrypoint.sh|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/bin/docker-entrypoint.sh]
 defines how each service is run.
* 
[docker-compose.yml|https://github.com/dennisobrien/airflow-docker-compose-example/blob/master/docker-compose.yml]
 orchestrates the components.

Let me know if there are additional tests I can do to narrow this down.  I 
suspect since others have not complained about this that it is probably 
something related to my setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to