[ https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Arnold updated AIRFLOW-2215: --------------------------------- Description: The celery task that is ran by CeleryExecutor does not preserve the environment variables when launching task instances in a subprocess. specifically, AIRFLOW_HOME is lost. In my environment, this results in the command failing. Typically the command is something like 'airflow run <dag> <operator> <timestamp> -sd < dag path>' @app.task def execute_command(command): log = LoggingMixin().log log.info("Executing command in Celery: %s", command) env = os.environ.copy() try: subprocess.check_call(command, shell=True, close_fds=True, env=env) except subprocess.CalledProcessError as e: log.error(e) raise AirflowException('Celery command failed') I believe the environment vars should be preserved in the subtask. Also, the logging kind of sucks. > celery task launches subprocess without environment vars > -------------------------------------------------------- > > Key: AIRFLOW-2215 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2215 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler, worker > Affects Versions: 1.9.0 > Reporter: John Arnold > Priority: Major > > The celery task that is ran by CeleryExecutor does not preserve the > environment variables when launching task instances in a subprocess. > specifically, AIRFLOW_HOME is lost. In my environment, this results in the > command failing. Typically the command is something like 'airflow run <dag> > <operator> <timestamp> -sd < dag path>' > @app.task > def execute_command(command): > log = LoggingMixin().log > log.info("Executing command in Celery: %s", command) > env = os.environ.copy() > try: > subprocess.check_call(command, shell=True, close_fds=True, env=env) > except subprocess.CalledProcessError as e: > log.error(e) > raise AirflowException('Celery command failed') > > I believe the environment vars should be preserved in the subtask. Also, the > logging kind of sucks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)