[
https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Arnold updated AIRFLOW-2215:
---------------------------------
Description:
The celery task that is ran by CeleryExecutor does not preserve the environment
variables when launching task instances in a subprocess. specifically,
AIRFLOW_HOME is lost. In my environment, this results in the command failing.
Typically the command is something like 'airflow run <dag> <operator>
<timestamp> -sd < dag path>'
@app.task
def execute_command(command):
log = LoggingMixin().log
log.info("Executing command in Celery: %s", command)
env = os.environ.copy()
try:
subprocess.check_call(command, shell=True, close_fds=True, env=env)
except subprocess.CalledProcessError as e:
log.error(e)
raise AirflowException('Celery command failed')
I believe the environment vars should be preserved in the subtask. Also, the
logging kind of sucks.
> celery task launches subprocess without environment vars
> --------------------------------------------------------
>
> Key: AIRFLOW-2215
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, worker
> Affects Versions: 1.9.0
> Reporter: John Arnold
> Priority: Major
>
> The celery task that is ran by CeleryExecutor does not preserve the
> environment variables when launching task instances in a subprocess.
> specifically, AIRFLOW_HOME is lost. In my environment, this results in the
> command failing. Typically the command is something like 'airflow run <dag>
> <operator> <timestamp> -sd < dag path>'
> @app.task
> def execute_command(command):
> log = LoggingMixin().log
> log.info("Executing command in Celery: %s", command)
> env = os.environ.copy()
> try:
> subprocess.check_call(command, shell=True, close_fds=True, env=env)
> except subprocess.CalledProcessError as e:
> log.error(e)
> raise AirflowException('Celery command failed')
>
> I believe the environment vars should be preserved in the subtask. Also, the
> logging kind of sucks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
