[jira] [Commented] (AIRFLOW-2215) celery task launches subprocess without environment vars
[
https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423503#comment-16423503
]
ASF subversion and git services commented on AIRFLOW-2215:
--
Commit d1f94fe20ea47f18f353ed47a07bc589a4a9c063 in incubator-airflow's branch
refs/heads/master from [~johnarnold]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d1f94fe ]
[AIRFLOW-2215] Pass environment to subproces.Popen in base_task_runner
Closes #3183 from johnarnold/env2
> celery task launches subprocess without environment vars
>
>
> Key: AIRFLOW-2215
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, worker
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Priority: Major
> Fix For: 2.0.0
>
>
> The celery task that is ran by CeleryExecutor does not preserve the
> environment variables when launching task instances in a subprocess.
> specifically, AIRFLOW_HOME is lost. In my environment, this results in the
> command failing. Typically the command is something like 'airflow run
> -sd < dag path>'
> @app.task
> def execute_command(command):
> log = LoggingMixin().log
> log.info("Executing command in Celery: %s", command)
> env = os.environ.copy()
> try:
> subprocess.check_call(command, shell=True, close_fds=True, env=env)
> except subprocess.CalledProcessError as e:
> log.error(e)
> raise AirflowException('Celery command failed')
>
> I believe the environment vars should be preserved in the subtask. Also, the
> logging kind of sucks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2215) celery task launches subprocess without environment vars
[
https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400056#comment-16400056
]
ASF subversion and git services commented on AIRFLOW-2215:
--
Commit a8174047b24ec4c3eb62e5651b39ca44dc8dd94b in incubator-airflow's branch
refs/heads/master from [~johnarnold]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a817404 ]
[AIRFLOW-2215] Update celery task to preserve environment variables and improve
logging on exception
Closes #3126 from johnarnold/celery_env
> celery task launches subprocess without environment vars
>
>
> Key: AIRFLOW-2215
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler, worker
>Affects Versions: 1.9.0
>Reporter: John Arnold
>Priority: Major
> Fix For: 2.0.0
>
>
> The celery task that is ran by CeleryExecutor does not preserve the
> environment variables when launching task instances in a subprocess.
> specifically, AIRFLOW_HOME is lost. In my environment, this results in the
> command failing. Typically the command is something like 'airflow run
> -sd < dag path>'
> @app.task
> def execute_command(command):
> log = LoggingMixin().log
> log.info("Executing command in Celery: %s", command)
> env = os.environ.copy()
> try:
> subprocess.check_call(command, shell=True, close_fds=True, env=env)
> except subprocess.CalledProcessError as e:
> log.error(e)
> raise AirflowException('Celery command failed')
>
> I believe the environment vars should be preserved in the subtask. Also, the
> logging kind of sucks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
