Nimrod Morag created AIRFLOW-3394:
-------------------------------------
Summary: Cannot redirect output from subprocesses
Key: AIRFLOW-3394
URL: https://issues.apache.org/jira/browse/AIRFLOW-3394
Project: Apache Airflow
Issue Type: Bug
Components: logging
Affects Versions: 1.10.1
Environment: ubuntu 16.04
python 3.5.2
Reporter: Nimrod Morag
When using PythonOperator, I'm using the subprocess module and redirecting the
output to sys.stdout in the parent process
since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object
has no attribute 'fileno' Traceback (most recent call last): File
"/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in
_run_raw_task result = task_copy.execute(context=context) File
"/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch =
super(MyMultiBranchPythonOperator, self).execute(context) File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 95, in execute return_value = self.execute_callable() File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 100, in execute_callable return self.python_callable(*self.op_args,
**self.op_kwargs) File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in
triggerTraining run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking
no\n" >> ~/.ssh/config') File "/vagrant/pysrc/Airflow/Utils.py", line 23, in
run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr,
shell=True): File "/usr/lib/python3.5/subprocess.py", line 693, in run with
Popen(*popenargs, **kwargs) as process: File
"/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) =
self._get_handles(stdin, stdout, stderr) File
"/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite =
stdout.fileno() AttributeError: 'StreamLogWriter' object has no attribute
'fileno' [2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as
FAILED.}}
{{Fixed locally by adding to the StreamLogWriter class:}}
{{def fileno(self):}}
{{ return 1}}
{{(1 is stdout's file descriptor)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
