[
https://issues.apache.org/jira/browse/AIRFLOW-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nimrod Morag updated AIRFLOW-3394:
----------------------------------
Description:
When using PythonOperator, I'm using the subprocess module and redirecting the
output to sys.stdout in the parent process
since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object
has no attribute 'fileno' Traceback (most recent call last):}}
{{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659,
in _run_raw_task result = task_copy.execute(context=context)_}}
{{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch =
super(MyMultiBranchPythonOperator, self).execute(context)_}}
{{_File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 95, in execute return_value = self.execute_callable()_}}
{{_File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 100, in execute_callable return self.python_callable(*self.op_args,
**self.op_kwargs)_}}
{{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining
run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >>
~/.ssh/config')_}}
{{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not
subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}
{{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with
Popen(*popenargs, **kwargs) as process:_}}
{{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread,
errwrite) = self._get_handles(stdin, stdout, stderr)}}
{{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite
= stdout.fileno()}}
{{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}
{{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
{{Fixed locally by adding to the StreamLogWriter class:}}
{{def fileno(self):}}
{{ return 1}}
{{(1 is stdout's file descriptor)}}
was:
When using PythonOperator, I'm using the subprocess module and redirecting the
output to sys.stdout in the parent process
since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object
has no attribute 'fileno' Traceback (most recent call last): File
"/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in
_run_raw_task result = task_copy.execute(context=context) File
"/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch =
super(MyMultiBranchPythonOperator, self).execute(context) File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 95, in execute return_value = self.execute_callable() File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 100, in execute_callable return self.python_callable(*self.op_args,
**self.op_kwargs) File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in
triggerTraining run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking
no\n" >> ~/.ssh/config') File "/vagrant/pysrc/Airflow/Utils.py", line 23, in
run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr,
shell=True): File "/usr/lib/python3.5/subprocess.py", line 693, in run with
Popen(*popenargs, **kwargs) as process: File
"/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) =
self._get_handles(stdin, stdout, stderr) File
"/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite =
stdout.fileno() AttributeError: 'StreamLogWriter' object has no attribute
'fileno' [2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as
FAILED.}}
{{Fixed locally by adding to the StreamLogWriter class:}}
{{def fileno(self):}}
{{ return 1}}
{{(1 is stdout's file descriptor)}}
> Cannot redirect output from subprocesses
> ----------------------------------------
>
> Key: AIRFLOW-3394
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3394
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
> Affects Versions: 1.10.1
> Environment: ubuntu 16.04
> python 3.5.2
> Reporter: Nimrod Morag
> Priority: Major
> Labels: beginner, easyfix, patch, usability
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> When using PythonOperator, I'm using the subprocess module and redirecting
> the output to sys.stdout in the parent process
>
> since `StreamLogWriter` doesn't implement fileno() this causes the task to
> fail:
> {{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter'
> object has no attribute 'fileno' Traceback (most recent call last):}}
> {{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659,
> in _run_raw_task result = task_copy.execute(context=context)_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch =
> super(MyMultiBranchPythonOperator, self).execute(context)_}}
> {{_File
> "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
> line 95, in execute return_value = self.execute_callable()_}}
> {{_File
> "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
> line 100, in execute_callable return self.python_callable(*self.op_args,
> **self.op_kwargs)_}}
> {{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining
> run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >>
> ~/.ssh/config')_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not
> subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with
> Popen(*popenargs, **kwargs) as process:_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread,
> errwrite) = self._get_handles(stdin, stdout, stderr)}}
> {{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles
> c2pwrite = stdout.fileno()}}
> {{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}
> {{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
>
> {{Fixed locally by adding to the StreamLogWriter class:}}
> {{def fileno(self):}}
> {{ return 1}}
>
> {{(1 is stdout's file descriptor)}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
