Maximilian Roos created AIRFLOW-1758:
----------------------------------------
Summary: Print full traceback on errors
Key: AIRFLOW-1758
URL: https://issues.apache.org/jira/browse/AIRFLOW-1758
Project: Apache Airflow
Issue Type: Improvement
Components: cli
Reporter: Maximilian Roos
Priority: Minor
Currently when there is a failure during a run, it's difficult to see what the
cause was. Could we at least print the python stack trace?
As an example:
```
[2017-10-26 21:43:38,155] {models.py:1563} ERROR - DataFlow failed with return
code 1
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1461,
in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/dataflow_operator.py",
line 192, in execute
self.py_file, self.py_options)
File
"/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 155, in start_python_dataflow
task_id, variables, dataflow, name, ["python"] + py_options)
File
"/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 141, in _start_dataflow
_Dataflow(cmd).wait_for_done()
File
"/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 122, in wait_for_done
self._proc.returncode))
Exception: DataFlow failed with return code 1
```
I then need to jump into a repl and attempt simulate the command that airflow
would have run, which is both difficult and error prone. (Or is there a simpler
way of doing this??)
I then get a better stack-trace:
```
File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line
328, in run
return self.runner.run(self)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 283, in run
self.dataflow_client.create_job(self.job), self)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py",
line 168, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 423, in create_job
self.create_job_description(job)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 446, in create_job_description
job.options, file_copy=self._gcs_file_copy)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 347, in stage_job_resources
build_setup_args)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 439, in _build_setup_package
os.chdir(os.path.dirname(setup_file))
OSError: [Errno 2] No such file or directory: ''
```
Somewhat related to: https://issues.apache.org/jira/browse/AIRFLOW-174
(I'm using the DataFlowPythonOperator at the moment, but I suspect the issue is
wider)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
