Kris Geusebroek created AIRFLOW-1219:
----------------------------------------
Summary: Spark submit hook fix realtime logging.
Key: AIRFLOW-1219
URL: https://issues.apache.org/jira/browse/AIRFLOW-1219
Project: Apache Airflow
Issue Type: Bug
Reporter: Kris Geusebroek
Redirecting the logging to airflow doesn't work correctly.
Spark's logging goes to stderr and output of the spark program goes to stdout.
In the loop of the two sources stdout is used first.
It seems to be blocking until output is written at the end of the spark-submit
execution.
Fix this by connecting stderr to stdout in the subprocess invocation and only
pipe stdout to the airflow logging
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
