[
https://issues.apache.org/jira/browse/AIRFLOW-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985223#comment-16985223
]
Forestlzj commented on AIRFLOW-3128:
------------------------------------
my workaround:
{code:java}
# https://issues.apache.org/jira/browse/AIRFLOW-3128
for line in iter(self._sp.stdout.readline, b''):
line = line.decode('utf-8').strip()
self.log.info(line)
self._sp.stdout.close()
returncode = self._sp.wait()
if returncode:
raise AirflowException(
"Cannot execute {} on {}. Process exit code: {}.".format(
cmd, self._conn.host, returncode)
){code}
> SparkSqlOperator prints blank info endlessly after SparkContext stopped
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-3128
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3128
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
> Reporter: Forestlzj
> Priority: Major
> Fix For: 1.10.0
>
>
> environment: hdp 2.6.2 with spark 2
> spark_sql_test =
> SparkSqlOperator(task_id='spark_sql_test',dag=dag,sql='select * from
> default.test limit
> 2;',name='airflow_spark_sql_test',executor_cores=2,num_executors=2)
> Log:
> [2018-09-28 17:14:08,754] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08
> INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!\n'
> [2018-09-28 17:14:08,778] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08
> INFO SparkContext: {color:#205081}Successfully stopped SparkContext\n'{color}
> [2018-09-28 17:14:08,780] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08
> INFO ShutdownHookManager: Shutdown hook called\n'
> [2018-09-28 17:14:08,780] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08
> INFO ShutdownHookManager: Deleting directory
> /tmp/spark-ba3b2fef-ae75-4e65-afb2-1b029b2c5c23\n'
> [2018-09-28 17:14:08,781] \{spark_sql_hook.py:148} INFO - b'18/09/28 17:14:08
> INFO ShutdownHookManager: Deleting directory
> /tmp/spark-ba397c83-4003-4006-8ce9-179cc2df2dd7\n'
> {color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,336] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
> {color:#FF0000}[2018-09-28 17:14:09,337] \{spark_sql_hook.py:148} INFO -
> b''{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
