t oo created AIRFLOW-6359:
-----------------------------
Summary: spark_submit_hook.py polling too frequent
Key: AIRFLOW-6359
URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
Project: Apache Airflow
Issue Type: Improvement
Components: hooks
Affects Versions: 1.10.6
Reporter: t oo
spark polling every second is too frequent, use config to change to desired
interval, ie 15 seconds
contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
instead of every second
BEFORE
Sleep for 1 second as we do not want to spam the cluster
time.sleep(1)
AFTER
import airflow
from airflow import configuration as conf
Sleep for n second as we do not want to spam the cluster
_poll_interval = conf.getint('sparksubmit', 'poll_interval')
time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
