[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022319#comment-17022319
]
ASF subversion and git services commented on AIRFLOW-6359:
----------------------------------------------------------
Commit 432de799fa7b0bf0fb0ecec37082dc12847ce077 in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=432de79 ]
[AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)
(cherry picked from commit 6cc1123e6f2ed09b0fa42fa85fcac97a06ba0924)
> spark_submit_hook.py polling too frequent
> -----------------------------------------
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
> Affects Versions: 1.10.6
> Reporter: t oo
> Assignee: t oo
> Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
