[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022318#comment-17022318
]
ASF subversion and git services commented on AIRFLOW-6359:
----------------------------------------------------------
Commit 1591b656388ddabf9ac2765bc94e59dc458f7525 in airflow's branch
refs/heads/v1-10-test from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1591b65 ]
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
* AIRFLOW-6359
* pep checks
* make travis CI refire
* new section = bad
* move poll_interval into the operator instead of .cfg
* review comments
(cherry picked from commit 83ba8e638fb539afd6dac495a8c59012d0aaa9f4)
> spark_submit_hook.py polling too frequent
> -----------------------------------------
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
> Affects Versions: 1.10.6
> Reporter: t oo
> Assignee: t oo
> Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
