nuclearpinguin commented on a change in pull request #6909: [AIRFLOW-6359]
spark_submit_hook.py status polling interval config
URL: https://github.com/apache/airflow/pull/6909#discussion_r361816610
##########
File path: airflow/contrib/hooks/spark_submit_hook.py
##########
@@ -492,8 +496,11 @@ def _start_driver_status_tracking(self):
while self._driver_status not in ["FINISHED", "UNKNOWN",
"KILLED", "FAILED", "ERROR"]:
- # Sleep for 1 second as we do not want to spam the cluster
- time.sleep(1)
+ # Sleep for n seconds as we do not want to spam the cluster
+ if self._status_poll_interval:
Review comment:
If we use suggestion from init then we can simply do:
```suggestion
time.sleep(self._status_poll_interval)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
