eladkal commented on a change in pull request #6909: [AIRFLOW-6359]
spark_submit_hook.py status polling interval config
URL: https://github.com/apache/airflow/pull/6909#discussion_r361838225
##########
File path: airflow/contrib/hooks/spark_submit_hook.py
##########
@@ -492,8 +496,8 @@ def _start_driver_status_tracking(self):
while self._driver_status not in ["FINISHED", "UNKNOWN",
"KILLED", "FAILED", "ERROR"]:
- # Sleep for 1 second as we do not want to spam the cluster
- time.sleep(1)
+ # Sleep for n seconds as we do not want to spam the cluster
+ time.sleep(self._status_poll_interval)
Review comment:
This actually means occupy worker and do nothing for n seconds is it not?
It was OK when it was 1 second but users may set it to even 5 min without
realising that it occupys the worker.
My comment here is more of a concern rather than an action to do.
Should this work by occupying the worker "indefinitely" or can it be
something like the sensors with (poke/reschedule)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
