jaketf opened a new issue #10293: URL: https://github.com/apache/airflow/issues/10293
**Description** The Dataproc hook now has separate methods for [submitting](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L778) and [waiting for job completion](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L705). We should improve this further by adding a method to simply poke the status of the job. Finally implement a `DataprocJobSensor` so we can use reschedule mode for jobs known to run for a long time. **Use case / motivation** For Long running dataproc jobs that are known to run for a long time (e.g. > 1 hr) we should not block a slot for the entire duration of the job. **Related Issues** N/A **Other notes ** @varundhussa is currently working on this for a customer with this use case. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
