vincbeck commented on code in PR #30757:
URL: https://github.com/apache/airflow/pull/30757#discussion_r1172883878
##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -315,6 +309,32 @@ def get_state(self, response, keys) -> str:
)
return get_state(response=response, keys=keys)
+ def cancel_running_jobs(self, application_id: str, waiter_config: dict =
{}):
+ """
+ List all jobs in an intermediate state and cancel them.
+ Then wait for those jobs to reach a terminal state.
+
+ Note: if new jobs are triggered while this operation is ongoing,
+ it's going to time out and return an error.
+ """
+ r = self.conn.list_job_runs(
+ applicationId=application_id, maxResults=50,
states=list(self.JOB_INTERMEDIATE_STATES)
+ )
+ job_ids = [jr["id"] for jr in r["jobRuns"]]
+ self.log.info("there are %s job(s) running for app %s", len(job_ids),
application_id)
+ if len(job_ids) > 0:
+ self.log.warning(
Review Comment:
Hmmm, to me warning is something that might be an error or something not
desired. I am not against using warning but we usually dont use it for
"impactful" things. As an example, when we delete a bucket (much more
impactful), we log it as an info. See
[here](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/s3.py#L114).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
