vandonr-amz commented on code in PR #30757:
URL: https://github.com/apache/airflow/pull/30757#discussion_r1172879459
##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -315,6 +309,32 @@ def get_state(self, response, keys) -> str:
)
return get_state(response=response, keys=keys)
+ def cancel_running_jobs(self, application_id: str, waiter_config: dict =
{}):
+ """
+ List all jobs in an intermediate state and cancel them.
+ Then wait for those jobs to reach a terminal state.
+
+ Note: if new jobs are triggered while this operation is ongoing,
+ it's going to time out and return an error.
+ """
+ r = self.conn.list_job_runs(
+ applicationId=application_id, maxResults=50,
states=list(self.JOB_INTERMEDIATE_STATES)
+ )
+ job_ids = [jr["id"] for jr in r["jobRuns"]]
+ self.log.info("there are %s job(s) running for app %s", len(job_ids),
application_id)
+ if len(job_ids) > 0:
+ self.log.warning(
Review Comment:
hmm, my reasoning was that cancelling jobs is pretty impactful, so it's
something that should be highlighted a bit more. You may not care about the
specific details of the steps of the operator, but you may want to know if it
cancelled jobs or not ? Even if you only set the log level to Warn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
