syedahsn commented on code in PR #30463:
URL: https://github.com/apache/airflow/pull/30463#discussion_r1212380235
##########
airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -901,36 +930,63 @@ def execute(self, context: Context) -> str | None:
raise AirflowException(f"Application Creation failed: {response}")
self.log.info("EMR serverless application created: %s", application_id)
-
- # This should be replaced with a boto waiter when available.
- waiter(
- get_state_callable=self.hook.conn.get_application,
- get_state_args={"applicationId": application_id},
- parse_response=["application", "state"],
- desired_state={"CREATED"},
- failure_states=EmrServerlessHook.APPLICATION_FAILURE_STATES,
- object_type="application",
- action="created",
- countdown=self.waiter_countdown,
- check_interval_seconds=self.waiter_check_interval_seconds,
- )
+ attempt = 0
+ waiter = self.hook.get_waiter("serverless_app_created")
+ while attempt < self.waiter_max_attempts:
Review Comment:
I get what you're saying, and this is certainly not an ideal use of the boto
waiters. Ideally, the waiters would allow logging natively, and we would end up
with a very clean way to wait while also getting information about the current
status of the resources. Unfortunately, until botocore supports that, we have
to manually work to get logging.
That being said, there is still benefit in using custom boto waiters. We get
to make use of botocore's code that allows us to create new waiters without
writing additional code. This is especially important with deferrable operators
because aiobotocore can use the same config file to generate an async waiter
without us having to write async code.
Although we don't get to enjoy the full benefit of using boto waiters, we do
gain something, which I think makes it worth doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
