vandonr-amz commented on code in PR #30757:
URL: https://github.com/apache/airflow/pull/30757#discussion_r1171935720
##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -254,11 +253,6 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
kwargs["client_type"] = "emr-serverless"
super().__init__(*args, **kwargs)
- @cached_property
- def conn(self):
- """Get the underlying boto3 EmrServerlessAPIService client (cached)"""
- return super().conn
-
Review Comment:
nothing to do with the sauce, but I took the opportunity to clean this as I
was around
##########
tests/system/providers/amazon/aws/example_emr_serverless.py:
##########
@@ -100,13 +101,14 @@
configuration_overrides=SPARK_CONFIGURATION_OVERRIDES,
)
# [END howto_operator_emr_serverless_start_job]
- start_job.waiter_check_interval_seconds = 10
+ start_job.wait_for_completion = False
# [START howto_sensor_emr_serverless_job]
wait_for_job = EmrServerlessJobSensor(
task_id="wait_for_job",
application_id=emr_serverless_app_id,
job_run_id=start_job.output,
+ target_states=EmrServerlessHook.JOB_INTERMEDIATE_STATES,
Review Comment:
we don't want to wait for the end of the job so that we can force-stop it in
the next step
(btw this makes the test faster to run because we don't have to wait for the
end of the run anymore 😄)
##########
tests/system/providers/amazon/aws/example_emr_serverless.py:
##########
@@ -100,13 +101,14 @@
configuration_overrides=SPARK_CONFIGURATION_OVERRIDES,
)
# [END howto_operator_emr_serverless_start_job]
- start_job.waiter_check_interval_seconds = 10
+ start_job.wait_for_completion = False
Review Comment:
instead of waiting for the job to finish, we just trigger it and continue...
##########
airflow/providers/amazon/aws/waiters/emr-serverless.json:
##########
@@ -0,0 +1,18 @@
+{
+ "version": 2,
+ "waiters": {
+ "no_job_running": {
+ "operation": "ListJobRuns",
+ "delay": 10,
+ "maxAttempts": 60,
+ "acceptors": [
+ {
+ "matcher": "path",
+ "argument": "length(jobRuns) == `0`",
+ "expected": true,
+ "state": "success"
+ }
+ ]
Review Comment:
I don't like too much the fact that I don't have a failure case for this
waiter, but I think there is nothing we can do about it...
The failure case would be count > prev_count, or detecting a new job_id that
we didn't see in the previous calls, which is way beyond the capabilities of
waiters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
