pankajastro commented on code in PR #32029:
URL: https://github.com/apache/airflow/pull/32029#discussion_r1235285972
##########
airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -507,6 +509,7 @@ def __init__(
max_tries: int | None = None,
tags: dict | None = None,
max_polling_attempts: int | None = None,
+ deferrable: bool = False,
Review Comment:
hmm, but https://github.com/apache/airflow/pull/31712 PR has not been merged
yet so I'll keep it like this until we merge
https://github.com/apache/airflow/pull/31712 wdyt?
##########
airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -555,6 +559,22 @@ def execute(self, context: Context) -> str | None:
self.client_request_token,
self.tags,
)
+ if self.deferrable:
+ timeout = (
+ timedelta(seconds=self.max_polling_attempts *
self.poll_interval + 60)
Review Comment:
60 sec we decided as just a buffer to minimize the chance of a trigger
getting lesser time than a user expected
https://github.com/apache/airflow/pull/30945#discussion_r1218436958
##########
airflow/providers/amazon/aws/triggers/emr.py:
##########
@@ -317,3 +317,75 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
await asyncio.sleep(int(self.poll_interval))
yield TriggerEvent({"status": "success", "job_id": self.job_id})
+
+
+class EmrStepSensorTrigger(BaseTrigger):
+ """
+ Poll for the status of EMR container until reaches terminal state.
+
+ :param virtual_cluster_id: Reference Emr cluster id
+ :param job_id: job_id to check the state
+ :param aws_conn_id: Reference to AWS connection id
+ :param poll_interval: polling period in seconds to check for the status
+ """
+
+ def __init__(
+ self,
+ job_flow_id: str,
+ step_id: str,
+ target_states: Iterable[str],
+ aws_conn_id: str = "aws_default",
+ poll_interval: int = 30,
Review Comment:
Aha, this is a typo, if you see the sensor I'm using poke_interval only :( .
Fixed 04ec3682d6f61d73258b3ed5e757a74dc54beac2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
