Subham-KRLX commented on issue #67178: URL: https://github.com/apache/airflow/issues/67178#issuecomment-4506530873
First the waiter throttling issue is being fixed in PR #67222 that addresses throttling errors being immediately failed instead of retried. Second your real problem is that the TaskInstance is being marked as FAILED at around 16 seconds while the job is still running in EMR. Your logs show the job submitted at 16:01:25 but terminated at 16:01:41. This suggests your execution timeout is too short or the task is not sending heartbeats properly. To investigate can you share what execution timeout you set on EmrServerlessStartJobOperator and whether deferrable is set to true. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
