june109109 commented on issue #50587:
URL: https://github.com/apache/airflow/issues/50587#issuecomment-3031420054
@ellisms
I'm sorry for late response.
```
~ $ aws emr describe-notebook-execution --notebook-execution-id
ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5
{
"NotebookExecution": {
"NotebookExecutionId": "ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5",
"EditorId": "e-AH8OW5O68ZRXGXYFFLF14HIAG",
"ExecutionEngine": {
"Id": "j-19O4HGHRC48IE",
"Type": "EMR",
"MasterInstanceSecurityGroupId": "sg-0c2a01931b0a299fd"
},
"NotebookExecutionName": "user_operator_test-user_nb",
"NotebookParams": "{\"query\": \"SELECT * FROM
tos.tcic_nrtm_integ_cont_prod WHERE strd_dt = '250702'\", \"key2\": 123,
\"default_datestrings\": {\"yyyymmdd\": \"20250703\", \"yyyy-mm-dd\":
\"2025-07-03\"}}",
"Status": "FINISHED",
"StartTime": "2025-07-03T08:09:36.552000+00:00",
"EndTime": "2025-07-03T08:10:52.757000+00:00",
"Arn":
"arn:aws:elasticmapreduce:ap-northeast-2:302263049595:notebook-execution/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5",
"OutputNotebookURI":
"s3://cosmos-computing-test-emr-20250609234141565800000008/poc/e-AH8OW5O68ZRXGXYFFLF14HIAG/executions/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5/query.ipynb",
"LastStateChangeReason": "Execution is finished for cluster
j-19O4HGHRC48IE.",
"NotebookInstanceSecurityGroupId": "",
"Tags": [],
"OutputNotebookS3Location": {
"Bucket": "cosmos-computing-test-emr-20250609234141565800000008",
"Key":
"poc/e-AH8OW5O68ZRXGXYFFLF14HIAG/executions/ex-J0BYN0I0N06S4UB8DAAFKBJXKKWO5/query.ipynb"
},
"OutputNotebookFormat": "null"
}
}
```
It seems issue occurs because the EMR waiter's internal argument path for
status is set to "notebookexecution.status",
but the actual response from describe_notebook_execution uses
"NotebookExecution.Status" (with upper case letter). The waiter may looks for
response["notebookexecution"]["status"], but actual response is
response["NotebookExecution"]["Status"], so waits while notebook is finished.
I tried custom waiter definition and found out it works well with my case.
```
NOTEBOOK_WAITER_MODEL = {
"version": 2,
"waiters": {
"notebook_execution_terminal": {
"operation": "DescribeNotebookExecution",
"delay": 30,
"maxAttempts": 120,
"acceptors": [
{
"matcher": "path",
"argument": "NotebookExecution.Status",
"expected": "FINISHED",
"state": "success"
},
{
"matcher": "path",
"argument": "NotebookExecution.Status",
"expected": "FAILED",
"state": "failure"
},
]
}
}
}
emr_hook = EmrHook(aws_conn_id=self.aws_conn_id)
emr_client = emr_hook.get_conn()
waiter = BaseBotoWaiter(emr_client,
NOTEBOOK_WAITER_MODEL).waiter("notebook_execution_terminal")
waiter.wait(
NotebookExecutionId=notebook_execution_id,
WaiterConfig=prune_dict({
"Delay": self.waiter_delay,
"MaxAttempts": self.waiter_max_attempts,
}),
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]