syedahsn commented on PR #32151:
URL: https://github.com/apache/airflow/pull/32151#issuecomment-1608519851
The error messages might not explain the root cause of the problem, but they
will at least be a starting point for an investigation. This is an example
where I manually deleted the cluster before the job was run:
```
[2023-06-26, 23:47:57 UTC] {emr.py:146} INFO - Adding steps to
j-2D6XS5VTI1WGZ
[2023-06-26, 23:47:57 UTC] {emr.py:171} INFO - Steps
['s-08516095GAHC1SG5B4N'] added to JobFlow
[2023-06-26, 23:47:57 UTC] {waiter_with_logging.py:74} INFO - EMR Step
status is: PENDING - {}
[2023-06-26, 23:48:36 UTC] {emr.py:188} ERROR - EMR Steps failed:
{'Message': 'Job terminated'}
[2023-06-26, 23:48:36 UTC] {taskinstance.py:1916} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/opt/airflow/airflow/providers/amazon/aws/utils/waiter_with_logging.py", line
68, in wait
waiter.wait(**args, WaiterConfig={"MaxAttempts": 1})
File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 55,
in wait
Waiter.wait(self, **kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line
375, in wait
raise WaiterError(
botocore.exceptions.WaiterError: Waiter StepComplete failed: Waiter
encountered a terminal failure state: For expression "Step.Status.State" we
matched expected path: "CANCELLED"
```
This message is logged `EMR Steps failed: {'Message': 'Job terminated'}`,
which provides some indication of what happened.
If a job fails due to some issue with the job itself, then, like you said,
that information will not be logged here.
I agree its not much, but its better than nothing. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
