vincbeck commented on issue #39026:
URL: https://github.com/apache/airflow/issues/39026#issuecomment-2146150594
```
[2024-06-03, 21:20:21 UTC] {ecs.py:638} INFO - ECS Task started: {'tasks':
[{'attachments': [{'id': '0fbe5b10-140f-4f0e-a784-d2926a536039', 'type':
'ElasticNetworkInterface', 'status': 'PRECREATED', 'details': [{'name':
'subnetId', 'value': 'subnet-03fe35179d25a0036'}]}], 'attributes': [{'name':
'ecs.cpu-architecture', 'value': 'x86_64'}], 'availabilityZone': 'us-east-1b',
'clusterArn':
'arn:aws:ecs:us-east-1:722404908466:cluster/SysTestCluster_example_ecs',
'containerInstanceArn':
'arn:aws:ecs:us-east-1:722404908466:container-instance/SysTestCluster_example_ecs/2c5a500f118248869ab01a84f9ece973',
'containers': [{'containerArn':
'arn:aws:ecs:us-east-1:722404908466:container/SysTestCluster_example_ecs/b392893f198842678e63d2258de21bd8/40baa9f8-fc4f-4709-b7f2-d48275d18e0f',
'taskArn':
'arn:aws:ecs:us-east-1:722404908466:task/SysTestCluster_example_ecs/b392893f198842678e63d2258de21bd8',
'name': 'enve15c32d4-container', 'image': 'ubuntu', 'lastStatus': 'PENDING',
'networkInterfaces': []
, 'cpu': '0'}], 'cpu': '256', 'createdAt': datetime.datetime(2024, 6, 3, 21,
20, 21, 47000, tzinfo=tzlocal()), 'desiredStatus': 'RUNNING',
'enableExecuteCommand': False, 'group': 'family:enve15c32d4-task-definition',
'lastStatus': 'PROVISIONING', 'launchType': 'EC2', 'memory': '512',
'overrides': {'containerOverrides': [{'name': 'enve15c32d4-container',
'command': ['for i in $(seq 1 10); do date; sleep 10; done']}],
'inferenceAcceleratorOverrides': []}, 'startedBy': '***', 'tags': [],
'taskArn':
'arn:aws:ecs:us-east-1:722404908466:task/SysTestCluster_example_ecs/b392893f198842678e63d2258de21bd8',
'taskDefinitionArn':
'arn:aws:ecs:us-east-1:722404908466:task-definition/enve15c32d4-task-definition:1',
'version': 1}], 'failures': [], 'ResponseMetadata': {'RequestId':
'094dc144-7408-4e5c-b64c-1171f80064ed', 'HTTPStatusCode': 200, 'HTTPHeaders':
{'x-amzn-requestid': '094dc144-7408-4e5c-b64c-1171f80064ed', 'content-type':
'application/x-amz-json-1.1', 'content-length': '1550', 'date': 'Mo
n, 03 Jun 2024 21:20:20 GMT'}, 'RetryAttempts': 0}}
[2024-06-03, 21:20:21 UTC] {ecs.py:641} INFO - ECS task ID is:
b392893f198842678e63d2258de21bd8
[2024-06-03, 21:20:21 UTC] {ecs.py:564} INFO - Starting ECS Task Log Fetcher
[2024-06-03, 21:20:26 UTC] {base.py:84} INFO - Using connection ID
'aws_default' for task execution.
[2024-06-03, 21:20:26 UTC] {credentials.py:1278} INFO - Found credentials in
shared credentials file: ~/.aws/credentials
[2024-06-03, 21:20:26 UTC] {task_log_fetcher.py:76} INFO - Cannot find log
stream yet, it can take a couple of seconds to show up. If this error persists,
check that the log group and stream are correct: group: /ecs_test/enve15c32d4
stream: ecs/enve15c32d4-container/b392893f198842678e63d2258de21bd8
[2024-06-03, 21:20:32 UTC] {task_log_fetcher.py:76} INFO - Cannot find log
stream yet, it can take a couple of seconds to show up. If this error persists,
check that the log group and stream are correct: group: /ecs_test/enve15c32d4
stream: ecs/enve15c32d4-container/b392893f198842678e63d2258de21bd8
[2024-06-03, 21:20:52 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:20:44,439] Mon Jun 3 21:20:44 UTC 2024
[2024-06-03, 21:21:03 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:20:54,441] Mon Jun 3 21:20:54 UTC 2024
[2024-06-03, 21:21:13 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:04,443] Mon Jun 3 21:21:04 UTC 2024
[2024-06-03, 21:21:24 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:14,445] Mon Jun 3 21:21:14 UTC 2024
[2024-06-03, 21:21:29 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:24,447] Mon Jun 3 21:21:24 UTC 2024
[2024-06-03, 21:21:40 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:34,449] Mon Jun 3 21:21:34 UTC 2024
[2024-06-03, 21:21:50 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:44,451] Mon Jun 3 21:21:44 UTC 2024
[2024-06-03, 21:22:01 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:21:54,453] Mon Jun 3 21:21:54 UTC 2024
[2024-06-03, 21:22:11 UTC] {task_log_fetcher.py:63} INFO - [2024-06-03
21:22:04,455] Mon Jun 3 21:22:04 UTC 2024
[2024-06-03, 21:22:20 UTC] {timeout.py:68} ERROR - Process timed out, PID:
2131
[2024-06-03, 21:22:20 UTC] {taskinstance.py:741} INFO - ::group::Post task
execution logs
[2024-06-03, 21:22:20 UTC] {ecs.py:751} INFO - {'task': {'attachments': [],
'availabilityZone': 'us-east-1b', 'clusterArn':
'arn:aws:ecs:us-east-1:722404908466:cluster/SysTestCluster_example_ecs',
'connectivity': 'CONNECTED', 'connectivityAt': datetime.datetime(2024, 6, 3,
21, 20, 21, 47000, tzinfo=tzlocal()), 'containerInstanceArn':
'arn:aws:ecs:us-east-1:722404908466:container-instance/SysTestCluster_example_ecs/2c5a500f118248869ab01a84f9ece973',
'containers': [], 'cpu': '256', 'createdAt': datetime.datetime(2024, 6, 3, 21,
20, 21, 47000, tzinfo=tzlocal()), 'desiredStatus': 'STOPPED',
'enableExecuteCommand': False, 'executionStoppedAt': datetime.datetime(2024, 6,
3, 21, 22, 14, 528000, tzinfo=tzlocal()), 'group':
'family:enve15c32d4-task-definition', 'lastStatus': 'DEPROVISIONING',
'launchType': 'EC2', 'memory': '512', 'overrides': {'containerOverrides': [],
'inferenceAcceleratorOverrides': []}, 'pullStartedAt': datetime.datetime(2024,
6, 3, 21, 20, 34, 432000, tzinfo=tzlocal())
, 'pullStoppedAt': datetime.datetime(2024, 6, 3, 21, 20, 34, 579000,
tzinfo=tzlocal()), 'startedAt': datetime.datetime(2024, 6, 3, 21, 20, 34,
471000, tzinfo=tzlocal()), 'startedBy': '***', 'stopCode':
'EssentialContainerExited', 'stoppedReason': 'Essential container in task
exited', 'stoppingAt': datetime.datetime(2024, 6, 3, 21, 22, 15,
tzinfo=tzlocal()), 'tags': [], 'taskArn':
'arn:aws:ecs:us-east-1:722404908466:task/SysTestCluster_example_ecs/b392893f198842678e63d2258de21bd8',
'taskDefinitionArn':
'arn:aws:ecs:us-east-1:722404908466:task-definition/enve15c32d4-task-definition:1',
'version': 4}, 'ResponseMetadata': {'RequestId':
'd60743a6-6116-4b48-99e7-df188833aab5', 'HTTPStatusCode': 200, 'HTTPHeaders':
{'x-amzn-requestid': 'd60743a6-6116-4b48-99e7-df188833aab5', 'content-type':
'application/x-amz-json-1.1', 'content-length': '1148', 'date': 'Mon, 03 Jun
2024 21:22:19 GMT'}, 'RetryAttempts': 0}}
[2024-06-03, 21:22:20 UTC] {taskinstance.py:3137} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/taskinstance.py", line 760, in
_execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/models/taskinstance.py", line 728, in
_execute_callable
return ExecutionCallableRunner(
File "/opt/airflow/airflow/utils/operator_helpers.py", line 250, in run
return self.func(*args, **kwargs)
File "/opt/airflow/airflow/models/baseoperator.py", line 406, in wrapper
return func(self, *args, **kwargs)
File "/opt/airflow/airflow/providers/amazon/aws/operators/ecs.py", line
569, in execute
self._wait_for_task_ended()
File "/opt/airflow/airflow/providers/amazon/aws/operators/ecs.py", line
663, in _wait_for_task_ended
waiter.wait(
File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 55,
in wait
Waiter.wait(self, **kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line
393, in wait
time.sleep(sleep_amount)
File "/opt/airflow/airflow/utils/timeout.py", line 69, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 2131
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
