GitHub user uplsh580 edited a discussion: S3 Remote Logging Failure After
Upgrading Airflow from 2.9.3 to 2.10.3
Hello Airflow Community,
I recently upgraded my Airflow environment from version 2.9.3 to 2.10.3. After
the upgrade, I encountered a critical issue where S3 remote logging is failing.
Details:
- Airflow Version Before Upgrade: 2.9.3 (w/ apache-airflow-providers-amazon
8.25.0)
- Airflow Version After Upgrade: 2.10.3 (w/ apache-airflow-providers-amazon
9.2.0)
- Issue: Logs are not being uploaded to our S3 bucket in the private cloud, but
existing logs can still be retrieved without issue.
- When viewing task logs that were previously uploaded to S3, the web server
retrieves the logs from S3 and displays them correctly.
-
- However, since version 2.10.5, it seems that task logs are not being
uploaded to S3 correctly.
-
- The worker generates the following error:
```
│ botocore.exceptions.ClientError: An error occurred (MissingContentLength)
when calling the PutObject operation: The internal error code is -2011
│ [2025-05-27 09:52:29,115: INFO/ForkPoolWorker-16] Task
airflow.providers.celery.executors.celery_executor_utils.execute_command[5c4a3859-ab58-4729-bd77-2fc5c7f6c0db]
succeeded in 2.15678022056818s: None
│ [2025-05-27 09:52:29,241: WARNING/ForkPoolWorker-15] Failed attempt to write
logs to
s3://airflow-adhoc/logs/dag_id=tutorial/run_id=scheduled__2025-05-26T00:00:00+00:00/task_id=sleep/attempt=1.log,
will retry
│ [2025-05-27 09:52:29,248: ERROR/ForkPoolWorker-15] Could not write logs to
s3://airflow-adhoc/logs/dag_id=tutorial/run_id=scheduled__2025-05-26T00:00:00+00:00/task_id=sleep/attempt=1.log
│ Traceback (most recent call last):
│ File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/log/s3_task_handler.py",
line 179, in s3_write
│ self.hook.load_string(
│ File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 153, in wrapper
│ return func(*bound_args.args, **bound_args.kwargs)
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 126, in wrapper
│ return func(*bound_args.args, **bound_args.kwargs)
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 1198, in load_string
│ self._upload_file_obj(f, key, bucket_name, replace, encrypt, acl_policy)
│ File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 1281, in _upload_file_obj
│ client.upload_fileobj(
│ File
"/home/airflow/.local/lib/python3.12/site-packages/boto3/s3/inject.py", line
642, in upload_fileobj
│ return future.result()
│ ^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/s3transfer/futures.py", line
103, in result
│ return self._coordinator.result()
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/s3transfer/futures.py", line
264, in result
│ raise self._exception
│ File
"/home/airflow/.local/lib/python3.12/site-packages/s3transfer/tasks.py", line
135, in __call__
│ return self._execute_main(kwargs)
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/s3transfer/tasks.py", line
158, in _execute_main
│ return_value = self._main(**kwargs)
│ ^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/s3transfer/upload.py", line
796, in _main
│ client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
│ File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/client.py", line
569, in _api_call
│ return self._make_api_call(operation_name, kwargs)
│ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ File
"/home/airflow/.local/lib/python3.12/site-packages/botocore/client.py", line
1023, in _make_api_call
│ raise error_class(parsed_response, operation_name)
│ botocore.exceptions.ClientError: An error occurred (MissingContentLength)
when calling the PutObject operation: The internal error code is -2011
│ [2025-05-27 09:52:29,278: INFO/ForkPoolWorker-15] Task
airflow.providers.celery.executors.celery_executor_utils.execute_command[28f43837-d9d8-43a5-a1f6-e1066aa80cf6]
succeeded in 6.948338359594345s: None
```
- The connection information is configured as follows:
```
conn_type=aws
extra={
"aws_access_key_id": "{ACCESS_KEY_ID}",
"aws_secret_access_key": "{ACCESS_KEY}",
"endpoint_url": "{OUR_ORG_ENDPOINT_URL}",
"service_config": {
"s3": {
"bucket_name": "{MY_BUCKET_NAME}"
}
}
}
```
- Airflow config
```
logging:
remote_logging: 'True'
remote_base_log_folder: 's3://{MY_BUCKET_NAME}/logs'
remote_log_conn_id: vos_test
delete_local_logs: 'True'
colored_console_log: 'True'
```
The connection settings have not changed from version 2.9.3 to 2.10.5. I added
the service_config based on the [breaking changes in the Amazon provider
9.0.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.2.0/changelog.html#:~:text=Removed%20S3%20connection%20type.%20Please%20use%20aws%20as%20conn_type%20instead%2C%20and%20specify%20bucket_name%20in%20service_config.s3%20within%20extras),
as it was continuously failing, but it still does not work.
I'm not sure which setting I might have configured incorrectly.
Could the connection settings be incorrect?
FYI. When I downgrade to version 2.9.3 without changing any config of
connections, remote logging works correctly.
GitHub link: https://github.com/apache/airflow/discussions/51106
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
