ashb opened a new issue #9118: URL: https://github.com/apache/airflow/issues/9118
**Description** As mentioned in #8212 if you have configured S3 logs, but there is a problem then this is never surfaced to the UI (nor the webserver logs) making this very hard to debug. All you see in the UI is this: ``` *** Log file does not exist: /usr/local/airflow/logs/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log *** Fetching from: http://MY_DAG_NAME-0dde5ff5a786437cb14234:8793/log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='MY_DAG_NAME-0dde5ff5a786437cb14234', port=8793): Max retries exceeded with url: /log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f708332fc90>: Failed to establish a new connection: [Errno -2] Name or service not known')) ``` In one such case I was debugging, I found this error when attempting to communicate with S3: ```python console >>> c.list_buckets() [2020-06-03 08:27:24,662] {connectionpool.py:735} INFO - Starting new HTTPS connection (1): s3.amazonaws.com Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.7/site-packages/botocore/client.py", line 314, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/lib/python3.7/site-packages/botocore/client.py", line 612, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records. ``` We should add a `***` line showing we attempted to fetch the logs form S3, and the error _that it failed at least_. Right now on error the S3TaskHandler is totally silent in case of error. This is bad. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
