ashb opened a new issue #9118:
URL: https://github.com/apache/airflow/issues/9118


   **Description**
   
   As mentioned in #8212 if you have configured S3 logs, but there is a problem 
then this is never surfaced to the UI (nor the webserver logs) making this very 
hard to debug.
   
   All you see in the UI is this:
   
   ```
   *** Log file does not exist: 
/usr/local/airflow/logs/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log
   *** Fetching from: 
http://MY_DAG_NAME-0dde5ff5a786437cb14234:8793/log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log
   *** Failed to fetch log file from worker. 
HTTPConnectionPool(host='MY_DAG_NAME-0dde5ff5a786437cb14234', port=8793): Max 
retries exceeded with url: 
/log/MY_DAG_NAME/MY_TASK_NAME/2020-04-07T20:59:19.312402+00:00/6.log (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f708332fc90>: Failed to establish a new connection: [Errno -2] Name or 
service not known'))
   ```
   
   In one such case I was debugging, I found this error when attempting to 
communicate with S3:
   
   ```python console
   >>> c.list_buckets()
   [2020-06-03 08:27:24,662] {connectionpool.py:735} INFO - Starting new HTTPS 
connection (1): s3.amazonaws.com
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/usr/lib/python3.7/site-packages/botocore/client.py", line 314, in 
_api_call
       return self._make_api_call(operation_name, kwargs)
     File "/usr/lib/python3.7/site-packages/botocore/client.py", line 612, in 
_make_api_call
       raise error_class(parsed_response, operation_name)
   botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when 
calling the ListBuckets operation: The AWS Access Key Id you provided does not 
exist in our records.
   ```
   
   We should add a `***` line showing we attempted to fetch the logs form S3, 
and the error _that it failed at least_. Right now on error the S3TaskHandler 
is totally silent in case of error. This is bad.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to