Jorge Alpedrinha Ramos created AIRFLOW-1037: -----------------------------------------------
Summary: Webserver can't fetch large log files from Remote worker Key: AIRFLOW-1037 URL: https://issues.apache.org/jira/browse/AIRFLOW-1037 Project: Apache Airflow Issue Type: Bug Components: celery, logging, webserver Affects Versions: Airflow 1.8 Environment: Running airflow with docker using CeleryExecutor Reporter: Jorge Alpedrinha Ramos When tasks logs are too big, the webserver isn't able to fetch them. This may be related with https://github.com/kennethreitz/requests/issues/2422. I've recovered a stacktrace from the worker: {code} worker_1 | [2017-03-24 10:29:39,574] {_internal.py:87} INFO - 172.18.0.8 - - [24/Mar/2017 10:29:39] "GET /log/rates.ticker-to-analytics/parse-syslog/2017-01-12T07:45:00 HTTP/1.1" 200 - worker_1 | Traceback (most recent call last): worker_1 | File "/usr/local/bin/airflow", line 28, in <module> worker_1 | args.func(args) worker_1 | File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 858, in serve_logs worker_1 | host='0.0.0.0', port=WORKER_LOG_SERVER_PORT) worker_1 | File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 843, in run worker_1 | run_simple(host, port, self, **options) worker_1 | File "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 736, in run_simple worker_1 | inner() worker_1 | File "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 699, in inner worker_1 | srv.serve_forever() worker_1 | File "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 536, in serve_forever worker_1 | HTTPServer.serve_forever(self) worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 233, in serve_forever worker_1 | self._handle_request_noblock() worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 292, in _handle_request_noblock worker_1 | self.handle_error(request, client_address) worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 290, in _handle_request_noblock worker_1 | self.process_request(request, client_address) worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 318, in process_request worker_1 | self.finish_request(request, client_address) worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 331, in finish_request worker_1 | self.RequestHandlerClass(request, client_address, self) worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 654, in __init__ worker_1 | self.finish() worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 713, in finish worker_1 | self.wfile.close() worker_1 | File "/usr/local/lib/python2.7/socket.py", line 283, in close worker_1 | self.flush() worker_1 | File "/usr/local/lib/python2.7/socket.py", line 307, in flush worker_1 | self._sock.sendall(view[write_offset:write_offset+buffer_size]) worker_1 | socket.error: [Errno 32] Broken pipe {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)