Hi,
I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came
across an issue where uploading files larger than ~3GB via the API into
histories and libraries fails.
Steps to reproduce (using BioBlend 0.7.0):
from bioblend.galaxy.objects import GalaxyInstance
gi = GalaxyInstance("galaxy-dev", "api_key")
h = gi.histories.create()
h.upload_dataset('/path/to/multigigabytefile')
Observed results:
ConnectionError: Unexpected response from galaxy: 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.4.6 (Ubuntu)
It looks like Nginx is accepting the full incoming data transfer over the
network and saving it to disk successfully, and the error occurs during
the subsequent forwarding of the request to Galaxy. There doesn't seem to
be a particular threshold for file size (for example, 2GB and 2.7GB file
can be uploaded successfully). Also, large file uploads do work when
Galaxy is not running behind Nginx.
Relevant log entries and config details are attached. I'd appreciate any
help.
Thanks,
Ilya
Nginx log:
2016/04/22 19:03:19 [error] 12825#0: *14311 upstream timed out (110: Connection
timed out) while sending request to upstream, client: 134.174.183.88, server: ,
request: "POST /api/tools HTTP/1.1", upstream:
"http://127.0.0.1:8080/api/tools;, host: "galaxy-dev"
Galaxy log:
134.174.183.88 - - [22/Apr/2016:19:01:56 +] "POST /api/tools HTTP/1.0" 500
- "-" "python-requests/2.9.1"
Error - : The client disconnected
while sending the POST/PUT body (1183825676 more bytes were expected)
URL: http://galaxy-dev/api/tools
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/middleware/error.py',
line 151 in __call__
app_iter = self.application(environ, sr_checker)
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/recursive.py',
line 85 in __call__
return self.application(environ, start_response)
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py',
line 640 in __call__
return self.application(environ, start_response)
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 126 in
__call__
return self.handle_request( environ, start_response )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 153 in
handle_request
trans = self.transaction_factory( environ )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 66 in
self.set_transaction_factory( lambda e: self.transaction_chooser( e,
galaxy_app, session_cookie ) )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 97 in
transaction_chooser
return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 193 in
__init__
self.error_message = self._authenticate_api( session_cookie )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 308 in
_authenticate_api
api_key = self.request.params.get('key', None)
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 853 in params
params = NestedMultiDict(self.GET, self.POST)
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 789 in POST
self.make_body_seekable()
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 943 in make_body_seekable
self.copy_body()
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 963 in copy_body
did_copy = self._copy_body_tempfile()
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 980 in _copy_body_tempfile
data = input.read(min(todo, 65536))
File
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
line 1549 in readinto
+ "(%d more bytes were expected)" % self.remaining
DisconnectionError: The client disconnected while sending the POST/PUT body
(1183825676 more bytes were expected)
CGI Variables
-
CONTENT_LENGTH: '4628837012'
CONTENT_TYPE: 'multipart/form-data; boundary=cf3af77a8a3445f98ce91060761385a7'
HTTP_ACCEPT: '*/*'
HTTP_ACCEPT_ENCODING: 'gzip, deflate'
HTTP_CONNECTION: 'close'
HTTP_HOST: 'galaxy-dev'
HTTP_USER_AGENT: 'python-requests/2.9.1'
HTTP_X_FORWARDED_FOR: '134.174.183.88'
HTTP_X_FORWARDED_HOST: 'galaxy-dev'
ORGINAL_HTTP_HOST: 'galaxy_app'
ORGINAL_REMOTE_ADDR: '127.0.0.1'
PATH_INFO: '/api/tools'
REMOTE_ADDR: '134.174.183.88'
REQUEST_METHOD: 'POST'
SERVER_NAME: '127.0.0.1'
SERVER_PORT: '8080'
SERVER_PROTOCOL: 'HTTP/1.0'
WSGI Variables
--
application:
is_api_request: True
paste.expected_exceptions: []
paste.httpexceptions:
paste.httpserver.proxy.host: 'dummy'
paste.httpserver.proxy.scheme: 'http'
paste.httpserver.thread_pool:
paste.recursive.forward:
paste.recursive.include:
paste.recursive.include_app_iter:
paste.recursive.script_name: ''
paste.throw_errors: True
request_id: 'af50af2608bc11e6b6a90a4d98d6c597'
webob._body_file:
