Hi,

I'm running Galaxy 16.01 behind Nginx 1.4.6 on Ubuntu 14.04 and I came
across an issue where uploading files larger than ~3GB via the API into
histories and libraries fails.

Steps to reproduce (using BioBlend 0.7.0):

from bioblend.galaxy.objects import GalaxyInstance
gi = GalaxyInstance("galaxy-dev", "api_key")
h = gi.histories.create()
h.upload_dataset('/path/to/multigigabytefile')

Observed results:

ConnectionError: Unexpected response from galaxy: 504: <html>

<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

It looks like Nginx is accepting the full incoming data transfer over the
network and saving it to disk successfully, and the error occurs during
the subsequent forwarding of the request to Galaxy. There doesn't seem to
be a particular threshold for file size (for example, 2GB and 2.7GB file
can be uploaded successfully). Also, large file uploads do work when
Galaxy is not running behind Nginx.


Relevant log entries and config details are attached. I'd appreciate any
help.

Thanks,
Ilya


Nginx log:

2016/04/22 19:03:19 [error] 12825#0: *14311 upstream timed out (110: Connection 
timed out) while sending request to upstream, client: 134.174.183.88, server: , 
request: "POST /api/tools HTTP/1.1", upstream: 
"http://127.0.0.1:8080/api/tools";, host: "galaxy-dev"


Galaxy log:

134.174.183.88 - - [22/Apr/2016:19:01:56 +0000] "POST /api/tools HTTP/1.0" 500 
- "-" "python-requests/2.9.1"
Error - <class 'webob.request.DisconnectionError'>: The client disconnected 
while sending the POST/PUT body (1183825676 more bytes were expected)
URL: http://galaxy-dev/api/tools
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/middleware/error.py', 
line 151 in __call__
app_iter = self.application(environ, sr_checker)
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/recursive.py',
 line 85 in __call__
return self.application(environ, start_response)
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py',
 line 640 in __call__
return self.application(environ, start_response)
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 126 in 
__call__
return self.handle_request( environ, start_response )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 153 in 
handle_request
trans = self.transaction_factory( environ )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 66 in 
<lambda>
self.set_transaction_factory( lambda e: self.transaction_chooser( e, 
galaxy_app, session_cookie ) )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 97 in 
transaction_chooser
return GalaxyWebTransaction( environ, galaxy_app, self, session_cookie )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 193 in 
__init__
self.error_message = self._authenticate_api( session_cookie )
File '/mnt/galaxy/galaxy-app/lib/galaxy/web/framework/webapp.py', line 308 in 
_authenticate_api
api_key = self.request.params.get('key', None)
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 853 in params
params = NestedMultiDict(self.GET, self.POST)
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 789 in POST
self.make_body_seekable()
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 943 in make_body_seekable
self.copy_body()
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 963 in copy_body
did_copy = self._copy_body_tempfile()
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 980 in _copy_body_tempfile
data = input.read(min(todo, 65536))
File 
'/mnt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/webob/request.py',
 line 1549 in readinto
+ "(%d more bytes were expected)" % self.remaining
DisconnectionError: The client disconnected while sending the POST/PUT body 
(1183825676 more bytes were expected)

CGI Variables
-------------
CONTENT_LENGTH: '4628837012'
CONTENT_TYPE: 'multipart/form-data; boundary=cf3af77a8a3445f98ce91060761385a7'
HTTP_ACCEPT: '*/*'
HTTP_ACCEPT_ENCODING: 'gzip, deflate'
HTTP_CONNECTION: 'close'
HTTP_HOST: 'galaxy-dev'
HTTP_USER_AGENT: 'python-requests/2.9.1'
HTTP_X_FORWARDED_FOR: '134.174.183.88'
HTTP_X_FORWARDED_HOST: 'galaxy-dev'
ORGINAL_HTTP_HOST: 'galaxy_app'
ORGINAL_REMOTE_ADDR: '127.0.0.1'
PATH_INFO: '/api/tools'
REMOTE_ADDR: '134.174.183.88'
REQUEST_METHOD: 'POST'
SERVER_NAME: '127.0.0.1'
SERVER_PORT: '8080'
SERVER_PROTOCOL: 'HTTP/1.0'

WSGI Variables
--------------
application: <paste.recursive.RecursiveMiddleware object at 0x7fcce108fcd0>
is_api_request: True
paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>]
paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object at 
0x7fcce108fc50>
paste.httpserver.proxy.host: 'dummy'
paste.httpserver.proxy.scheme: 'http'
paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 
0x7fcce0aabb90>
paste.recursive.forward: <paste.recursive.Forwarder from />
paste.recursive.include: <paste.recursive.Includer from />
paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from />
paste.recursive.script_name: ''
paste.throw_errors: True
request_id: 'af50af2608bc11e6b6a90a4d98d6c597'
webob._body_file: (<_io.BufferedReader>, <socket._fileobject object at 
0x7fccc43f62d0 length=4628837012>)
webob._parsed_query_vars: (GET([]), '')
wsgi process: 'Multithreaded'


Nginx config:

/etc/nginx/sites-enabled/default.server:
upstream galaxy_app {
  server 127.0.0.1:8080;
}
server {
  listen 80;
  client_max_body_size 10G;
  proxy_read_timeout 1200s;

  include /etc/nginx/sites-enabled/*.locations;
}

/etc/nginx/sites-enabled/galaxy.locations:
location /_upload {
  upload_store /mnt/galaxy/upload_store;
  upload_pass_form_field "";
  upload_set_form_field "__${upload_field_name}__is_composite" "true";
  upload_set_form_field "__${upload_field_name}__keys" "name path";
  upload_set_form_field "${upload_field_name}_name" "$upload_file_name";
  upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path";
  upload_pass_args on;
  upload_pass /_upload_done;
}
location /_upload_done {
  set $dst /api/tools;
  if ($args ~ nginx_redir=([^&]+)) {
    set $dst $1;
  }
  rewrite "" $dst;
}
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to