[
https://issues.apache.org/jira/browse/AIRAVATA-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739579#comment-16739579
]
Marcus Christie commented on AIRAVATA-2692:
-------------------------------------------
I've been able to reproduce this problem locally. So that rules out it being a
timeout in Apache or mod_wsgi. It appears to be a timeout in Thrift.
The Thrift connection to the API server is opened at the beginning of the
request. It appears that after 15 seconds of nothing being written to it the
connection closes and the first thrift API call after that 15 seconds fails
because the connection is closed.
A better approach would be to use a connection pool for thrift connections and
get connections on demand, as needed. I found the following Thrift connection
pool library: https://github.com/Thriftpy/thrift_connector. I threw together a
quick test and with the {{time.sleep(15)}} the upload still completes.
I'll shift gears on this issue and work on integrating this thrift connection
pool library.
> Django: hitting a 15 second timeout of some sort with large file uploads
> ------------------------------------------------------------------------
>
> Key: AIRAVATA-2692
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2692
> Project: Airavata
> Issue Type: Bug
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> For some reason it fails to get the project when doing a file upload
> {noformat}
> [Fri Feb 23 03:46:37.783086 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] [2018-02-23 03:46:37,782
> django_airavata.apps.api.views:652 ERROR] Failed to upload file
> [Fri Feb 23 03:46:37.783108 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] Traceback (most recent call last):
> [Fri Feb 23 03:46:37.783113 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/django-airavata-gateway/django_airavata/apps/api/views.py",
> line 642, in upload_input_file
> [Fri Feb 23 03:46:37.783118 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] request.authz_token, project_id)
> [Fri Feb 23 03:46:37.783122 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
> line 4733, in getProject
> [Fri Feb 23 03:46:37.783126 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] return self.recv_getProject()
> [Fri Feb 23 03:46:37.783130 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/django-airavata-gateway/airavata/api/Airavata.py",
> line 4746, in recv_getProject
> [Fri Feb 23 03:46:37.783134 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] (fname, mtype, rseqid) = iprot.readMessageBegin()
> [Fri Feb 23 03:46:37.783138 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
> line 134, in readMessageBegin
> [Fri Feb 23 03:46:37.783152 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] sz = self.readI32()
> [Fri Feb 23 03:46:37.783185 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/protocol/TBinaryProtocol.py",
> line 217, in readI32
> [Fri Feb 23 03:46:37.783190 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] buff = self.trans.readAll(4)
> [Fri Feb 23 03:46:37.783194 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
> line 60, in readAll
> [Fri Feb 23 03:46:37.783198 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] chunk = self.read(sz - have)
> [Fri Feb 23 03:46:37.783202 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TTransport.py",
> line 161, in read
> [Fri Feb 23 03:46:37.783206 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] self.__rbuf = BufferIO(self.__trans.read(max(sz,
> self.__rbuf_size)))
> [Fri Feb 23 03:46:37.783210 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] File
> "/var/www/portals/django-seagrid/venv/lib64/python3.4/site-packages/thrift/transport/TSocket.py",
> line 132, in read
> [Fri Feb 23 03:46:37.783239 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] message='TSocket read 0 bytes')
> [Fri Feb 23 03:46:37.783251 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341] thrift.transport.TTransport.TTransportException: TSocket
> read 0 bytes
> [Fri Feb 23 03:46:37.783263 2018] [wsgi:error] [pid 5145] [remote
> 68.45.52.105:56341]
> {noformat}
> This is hitting a 15 second timeout. Not sure where it is at.
> Maybe in mod_wsgi?
> https://groups.google.com/forum/#!topic/modwsgi/zuhFREjOE8M
> http://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html?highlight=timeout
> I know it is a 15 second timeout thing and not related to the size of the
> file. If I put {{time.sleep(15)}} in my Django view handler and upload a very
> small file it triggers the same Thrift error. Weird that it causes Thrift to
> generate this error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
