Fedor Kovalev created THRIFT-2436:
-------------------------------------
Summary: Pass more 2gb unziped message in zlibtransport in python
lib
Key: THRIFT-2436
URL: https://issues.apache.org/jira/browse/THRIFT-2436
Project: Thrift
Issue Type: Improvement
Components: Python - Library
Reporter: Fedor Kovalev
Priority: Minor
lib/py/src/transport/TZlibTransport.py
str 223: zbuf = self._zcomp_write.compress(wout)
Thrift is crushed if len(wout) > 2GB
Something like follow can fix it:
CHUNK_1GB = 1024*1024*1024
def flush(self):
'''
Flush any queued up data in the write buffer and ensure the
compression buffer is flushed out to the underlying transport
'''
zbuf_array = []
message_len = 0
self.__wbuf.seek(0)
while True:
chunk = self.__wbuf.read(self.CHUNK_1GB)
if not chunk:
break
zipped_chunk = self._zcomp_write.compress(chunk)
message_len += len(zipped_chunk)
self.bytes_out += len(chunk)
zbuf_array.append(zipped_chunk)
self.bytes_out_comp += message_len
ztail = self._zcomp_write.flush(zlib.Z_SYNC_FLUSH)
self.bytes_out_comp += len(ztail)
if (message_len + len(ztail)) > 0:
if zbuf_array:
zbuf_array[-1] = zbuf_array[-1] + ztail
else:
zbuf_array.append(ztail)
self.__wbuf = StringIO()
for chunk in zbuf_array:
self.__trans.write(chunk)
self.__trans.flush()
--
This message was sent by Atlassian JIRA
(v6.2#6252)
