Have you guys tried the fastbinary protocol or twisted?
after reading more source (and some great help from Randy Abernethy), I
*think* I understand how to use fastbinary in python.
For an application to cause thrift-generated python code to use
fastbinary, it seems necessary to do something like this try/except import
statement to select between TBinaryProtocol and TBinaryProtocolAccelerated
Is this the best practice? Is there a reason that TBinaryProtocol does
not do this for us under the hood?
from thrift.protocol.TBinaryProtocol import TBinaryProtocol,
TBinaryProtocolAccelerated
fastbinary_import_failure = None
try:
from thrift.protocol import fastbinary
## use faster C program to read/write
protocol = TBinaryProtocolAccelerated
except Exception, exc:
fastbinary_import_failure = exc
## fall back to pure python
protocol = TBinaryProtocol
Also, in order to make TBinaryProtocolAccelerated work with
TBufferedTransport wrapping a flat file, I had to add this "readAll"
method to the file object before passing into TBufferedTransport.
How does this differ from simply using read(sz)?
def readAll(self, sz):
'''
This method allows TBinaryProtocolAccelerated to actually function.
Copied from here
http://svn.apache.org/repos/asf/hive/trunk/service/lib/py/thrift/transport/TTransport.py
'''
buff = ''
have = 0
while (have < sz):
chunk = self.read(sz-have)
have += len(chunk)
buff += chunk
if len(chunk) == 0:
raise EOFError()
return buff
The code is all here:
https://github.com/trec-kba/streamcorpus/commit/68d752cf7f09c726bf5701324a65d78787349570
jrf
