hello, TL;DR version: i need to transfer a huge amount of data (millions of small objects) using a single call to a THsHaServer (or TThreaderSelectorServer)
The long version: i built a java server around TThreadPoolServer that accepts an extremely large amount of data (millions of small objects) via a single call every hour or so (from a python script) and also accepts various requests for small amounts of data tens of times per second. the requests come from python, java, and perl hence why i went with thrift in the first place. this worked great using TBufferedTransport until i had to modify the java server to support async clients. i've rewritten my service to use the THsHaServer and switched to TFramedTransport on the clients. everything worked great (my test suite was happy) but it turns out in production the hourly data update call far far exceeds the maximum size allowed in TFramedTransport: Frame size (150283373) larger than max length (16384000)! yes i realize this is not what thrift was really designed for but it was working great until the async requirement came about. the data is a list of millions of small objects, not one large piece of data that could be streamed, by the way. my options at this point are (in order of difficulty): . run a second thrift server in my java server that uses TThreadPoolServer on a different port to allow using TBufferedTransport for the update call . find a different (non-thrift) mechanism to transfer data from the python update script to the java service . rewrite the update logic to chunk the data none of these are particularly appealing due to the nature of how my service works (the update needs to be fairly atomic, the objects are not trivial, etc.). i'm going to go with option a unless there is a way to transfer a huge amount of data in a single call that does work with HsHa (or ThreadedSelector) and was just wondering if anyone had any better ideas. thanks, ~j -- jules cisek | [email protected]
