While I am the largest of thrift fans, it is worth noting that Thrift is great for RPC, that is to say, creating really fast cross language microservice interfaces. It is not great for large scale data transfer. Cassandra was originally built with a Thrift API but moved to CQL and a native protocol, purpose built for returning large datasets. API interactions that are characterized by fast, modestly sized transfers work great with Thrift but transferring anything bigger than a few megs, while possible, may not be practical or optimal. I think you could say this about RPC systems in general (protobuf/grpc, MSRPC, RMI, etc.).
On Fri, Apr 22, 2016 at 5:43 AM, Edward Capriolo <[email protected]> wrote: > Thrif messages have to be buffered into memory. I suggest to use paging n > rows at a time to keep latency predictable. > > On Friday, April 22, 2016, Tenghuan He <[email protected]> wrote: > > > Hi there > > > > I have a PostgreSQL database on my server machine, which can only be > > queried locally. Now I want to query the data from another machine. It > > seems that Thrift RPC is a natural and good choice. However I found there > > is some problem when the ResultSet is very large, say there are millions > of > > rows in the ResultSet. Since ResultSet is not serializable and > serializing > > it is meaningless. Returning the materialized rows all at once consumes > two > > much memory. > > I consider returning such as 1000 rows each time > > Is there any other idea or advice? > > > > Thanks in advance > > > > Tenghuan He > > > > > -- > Sorry this was sent from mobile. Will do less grammar and spell check than > usual. >
