Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread dirtysalt1...@gmail.com
3ku for you reply. Yes, actually I know this demand will add a great deal of complication. So I just want to know whether you designers have some idea about how to implement it. en.actually I have test the performance of protobuf,thrift and some other opensource products. protobuf's pack

Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread dirtysalt1...@gmail.com
3ku for your reply:). enActually my application is that I CAN allocate enough ram for the in-memory message object. What I really want is to restrict the size of messages which are sent/received by client/server. Right,now ZeroCopyOutputStream can work. But actually my app is like a RPC

Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread Kenton Varda
That's correct. Sorry, but encoding large chunks of data that cannot be parsed or serialized all at once is an explicit non-goal of protocol buffers. Fulfilling such needs involves adding a great deal of complication to the system. Your case may seem relatively simple, but other, more complicate

Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread Jason Hsueh
Ah, one option I missed is using an implementation of io::ZeroCopyOutputStream like io::FileOutputStream, which uses a fixed size buffer and flushes data to the file (socket) when the buffer is full. Then serializing a large message won't consume a lot of memory. Perhaps this is what you really wan

Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread dirtysalt1...@gmail.com
I think what you means that "I should redesign protocol which implements stream functionality based on protobuf INSTEAD OF expecting protobuf to implement it." What I used to thought is "App -> Protobuf -> Stream Functionality[Protobuf provides stream functionality directly.On the top, my ap

Re: [protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread Jason Hsueh
This really needs to be handled in the application since protobuf has no idea which fields are expendable or can be truncated. What I was trying to suggest earlier was to construct many Req protobufs and serialize those individually. i.e., instead of 1 Req proto with 1,000,000 page ids, construct 1

[protobuf] Re: Questions/Ideas about Protobuf

2010-06-03 Thread dirlt
3ku for you relply:).For the first one,I think your answer is quite clear. But to the second one,en,I want to produce the serialization of Req. Let me explain again:). assume my application is like this: 0.server app wants to send 1,000,000 pageids to client 1.if server app sends 1,000,000 pages