[protobuf] Do Protocol Buffers read entire collection of messages into memory from file?

2010-10-19 Thread ksamdev
Hi, I am wondering how do Protocol Buffers read input files? Is the entire file read into memory or some proxy technique is used and entries are read only when required? This is a vital feature for large lists, say, some dataset with 10^9 messages. Do Protocol Buffers use any additional

[protobuf] Picking up the message type / description from an unknown byte[]

2010-10-19 Thread jon.mithe
Hi, I am thinking about using the protocol buffers over XML for a server I am writing. Is there a way to pick up on what message type / descriptor it is without using the specific compiled .proto implementation? The server needs to be very flexible / modular. So I was hoping to develop modules

Re: [protobuf] Do Protocol Buffers read entire collection of messages into memory from file?

2010-10-19 Thread Henner Zeller
On Tue, Oct 19, 2010 at 06:45, ksamdev ksam...@gmail.com wrote: Hi, I am wondering how do Protocol Buffers read input files? Is the entire file read into memory or some proxy technique is used and entries are read only when required? If you have a sequence of messages you process then you'd

Re: [protobuf] valgrind invalid write and double free errors

2010-10-19 Thread Kenton Varda
These don't look like they have anything to do with the I/O change. The FileDescriptor class describes the contents of a .proto file (it's not related to OS file descriptors). The problems you're seeing appear to be happening while running global destructors -- i.e. at program shutdown. Are you

Re: [protobuf] How to compile protocol buffers with stlport on MSVC9

2010-10-19 Thread Kenton Varda
If you're using stlport, make sure _STLPORT_VERSION is #defined. Then the protobuf code should figure out to use it instead of MSVC. You will also have to make sure that the #defines in config.h are correct for stlport. On Sun, Oct 17, 2010 at 8:03 PM, yang kojiyij...@gmail.com wrote: I am

Re: [protobuf] Can we Compress and decompress

2010-10-19 Thread Kenton Varda
Compression is not a built-in feature of protocol buffers, but it is easy and often useful to apply compression on top of the protobuf encoding. In Java, write your data to a GZIPOoutputStream, and then read from a GZIPInputStream. These classes are part of the standard Java library, not the

Re: [protobuf] Picking up the message type / description from an unknown byte[]

2010-10-19 Thread Kenton Varda
On Tue, Oct 19, 2010 at 8:17 AM, jon.mithe jon.mi...@gmail.com wrote: Hi, I am thinking about using the protocol buffers over XML for a server I am writing. Is there a way to pick up on what message type / descriptor it is without using the specific compiled .proto implementation? No.

Re: [protobuf] Can we Compress and decompress

2010-10-19 Thread Igor Gatis
Are C++ and Java GZip streams compatible? On Tue, Oct 19, 2010 at 4:34 PM, Kenton Varda ken...@google.com wrote: Compression is not a built-in feature of protocol buffers, but it is easy and often useful to apply compression on top of the protobuf encoding. In Java, write your data to a

Re: [protobuf] Can we Compress and decompress

2010-10-19 Thread Marc Gravell
In terms of output? GZIP / DEFLATE / etc implementations should be interoperable between platforms, yes. However IIRC the compressed stream is not guaranteed to be byte-identical between specific implementations (but the *decompressed* stream should be byte-identical) Marc On 19 October 2010

Re: [protobuf] Can we Compress and decompress

2010-10-19 Thread Igor Gatis
I mean: if I use Java's GzipOutputStream and then read compressed data using C++'s GzipInputStream, will it work? In other words, are generated streams pure Gzip compliant, in the sense there are no special headers. Could I use unix gzip to unzip a file which its content was generated by Java or

Re: [protobuf] Can we Compress and decompress

2010-10-19 Thread Kenton Varda
On Tue, Oct 19, 2010 at 1:10 PM, Igor Gatis igorga...@gmail.com wrote: In other words, are generated streams pure Gzip compliant, in the sense there are no special headers. Could I use unix gzip to unzip a file which its content was generated by Java or C++ GzipOutputStream? Yes. For C++,

[protobuf] buffer sizes when sending messages from c++ to java

2010-10-19 Thread Paul
Hi, I am sending messages from a C++ client to a Java server, but I am wondering about the length of the buffer I am using to send it over. This is the C++ code: CLIENT SIDE CODE: *** the protocol buffers message is called snap1 *** int numBytesForDelim = sizeof(int); char *snap_buf2; snap_buf2

[protobuf] Re: How to compile protocol buffers with stlport on MSVC9

2010-10-19 Thread yang
On 10月20日, 上午2时28分, Kenton Varda ken...@google.com wrote: If you're using stlport, make sure _STLPORT_VERSION is #defined. Then the protobuf code should figure out to use it instead of MSVC. You will also have to make sure that the #defines in config.h are correct for stlport. thanks

[protobuf] Re: Issue 226 in protobuf: Python API doesn't support reading/writing delimited messages

2010-10-19 Thread protobuf
Updates: Labels: -Type-Defect Type-Enhancement Comment #1 on issue 226 by ken...@google.com: Python API doesn't support reading/writing delimited messages http://code.google.com/p/protobuf/issues/detail?id=226 (No comment was entered for this change.) -- You received this message