Re: [protobuf] Using a ByteBuffer instead of a ByteString?
On Jan 11, 2011, at 0:45 , Nicolae Mihalache wrote: But I have noticed in java that it is impossible to create a message containing a bytes fields without copying some buffers around. For example if I have a encoded message of 1MB with a few regular fields and one big bytes field, decoding the message will make a copy of the entire buffer instead of keeping a reference to it. By decoding I'm assuming you mean deserializing the message from a file or something. This is a disadvantage, but it makes things much easier: it means the buffer used to read data can be recycled for the next message. Without this copy, the library would need to do complicated tracking of chunks of memory to determine if they are in use or not. However, now that you mention it: in the case of big buffers, CodedInputStream.readBytes() gets called, which currently makes 2 copies of the data (it calls readRawBytes() then calls ByteString.copyFrom()). This could probably be fixed in CodedInputStream.readBytes(), which might improve performance a fair bit. I'll put this on my TODO list of things to look at, since I think my code does this pretty frequently. Even worse when encoding: if I read some data from file, does not seem possible to put it directly into a ByteString so I have to make first a byte[], then copy it into the ByteString and when encoding, it makes yet another byte[]. The copy cannot be avoided because it makes the API simpler (thread- safety, don't need to worry about the ByteBuffer being accidentally changed, etc). The latest version of Protocol Buffers in Subversion has ByteString.copyFrom(ByteBuffer) which will do what you want efficiently. Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Using the --proto_path option
I am trying to understand how --proto_path works. Let's say in my .proto file have the following import lines: import descriptor.proto; import google/protobuf/descriptor.proto; What would the path be for the option? I have tried using different path: protoc --proto_path=/usr/local/include/google/protobuf -- cpp_out=. protoc --proto_path=/usr/local/include --cpp_out=. The only way I have gotten it to work is to copy the descriptor.proto to the current directory for the first import line above. Or create sub-directories matching path in the second import line. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Using the --proto_path option
On Tue, Jan 11, 2011 at 1:30 PM, Collin blackfr...@gmail.com wrote: I am trying to understand how --proto_path works. Let's say in my .proto file have the following import lines: import descriptor.proto; import google/protobuf/descriptor.proto; Delete the first of these two imports, and then use --proto_path=/usr/local/include. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Truncated varint decoding a message just built
I see, I think the encoding problem was fixed in 2.3.0. The type checker is still wrong, though. On Thu, Jan 6, 2011 at 3:17 PM, BigBaaadBob bigbaaad...@gmail.com wrote: 2.2.0a-0.1ubuntu1 On Dec 28 2010, 2:55 pm, Kenton Varda ken...@google.com wrote: Strange. It looks like there is a bug in the type checker for booleans -- it will accept any integer. However, this doesn't explain the encoding error. Looking at the code, I can't see how this could happen; any non-zero integer value should result in 1 being written to the wire. The problem you describe sounds like an obscure problem that happens in C++, but I can't see why it would happen in Python. What version of protobufs are you using? On Thu, Dec 23, 2010 at 10:10 PM, BigBaaadBob bigbaaad...@gmail.com wrote: Turns out it is simple: put a big number into a bool. The encoding side doesn't complain, but the decoder does. A bug on my part, but suboptimal behavior for the library. On Dec 22, 6:09 pm, Kenton Varda ken...@google.com wrote: Can you provide a small, self-contained example program demonstrating this problem? On Wed, Dec 22, 2010 at 4:52 PM, BigBaaadBob bigbaaad...@gmail.com wrote: I have Python code doing something like this: m.whatever = somevalue; etc. s = m.SerializeToString() m.Clear() m.ParseFromString(s) And the m.ParseFromString throws google.protobuf.message.DecodeError: Truncated varint. (And this is the only error thrown! If I print m I don't get any errors.) It seems like protobuf should be able to read something it successfully creates, no? Suggestions? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com protobuf%2bunsubscr...@googlegroups.c om . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Problem with java generation when using imports
Can you provide a small, self-contained example? On Mon, Jan 10, 2011 at 7:13 AM, Meghana meghana.vishwan...@gmail.comwrote: Hi, I am using imports in my proto files. So, B has an import of A. When I generate the java code for B, all references to A are specified as A.A i.e. A(dot)A. It does not seem to be picking up the java_outer_classname from A. Any idea what's going wrong? Thanks. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] LIBPROTOBUF_EXPORT Missing Headers
OK. What is the problem? On Mon, Jan 10, 2011 at 11:46 AM, Collin blackfr...@gmail.com wrote: The following headers have LIBPROTOBUF_EXPORT in them but are installed with `make install.` google/protobuf/stubs/strutil.h google/protobuf/stubs/substitute.h -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Using a ByteBuffer instead of a ByteString?
On Mon, Jan 10, 2011 at 9:45 PM, Nicolae Mihalache xproma...@gmail.comwrote: Hello, I recently started to use GPB, great software! :) But I have noticed in java that it is impossible to create a message containing a bytes fields without copying some buffers around. For example if I have a encoded message of 1MB with a few regular fields and one big bytes field, decoding the message will make a copy of the entire buffer instead of keeping a reference to it. We are actually looking at fixing this by allowing ByteStrings to share buffers. Even worse when encoding: if I read some data from file, does not seem possible to put it directly into a ByteString so I have to make first a byte[], then copy it into the ByteString and when encoding, it makes yet another byte[]. ByteString provides multiple methods of construction. One is to copy from a byte array. Another is to use an OutputStream that writes into a ByteString. In future versions, we are looking at making it possible to concatenate ByteStrings without a copy. But yes, if you start with a byte[], and you want a ByteString with the same content, you are going to need to make a copy, because ByteString has to guarantee immutability. So my question: is it possible to make an exception from the immutability for the bytes fields and use java.nio.ByteBuffers instead of ByteStrings? No, sorry, making any exception to immutability would end up unraveling the whole library. You can go from ByteString to ByteBuffer without a copy (by calling asReadOnlyByteBuffer()), but you can't go the other way, because there is no way to know given a ByteBuffer pointer whether or not someone might be able to modify it in the future. Storing ByteBuffer in message objects directly has additional problems. ByteBuffer is a stateful class -- it maintains a pointer to the current read location, for example. So a protocol message object with ByteBuffers inside it would be thread-hostile no matter how you look at it. This just leads to too many problems... -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: 2.4.0 release candidate #1
On Wed, Jan 5, 2011 at 10:33 PM, Scratch scratch@gmail.com wrote: Thanks! That's right what I needed. And one more question - will sub-builders be introduced for lite package? Sorry, but probably not. The whole point of the lite package is to be as small as possible, and the sub-builders stuff creates a significant amount of code bloat. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
