Re: [protobuf] Parse a .proto file
On 1 November 2014 02:24, Pradeep Gollakota pradeep...@gmail.com wrote: Confirmed... When I replaced the md variable with the compiled Descriptor, it worked. I didn't think I was mixing the descriptors, e.g. the MessagePublish message is one that is produced via the compiled API and parsed using the DynamicMessage API. The isPii extension has been refactored into a separate proto that is precompiled into my codebase. I.E. the descriptor for MessagePublish should be loaded dynamically and the descriptor for the FieldOption I'm defining won't be loaded dynamically. As far as I can tell, there shouldn't be any mixing of the descriptor pools, though I may be wrong. This is exactly where the problem is, though - you have: MessagePublish descriptor D1 (from encoded descriptorset) references extension field F1 (from encoded descriptorset) Message descriptor D2 (from pregenerated code) references extension field F2 (from pregenerated code) So if you have a message built from D1 then it thinks it has a field F1; when you ask if it has extension F2 it says no! even though they're really the same thing. Any thoughts on how I can proceed with this project? It seems like a flaw in the API .. In the case I ran into, I could work around it as the processing code only wanted to work with non-option extensions when it had the precompiled code for the extension, so for those well-known message types I'd just look up the descriptor to use from the precompiled set rather than using the in-stream descriptor set (the message format included the descriptors inline). That doesn't really work here though. Can you inspect the options using field descriptors from the encoded descriptorset, rather than using Messages.pii from the pregenerated code? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Basically, you can't do that in pure Java - the compiler is a C++ binary, there is no Java version. Still, working with the output of --descriptor_set_out is probably the way to go here. If you have the .proto file ahead of time, you can pregenerate the descriptor output at build time and store it instead of the .proto file. If you don't have the .proto file ahead of time (and you can't redesign - this is not a good design) then you could run the compiler at runtime and read the output. Either way, now you have a parsed version of the message format as a protobuf-encoded message that you can read into your Java program and extract the Descriptors you need. If you're looking at a selfdescribing message format, then I'd go with using the parsed descriptors as your format description, not the text .proto file. Oliver On 31 October 2014 17:56, Pradeep Gollakota pradeep...@gmail.com wrote: Hi Oliver, Thanks for the response! I guess my question wasn't quite clear. In my java code I have a string which contains the content of a .proto file. Given this string, how can I create an instance of a Descriptor class so I can do DynamicMessage parsing. Thanks! - Pradeep On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote: On 30 October 2014 02:53, Pradeep Gollakota prade...@gmail.com wrote: I have a use case where I need to parse messages without having the corresponding precompiled classes in Java. So the DynamicMessage seems to be the correct fit, but I'm not sure how I can generate the DescriptorSet from the .proto definition. protoc --descriptor_set_out=FILE ? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
You may be running into issues where the set of descriptors associated with your parsed DynamicMessage (i.e. the ones you parsed at runtime) do not match the set of descriptors from your pregenerated code (which will be using their own descriptor pool). IIRC they're looked up by identity, so even if they have the same structure they won't match if loaded separately. It's a bit of a wart in the API - I'm not sure what the right way to do this is, if you try to mix pregenerated code dynamically loaded descriptors, all sorts of things break. Oliver On 1 November 2014 00:48, Pradeep Gollakota pradeep...@gmail.com wrote: Not really... one of the use cases I'm trying to solve for is an anonymization use case. We will have several app's writing protobuf records and the data will pass through an anonymization layer. The anonymizer inspects the schema's for all incoming data and will transform the pii fields. Since I will be defining the custom options that will be used by the app dev's, I will have precompiled classes available for reference just like the code shows. So what I'm trying to figure out is, using the DynamicMessage API and having parsed a Descriptor, how do I find all the fields which have been annotated with the (isPii = true) option. On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota prade...@gmail.com wrote: Boolean extension = fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
On 30 October 2014 02:53, Pradeep Gollakota pradeep...@gmail.com wrote: I have a use case where I need to parse messages without having the corresponding precompiled classes in Java. So the DynamicMessage seems to be the correct fit, but I'm not sure how I can generate the DescriptorSet from the .proto definition. protoc --descriptor_set_out=FILE ? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] C++ modifying field without knowing field accessor at compile time
On 24 October 2014 17:28, Peter Hubner peter.hub...@gmail.com wrote: All I am trying to achieve is to modify a message on the fly with a set of fields that are supplied to the program at runtime. simple case: // message declared as - const google::protobuf::Message message auto x = reflection-GetDouble(message, field); // do some modification to x reflection-SetDouble(message, field, x); my main question is how in the world do I get from a const ref (needed for GetDouble() ) to a pointer (needed for SetDouble() )??? const_castMessage*(message) This obviously casts away constness which is something to be careful about. If you want to mutate the message, why are you starting with a const ref? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Message extensions cannot have required fields.
Presumably this is because, if your example was allowed, a valid MyMsgA encoding would no longer be a valid MyMsgB encoding. Oliver On 25 October 2014 13:28, Anthony Sampton anthony.samp...@gmail.com wrote: I'm pretty sure it's not only me who faced with this problem, BUMP. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Message extensions cannot have required fields.
On 25 October 2014 14:15, Oliver Jowett oliver.jow...@gmail.com wrote: Presumably this is because, if your example was allowed, a valid MyMsgA encoding would no longer be a valid MyMsgB encoding. Sorry, I mean, would no longer be a valid encoding of MyMsgA as extended by the extension declared within MyMsgB. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Conversion between C structs and google protobufs
On 8 October 2014 06:51, Konstantin Utkin eternalego.erepub...@gmail.com wrote: Ugly. And what if I have multiple nesting, arrays? It leads to tons of macro with huge routine work, and It's simple to convert each C structure manually. But it is so boring! Maybe anyone has found a better way? If the mapping between your C structures and the messages is regular, and you have lots of messages to map, write something to read the message descriptor and generate the glue code for you? If you don't have a regular mapping there's no particularly easy way - you might need to write a little DSL to describe the mapping. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Version message converted from major.minor to gnu_dev_major.gnu_dev_minor
I would guess that you are #including sys/sysmacros.h (perhaps indirectly) which looks like this on my system: /* Access the functions with their traditional names. */ #define major(dev) gnu_dev_major (dev) #define minor(dev) gnu_dev_minor (dev) #define makedev(maj, min) gnu_dev_makedev (maj, min) Oliver On 30 September 2014 16:11, Michael Sumulong msumul...@gmail.com wrote: I have a message where version information is passed and my message fields contain a uint32's for major and minor numbers (as well as patch/pre_release/build info strings along the lines of semver.org). When I use the version message type in my program, the major/minor fields show up as gnu_dev_major/gnu_dev_minor. Is this normal behavior / documented anywhere? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Python and Java: different byte sequences and tag errors
On 23 May 2014 22:06, Alessandro Sivieri alessandro.sivi...@gmail.comwrote: I don't think the communication channel is causing any problem, because the array sequences are not simply truncated but really different (I would expect them to be identical). Check again; it looks like your Java path is mangling (only) bytes 128. Perhaps this path tries to interpret the byte sequence as a string, and charset transcoding mangles your data? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Python and Java: different byte sequences and tag errors
On 24 May 2014 01:23, Oliver Jowett oliver.jow...@gmail.com wrote: On 23 May 2014 22:06, Alessandro Sivieri alessandro.sivi...@gmail.comwrote: I don't think the communication channel is causing any problem, because the array sequences are not simply truncated but really different (I would expect them to be identical). Check again; it looks like your Java path is mangling (only) bytes 128. Perhaps this path tries to interpret the byte sequence as a string, and charset transcoding mangles your data? In fact looking at the exact data this is almost certain to be what's happening. Bytes 128 are being replaced by the three byte sequence 0xef 0xbf 0xbd which is the UTF-8 encoding of U+FFFD - the replacement character often used to replace unrepresentable characters when transcoding. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Use different encoding while serializing strings
The primary protobuf serialization format is the binary format, not the text format. Is there some reason you can't use the binary form? Oliver On 15 May 2014 18:33, Ganesh Sangle saygane...@gmail.com wrote: Let me clarify the question. 1. I did not say that protobuf is affected by \r\n. 2. The problem is this: Take any message with repeated fields, and serialize it to text. The serialized message contains \r\n which are inserted by protobuf, which it is using as some kind of delimiter. That is throwing off my code which is itself using \r\n as a delimiter. So the question is : is there a way to configure protobuf to use something else as a delimiter and not \r\n ? Thanks, Ganesh On Wednesday, May 14, 2014 11:48:37 PM UTC-7, Marc Gravell wrote: protobuf is a binary-safe protocol, and is not impacted by contents such as \r, \n or \t. In particular, text content is utf-8 encoded and length-prefixed - it simply *does not care* what is inside the text. I suspect any problem you are having relates to how you are transporting and processing the payload, not to protobuf itself. Because protobuf does not check for \r, \n or \t at any point. Marc On 15 May 2014 00:56, Ganesh Sangle sayga...@gmail.com wrote: Hi Guys, I am trying to use protobufs between two entities - one in python and another in c++. The advantage of using protobuf is i dont have to write serializing/deserializing code. However, there is a complication. The message that I create in python world, when i serialize it to be sent over to the other side, has '\r\n'. The code on the other side is already using \n as a delimiter and cutting out the strings. So what happens is that the message string gets split and when I try to re-assemble it it doent work. Since my code is a smaller part of a larger existing system, I wanted to know if there is a way to customize/configure protobuf so that it uses some other characters instead of '\n\t', or just use a different representation (all hex numbers or whatever) when converting it to string. Thanks, Ganesh -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com. To post to this group, send email to prot...@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- Regards, Marc -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] string-valued enum needed when implementing OpenRTB2.1 proto file
On 28 March 2014 15:54, Zhiqian Yuan onyourmark...@gmail.com wrote: Those two fields are string type, but like other integer enumerations, they both have a range to set value. So I decide they should be implemented as string-valued enum type. But there's no such a thing either in C++ or in proto. There is no such thing as a string-valued enum type in the protobuf world. You can either have an enum field (that happens to have an integer encoding under the covers, but that doesn't really matter to you); or you can have a string field and do whatever validation you want externally to protobuf. Are you trying to implement an existing message format, or are you trying to define a new message format? If you have control over the format, what's wrong with a standard enum field for these fields? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Python windows adding erronous 0x0D bytes in message and then failing to parse
Don't you want mode wb? Sounds like a classic case of opening a file in text mode and getting LF - CRLF translation happening on Windows. Oliver On 11 December 2013 06:04, Andrew Beck abec...@gmail.com wrote: Using python 2.7.6 on darwin and windows, both using protoc 2.5.0 and libs from 2.5.0, the python code on windows (see test.py in the attached zip) contains two extra bytes valued 0x0D in the message compared to the darwin version. These extra bytes are preventing c++ libprotobuf from parsing the file. Just to be sure it wasn't some null termination causing the parse to fail, I'm using ParsePartialFromArray in the exe. The same code run on darwin using the same version of python (CPython 2.7.6) and protobufs (2.5.0) generates the correct code and can be parsed with the windows and python c++ code. It appears only the python code on windows is producing incompatible messages. Attached is minimal code and setup for reproducing the problem: Entity.proto (protobuf def) main.cpp (c++ parsing code) test.py (python code generating protobuf message) test_Darwin.pb (the output from the python file on darwin) test_Windows.pb (the output from the python file on windows) wscript (simple waf script for building these files) Any help would be greatly appreciated since this is a major issue for my project. Thanks! /andrew -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Issue with SerializeToArray output
On 17 November 2013 22:21, Kiran Kumar kirankumar@gmail.com wrote: char const *rt_msg = (char *)malloc(512);; Rtmesg-SerializeToArray((void *)rt_msg, size); cout rt_msg; This prints absolutely junk. You are trying to print the encoded form of the protobuf message (which is not going to be human readable), and you're also treating it as a 0-delimited C-style string (which it is not - it may contain embedded 0 bytes) str2 = amqp_cstring_bytes(rt_msg); - This just crashes. You're not passing in a length, so I don't know how that function is ever going to work (crash or no crash) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Where is the (meta) .proto file which describes .desc files?
On Thu, Sep 5, 2013 at 1:52 PM, Chris Dew cms...@gmail.com wrote: Where is the (meta) .proto file which describes .desc files? I make .desc files with: protoc --descriptor_set_out=foo.desc --include_imports foo.proto. https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.proto Look for FileDescriptorSet. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] deserializing unknown protobuf message?
On Mon, Aug 19, 2013 at 8:37 PM, Tom Ward tw...@thefoundry.co.uk wrote: Basically I'm trying to work out how to deserialize a protobuf message without using the generated headers, as we're likely to get messages that weren't generated at compile time. I've looked through the documentation, but I only seem to be able to find ones that use generated classes to deserialize, or that use a Descriptor from a generated class to create a DynamicMessage, which I can't seem to work out how to do when we don't have the proto. The protobuf encoding isn't self-describing; generally you need either a compiled message definition or a Descriptor to make sense of an encoded message. Without one of those, there is not much you can do beyond splitting the message into opaque fields that you will have difficulty in interpreting further. As Descriptors can be turned into protobuf messages, one approach is to include a serialized Descriptor as part of the standard message framing; then you can use DynamicMessage on the receiver side. See https://developers.google.com/protocol-buffers/docs/techniques#self-description Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] ByteString using N bytes from an InputStream?
On Tue, Aug 6, 2013 at 7:18 PM, Feng Xiao xiaof...@google.com wrote: On Mon, Aug 5, 2013 at 9:31 PM, V.B. vidalborro...@gmail.com wrote: ... Actually, I just now took a closer look at the readChunk() method. Even that method makes an internal copy, so it looks like readChunk() isn't what we are looking for after all. Hmmm. It seems to me that readChunk() has done a redundant copy which can be eliminated (and should be). Does ByteString need to worry about hostile InputStreams? The stream could retain a reference to the bytearray it was given in the read() call. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] InvalidProtocolBufferException
On Wed, Jul 24, 2013 at 8:22 AM, Lotte R. Leben lo...@zombietetris.dewrote: *But* As far as I know readDelimited() only works with inputStreams, which kind of contradicts the whole UDP thing. Since adding the delimiter was a tad tricky, I wasn't confident that the following would work: - add delimiter in python as mentioned above - receive datagram in java - read first bit to know length = n - read next n bytes - PacketMessage.parseFrom(chunkOfnBytes) Should be fine. May be simplest to wrap your bytearray in a CodedInputStream directly via CodedInputStream.newInstance(). But see below - if the contents of the UDP datagram is entirely message data you shouldn't need a separate length at all. Or if there is no other data in the UDP datagram you could just use the datagram length directly (i.e. get rid of the trailing garbage - why are you padding the datagram in the first place?) We are not padding it on purpose and I'd love to know how or why that garbage data got in there. :( If you are using DatagramSocket.receive(DatagramPacket) this probably means you are not looking at packet.getLength() on return. The datagram received is read into the array owned by DatagramPacket, but it may not fill the entire array so you need to inspect and use the actual length returned rather than using the whole array. You want something approximately like this: DatagramPacket p = new DatagramPacket(new byte[MAX_DATAGRAM_SIZE], MAX_DATAGRAM_SIZE); socket.receive(p); PacketMessage message = PacketMessage.parseFrom(CodedInputStream.newInstance(p.getData(), p.getOffset(), p.getLength())); If that's not the problem, then I would look at the datagram on the wire (via e.g. wireshark) to see if the garbage is present there - that'd narrow it down to sender or receiver at least. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] InvalidProtocolBufferException
On Wed, Jul 24, 2013 at 12:27 AM, Andreas Reuter andreas.rou...@googlemail.com wrote: Hello, we are using Protobuf to communicate between two devices via Ad-hoc WiFi. One of the devices is running a Java application and the other a Python application. [...] 6. Upon closer inspection, we saw that the parser seems to be doing alright, successfully assembling the packet message. However the message is only a few bytes long, and after that the parser just keeps reading the byte array(buffer, which is 128 byte long). The byte array is filled with 0's. Upon reading a 0 the exception occurs. Protobuf messages are not self-delimiting; you must provide some other mechanism yourself to find the end of the message. The common way is to prefix the message with a length field - see https://developers.google.com/protocol-buffers/docs/techniques#streaming Or if there is no other data in the UDP datagram you could just use the datagram length directly (i.e. get rid of the trailing garbage - why are you padding the datagram in the first place?) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Performance aspect of submessage serialization
On Wed, May 15, 2013 at 11:16 AM, mailto.jo...@gmail.com wrote: I think it is more of a bug in the protoc implementation that it fail's to parse such a message. Do you have some test code to share? (I got as far as eyeballing the C++ implementation and it looks OK at a first glance) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Performance aspect of submessage serialization
On Wed, May 15, 2013 at 12:02 PM, Oliver Jowett oliver.jow...@gmail.comwrote:
On Wed, May 15, 2013 at 11:16 AM, mailto.jo...@gmail.com wrote:
I think it is more of a bug in the protoc implementation that it fail's to
parse such a message.
Do you have some test code to share?
(I got as far as eyeballing the C++ implementation and it looks OK at a
first glance)
In fact after a bit more work I can't reproduce your problem at all. What
are you doing differently to what I'm doing here?
Oliver
oliver@shiny:~$ protoc --version
libprotoc 2.4.1
oliver@shiny:~$ cat varint.proto
message VarintOuter {
required VarintInner inner = 1;
}
message VarintInner {
required int64 value = 1;
}
# Normal encoding:
oliver@shiny:~$ hexdump -C varint.bin
0a 02 08 64 |...d|
0004
oliver@shiny:~$ protoc varint.proto --decode=VarintOuter varint.bin
inner {
value: 100
}
# overly long encoding:
oliver@shiny:~$ hexdump -C varint-long.bin
0a 82 00 08 64|d|
0005
oliver@shiny:~$ protoc varint.proto --decode=VarintOuter varint-long.bin
inner {
value: 100
}
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Have issues on the design of protocol buffers...
On Thu, Apr 25, 2013 at 2:13 AM, Christian Gross christianhgr...@gmail.comwrote: So I am trying to wrap my head around protocol buffers and used it for a few tests, however, I found it problematic in my case for the following reasons: 1) Creating a multi message end point by mocking unions using optional types. In most serialization schemes I have the problem that when I serialize a message it requires a type. Some serialization implementations allow me to parse part of the message to determine what to do with the rest. With protocol buffers I have to use optional fields and reference to indicate which message or messages it is. Add on multi-messages and this becomes really complicated. The scheme I developed used MIME encoding and it seemed simpler as I could put multiple pieces together and then encode them. You are not forced to use unions - you can use external information (a separately encoded type value, or even a complete header message if you wanted) to determine the type to use. Constructing a map of types to default instances on the decode side is a useful trick here. The point of using the union-message approach is that protobuf is already quite good at going read type value, decode contents with the appropriate type - that's exactly what it does to decode fields - so having a separately encoded type can be redundant. 2) immutable objects. I get the theory, and know functional programming, but this one is a bit rough to digest. Namely, lets say I have a really big message due to the use of optional fields, and need to change one field it appears to me that there is going to be quite the performance hit. If you want to incrementally build the message, pass around the builder instance not the final immutable message. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] [java] serialize to the static buffer
https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/CodedOutputStream.html#newInstance(byte[], int, int) On Tue, Apr 23, 2013 at 7:12 PM, adam.h adam.p.haj...@gmail.com wrote: Hi Protobufs Apostles, There is nice function to deserialize from the buffer - mergeFrom(buf, offs, size), but how can you serialize similar way? I do not see anything like writeTo(buf, offs, size). I would like to avoid allocating bytes[] each time when serializing message - so what is proper way to achieve it? Thanks, Adam -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] accessor method for message type
On Sat, Apr 20, 2013 at 4:38 PM, Oliver Wang python...@gmail.com wrote: So the from generated code, I would expect an access method such as set_foo() set_bar and set_baz() so that I can connect Foo object to the message. But I don't see it at all, there are methods such as has_foo() has_bar() and has_baz() etc. What language are you using? For C++, see https://developers.google.com/protocol-buffers/docs/reference/cpp-generated, Singular Embedded Message Fields Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: how to calculate protocol buffer message size without parsing the full message?
On Sun, Apr 21, 2013 at 9:48 PM, Oliver Wang python...@gmail.com wrote: Can't you use ByteSize() method to get the size? Not when you're parsing - you don't have a message instance yet! Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] PB Buffer Error Can't Parse Message with 3 String Fields in Sequence
On Tue, Apr 16, 2013 at 4:43 PM, brsckap ped...@brsc.com wrote: std::ifstream inputStream; inputStream.open(pbInput); Text-mode open is corrupting your encoded message, probably. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] protobuf stringstream codedoutputstream issues
On Sun, Apr 14, 2013 at 7:26 PM, Predator33 predator...@gmail.com wrote: The only thing I could think of is maybe the protobuf stream doesn't flush itself out until the destructor is hit and since that never happens until after the fact, it never gets flushed? Yes, this will be what is happening. Arrange for the OstreamOutputStream to be destroyed before you try to use its output (e.g. wrap that declaration in another level of nesting) Is there some reason why you're using stringstream + OstreamOutputStream, rather than just using StringOutputStream? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] protoc - how to generate code for proto and its imports
On Fri, Apr 12, 2013 at 7:59 PM, Leonid G leonid.ge...@gmail.com wrote: I am actually working on a build system and in my case all file names are an absolute path, not relative. With absolute path method suggested above (without -I) fails with $ google/protobuf/2.3.0/gcc_4.1.2/64/bin/protoc --cpp_out=/mypath/test /mypath/test/importing.proto /mypath/test/imported.proto /mypath/test/importing.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think). Looks like in general case I do need to set -I to some value. The only remaining thing - understand what the correct value for -I. In your example, it would be -I /mypath Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] protoc - how to generate code for proto and its imports
On Fri, Apr 12, 2013 at 8:43 PM, Leonid G leonid.ge...@gmail.com wrote: Thanks, Oliver. problem is that build system is such that those protos are necessarily under common root. Not quite sure what you mean. I guess I'll figure it out. Well, you can perhaps think of it this way: you have one or more non-overlapping filesystem trees of protobuf files; you must specify the roots of those trees via -I arguments; imported files are searched for relative to the root of each tree; all command-line input files must lie within one of the trees. If you already have suitable subdirectory layout(s) containing your protobuf files, it's probably just a case of telling your build system where the roots of those trees are (which will turn into -I arguments), then consistently using absolute paths everywhere on the command line. (Personally I'd avoid placing your output anywhere near your input while you're at it ..) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] DecodeError (Truncated Message)
On Thu, Mar 28, 2013 at 12:54 AM, W. David Jarvis venant...@gmail.comwrote: If I compare the len() of the two strings (pre-writing/reading v. post-writing/reading) I tend to get two add'l chars on the reading, so I guess I'm confused by how this is getting truncated? Is python running into a newline char and interpreting it literally somehow? Don't use 'print', use file.write(). Using print adds a newline (or probably CRLF in your case). The encoded form is not self-delimiting so if you include an extra CRLF in the bytes given to the decoder, it'll happily try to interpret them, and then run out of data because the decoder's interpretation of the extra bytes implies there should be more data following them. I wouldn't rely on sys.stdout being binary-safe either. The encoded forms are binary, not text - you should treat them as such - if any conversion of linefeeds happens, it will corrupt the message. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Awkward error. How can this be?
On Fri, Mar 22, 2013 at 8:45 AM, kramer65 kram...@gmail.com wrote: I've been tinkering about this all night, but with a fresh view in the morning I still can' t understand what is wrong here. The line where the error originates is this: beat.ParseFromString(message) Are you sure your callback is being given a string? From a quick look at http://zeromq.github.com/pyzmq/api/generated/zmq.eventloop.zmqstream.html, it says callback must take exactly one argument, which will be a list, as returned by socket.recv_multipart() - i.e. not just a simple string. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Why to reinvent the wheel ?
On Thu, Mar 21, 2013 at 8:48 AM, Vic Devin vfn...@gmail.com wrote: What I mean is that they all try to solve, leaving aside all tech details, the same basic problem, i.e. remote communication between software entities. There are plenty of applications of protobuf (and ASN.1 for that matter) that do not involve remote communication at all. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Question about internationalization and localization
On Fri, Mar 15, 2013 at 11:59 PM, Feng Xiao xiaof...@google.com wrote: On Fri, Mar 15, 2013 at 8:23 AM, Joann Kent joj...@gmail.com wrote: Hi all, Within the code generated by GPB (I'm using Java btw) there are exceptions thrown and the messages are hard-coded and in English. Are there any plans to allow for internationalization/localization of these error messages? Not that I have heard of. I'm curious, do you know any project where exception messages are internationalized? The Postgresql JDBC driver does this. Typical code looks like this: throw new PSQLException(GT.tr(Returning autogenerated keys is only supported for 8.2 and later servers.), PSQLState.NOT_IMPLEMENTED); where GT.tr() is a static method that does java.util.ResourceBundle lookup at runtime, and acts as a gettext marker at build time. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Issue 269 in protobuf: Would like to have byte or int8 type for the message definition
On Thu, Mar 14, 2013 at 2:37 AM, Sumit Kumar kumar.su...@hotmail.comwrote: No you cant, on C++ bytes translates to string, wont allow for null. std::string handles NULs just fine, and protobuf's use of it for bytes fields also works fine. (I have code that's been in production for a few years that relies on this) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Linker error
On Tue, Mar 5, 2013 at 2:25 AM, Mohammad Husain farhan1...@hotmail.comwrote: g++ `pkg-config --cflags --libs protobuf` -o hello_protobuf hello_protobuf.o user.pb.o Probably just argument ordering. The library reference (which will come from pkg-config --libs) needs to appear after the object files that refer to symbols in the library. In particular the g++ manpage says It makes a difference where in the command you write [the -l option]; the linker searches and processes libraries and object files in the order they are specified. Thus, foo.o -lz bar.o searches library z after file foo.o but before bar.o. If bar.o refers to functions in z, those functions may not be loaded. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Again about delimiting multiple messages in streams
On Sat, Feb 2, 2013 at 12:02 PM, Roman roman.sima...@gmail.com wrote: Hello! I read a lot about delimiting messages by VarInt32 number before each serialized message. It requires some additional coding. In the same time I guess that parsing code of GPB must understand when it must stop deserialization. Maybe someone who is close to sources can help me. I assume two conditions for that: 1) End of stream 2) End of message 3) Other? Option 1) we do not consider since our stream must receive multiple messages like socket stream. Well, unfortunately, it is effectively option (1) that is actually used. There is nothing in the encoding itself that tells you where the message ends. This is why you need a separate message delimiting mechanism. Option 2) well can be detected. GPB encode data in order of TAG numbers. So if we add the last field in message (say) required bool eom = MAX_POSSIBLE_TAG_NUMBER a message parser must stop parsing after reading this field since no fields more expected. Fields don't have to be encoded in any particular order, so you can have a valid message encoding where the encoding of 'eom' is followed by other fields. Opinions? What you are describing is not how the current encoding works. I'm not sure whether you are asking whether this would work with the current encoding (it won't) or whether you are suggesting changes to the encoding (which I suspect will be a very uphill battle - why change what already works and break all the current users of the code? Remember there are multiple implementations of the protocol, too..) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Compress proto to string
On Tue, Jan 29, 2013 at 11:23 AM, David Granados deivi...@gmail.com wrote:
this does not work :-(
That's not really a useful description of the problem.
229 google::protobuf::io::GzipOutputStream
gzip_stream(arrayStream);
230 if (!pProto.SerializeToZeroCopyStream(gzip_stream))
231 {
232 std::cerr Error comprimiendo el proto a string.
std::endl;
233 }
You should call gzip_stream.Flush() or gzip_stream.Close() after this,
before using the results in arrayStream.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Compress proto to string
(Readding the list cc:)
On Tue, Jan 29, 2013 at 4:32 PM, David Granados deivi...@gmail.com wrote:
Does not work :-(
Size of proto before: 196, size of proto after gzip: 196
What measures this? Nothing in any code you quoted.
Now my code are:
206 template class T int
207 ProtoT::CompressProtoToString(std::string pOut,T pProto)
You never touch pOut.
208 {
209 // Resultado
210 int l_resultado = -1;
211
212 // Definimos el buffer al cual vamos a comprimir
213 const unsigned bufLength = 256;
214 unsigned char buffer[bufLength];
215
216 memset(buffer,0,256);
217
218
219 std::cout buffer std::endl;
220
221 // Obtenemos un array del proto serializado
222 ZeroCopyOutputStream* output = new
google::protobuf::io::ArrayOutputStream(buffer,bufLength);
223 // Introducimos las opciones de compression
224 GzipOutputStream::Options options;
225 options.format = GzipOutputStream::GZIP;
226 options.compression_level = NIVEL_COMPRESION;
What is the value of NIVEL_COMPRESION?
227
228 google::protobuf::io::GzipOutputStream
gzip_stream(output,options);
229 if (!pProto.SerializeToZeroCopyStream(gzip_stream))
230 {
231 std::cerr Error comprimiendo el proto a string.
std::endl;
232 }
233 gzip_stream.Close();
234 std::cout : buffer std::endl;
This will fail if any 0-bytes happen to get serialized (you're
treating buffer as a nul-terminated C-style string)
235
236 // Devolvemos el resultado
237 return l_resultado;
238 }
I suggest that you post a *complete*, compilable, testcase - it's just
too hard to diagnose what's going wrong by guessing what you are doing
in the surrounding code that you didn't provide.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] GzipOutputStream does not work in ubuntu 12.10
On Fri, Jan 18, 2013 at 9:58 PM, David Granados deivi...@gmail.com wrote: GZipOutputStream gzip_stream(file_stream, GzipOutputStream::Options()); main.cpp:11:4: error: ‘GZipOutputStream’ no se declaró en este ámbito You have the wrong capitalization (GZip vs Gzip) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How do I check if a message is complete?
On Sun, Jan 13, 2013 at 12:04 PM, Klaim - Joël Lamotte mjkl...@gmail.com wrote: On Sun, Jan 13, 2013 at 9:54 AM, Feng Xiao xiaof...@google.com wrote: With protobuf reflection you can easily write a function to check if all fields of a message are set. I see, thanks for pointing that, it's a good alternative. But wouldn't it be far more efficient if it was generated by protobuf in the generated types? At the moment I avoid using reflection where I can because of performance concerns. Maybe I should make a ticket to request a function. This seems like a function that would be only useful for your specific requirement. It would be no use, for example, in any situation where backwards message compatibility is needed. If reflection is too expensive (have you tried it?) then you might want to consider writing a code generator plugin to generate the code that you want (see https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin.pb) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Change wire type
On Thu, Dec 13, 2012 at 12:31 AM, Lilithfr alexandre.lam...@gmail.com wrote: Hello, I'm looking to change the type of a field from uint64 to fixed64 to reduce the encoding time but I want to keep the same id and some old application will still use the old definition. For my application, there is no different between uint64 and fixed64, they both encoded as uint64_t in C++, but the wire type change from 0 (Varint) to 1 (64Bit). Is the protobuf library flexible enough to handle this case ? See https://developers.google.com/protocol-buffers/docs/proto#updating (Short answer is no) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] CodedInputStream bufferSize is set incorrectly when reading from a byte[] slice
On Tue, Oct 9, 2012 at 8:08 AM, Jack Punt thegr...@gmail.com wrote: For example: CodedInputStream.newInstance(A.getBytes(), 5,5).isAtEnd() -- false [when obviously the buffer is 'empty' at this point] and readBytes() will likewise expect to find more bytes than are actually in the byte[] The convention elsewhere is generally that the second (length) parameter is a count, not an end index - so your example claims that bytes 5-9 (inclusive) should be used (and you should be providing an array of length = 10). See e.g. the ByteArrayInputStream ctor. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How to convert protobuf with same structure in java
On Tue, Jul 24, 2012 at 3:55 PM, Tobias Neef tob...@gmail.com wrote:
I try to convert the Java representation of a protobuf message into another
message with the same fields (ordering, names, types). I try to do it the
following way but this ends in a java.lang.InstantiationException:
public static T extends GeneratedMessage T commandToEvent(GeneratedMessage
command,
ClassT clazz) {
try {
Builder builder = clazz.newInstance().newBuilderForType();
Generated message classes don't have a public ctor, so newInstance() will fail.
Pass the default instance of the message class, not the class itself,
to identify the message type to build?
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Private constructor in ByteString
On Thu, Jun 21, 2012 at 7:08 AM, Anatoly Deyneka adeyn...@gmail.com wrote: Why ByteString has only private constructor? Copying arrays is overhead in my case. I believe it's that way to guarantee the immutability of a ByteString. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Is there any possible way bind Protocol Buffer with Hibernate,MyBatis Or JDBC
On Mon, Jun 18, 2012 at 3:07 AM, Jessen Gan jessen...@gmail.com wrote: Recently,I used Protocol Buffer as a middleware to trans data between server and mobile_client.Read data from database by hibernate into java entities,then add into Protocol Buffer Object property one by one.It's a very heavy method. But I googled many key worlds to find a shortcut to do the this.the result is null. If you have a set of rules that map your Java fields/methods to protobuf field names, then you could build the message reflectively via DynamicMessage.Builder, perhaps? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] incompatible type changes philosophy
On Wed, May 9, 2012 at 12:42 AM, Jeremy Stribling st...@nicira.com wrote: I'm wondering if anyone has experience with a scenario like this, and if there's a more elegant way to solve it. We do something a bit similar by advertising capabilities during a handshake at the start of each connection. If we need an incompatible changes in a message, we retain both forms as separate fields in the protobuf definition and add a new capability that says I understand the new form as well as the old form. The sender ensures the right field is set based on the recipient's advertised capabilities and which forms the sender understands. This only works if you can do a handshake, though - it wouldn't be any good for persistent storage or multicast/datagram-like situations. (In practice, it's been rare to actually do incompatible changes - more commonly we use the capabilities to negotiate behavioural changes) Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] proto files - user defined java objects as messages fileds
On Thu, Apr 19, 2012 at 11:36 AM, kandji adram...@googlemail.com wrote: Can i use a user-defined java class as a field type in a message declared in a .proto file? 2- I would like to serialize an instance of the class MyComplexClass.java. As you can see, I would use MyComplexClass as a field type in the message descritption. Is that possible? Or can I only use scalar/ primitive types that are built-in in java? No, you can't do that directly. You have a few options: 1) make MyComplexClass serializable, make the field type 'bytes', and stash the serialized form of the instance as a bytearray. This defeats the purpose of using protobuf, though - you lose interoperability with other languages, you are relying on Java's serialization rules for compatibility, and you have a speed penalty from using Java serialization. 2) create a separate protobuf message type for MyComplexClass, and so on for its fields until you reach primitive types that can be directly represented; then build some glue code to convert between the generated message classes and MyComplexClass. 3) if MyComplexClass is really just a data-holding class, you might want to consider using the message classes generated by step (2) directly, replacing MyComplexClass entirely. If you have many classes to apply (2) to, and they have a regular structure, then you could write a tool to automate the process of generating message definitions and glue code from classfiles. Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] using GzipOutputStream increases size
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: What's the point of Protocol Buffers?
Christopher Smith wrote: There is also the other end of the spectrum, where I am at. With a large Hadoop cluster and terabytes of data, the efficient storage and zippy parsing of protobufs is a huge deal. In many ways, protobufs allow you do do what XML promised, but much more efficiently. The other way to look at them is they are ASN.1 reduced to the simplest usefull feature set. Amusingly enough, we use protocol buffers to transport ASN.1-encoded data (SS7 TCAP messages). The protobuf API is far better than the API produced by the commercial ASN.1 compiler we use. -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] problem: protobuf (cpp) not serializing all fields to array?
phelyks wrote: if (!unp.ParseFromArray(buffer.c_array(), 0)) You are claiming that the array to parse from is zero bytes long. Unsurprisingly, trying to parse a zero-byte message doesn't give anything useful. Perhaps you meant to pass usedBufLen? -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: How to statically link protoc on linux?
Kenton Varda wrote: LDFLAGS=-static isn't good enough? Nope, something along the way eats it. -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Any sample messages/data for optimization out there?
Oliver Jowett wrote:
(For
example, I wonder if the Java code could benefit from the
switch-prediction tricks that the C++ code does?)
FWIW, I hacked something together to test this, producing code that
looks like this:
while (true) {
int tag = input.readTag();
switch (tag) {
case 10: {
setField1(input.readString());
if (!input.expectTag(16)) break;
}
case 16: {
setField2(input.readInt32());
if (!input.expectTag(24)) break;
}
case 24: {
setField3(input.readInt32());
if (!input.expectTag(34)) break;
}
where CodedInputStream.expectTag() behaves like the C++ equivalent.
But it made 1% difference according to ProtoBench, down in the
measurement noise - it was actually slower in some cases.
There is also something strange going on with stream decoding. This is
from a plain svn build without the above changes:
Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file
google_message1.dat
Deserialize from byte string: 17471413 iterations in 30.074s; 126.3199MB/s
Deserialize from byte array: 17389320 iterations in 30.009s; 125.99868MB/s
Deserialize from memory stream: 5050944 iterations in 29.372s; 37.391594MB/s
Benchmarking benchmarks.GoogleSpeed$SpeedMessage2 with file
google_message2.dat
Deserialize from byte string: 40052 iterations in 29.988s; 107.7192MB/s
Deserialize from byte array: 39587 iterations in 29.678s; 107.5807MB/s
Deserialize from memory stream: 40117 iterations in 29.9s; 108.21157MB/s
I wonder what's happening in the first memory stream case to make it 3x
slower than everything else?
-O
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] intern strings in java
2010/1/20 Kenton Varda ken...@google.com: Are you suggesting that the parser should automatically intern every String that it parses? That sounds like it could be expensive for those who don't need it. More than expensive - it's a permanent memory leak for any app that passes around many unique strings. It'd be a showstopper for our app. -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Problem in installing protobuf-2.2.0a on Solaris 10
mk wrote: Hi Could someone please help me to install protobuf-2.2.0a on Sun Soalris machine. I tried even to install it to /usr, but I am still getting these same errors: ld: fatal: file .libs/common.o: wrong ELF class: ELFCLASS64 ld: fatal: File processing errors. No output written to .libs/ libprotobuf.so.5.0.0 collect2: ld returned 1 exit status This looks like a configure-script bug I hit earlier that results in trying to mix 32- and 64-bit objects when linking. To avoid this, you need to explicitly pass -m32 or -m64 in *both* CFLAGS and CXXFLAGS. I configure my solaris builds like this: 32-bit: ./configure --prefix=/usr/local/protobuf/2.2.0 \ --without-zlib \ --disable-64bit-solaris \ CXXFLAGS=-m32 -g -DNDEBUG -fno-exceptions \ PROTOBUF_OPT_FLAG=-O2 \ CFLAGS=-m32 64-bit: ./configure --prefix=/usr/local/protobuf/2.2.0 \ --without-zlib \ CXXFLAGS=-m64 -g -DNDEBUG -fno-exceptions \ PROTOBUF_OPT_FLAG=-O2 \ CFLAGS=-m64 -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?
Nigel Pickard wrote: Here's my sample code. In this code, everything works for me -however when I change the GPBClient to use Data IO streams (just comment out the Object IO references and uncomment the Data IO references in GPBClient), that's when I get the com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. error. ObjectOutputStream doesn't just write raw bytes directly to the underlying stream, it also adds some additional header and framing data. So in general you can't write data with an ObjectOutputStream and read it back with a plain DataInputStream and expect things to work. There's really no reason to use ObjectOutputStream unless you are using Java serialization. -O -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: ActiveMQ implementation of protobuf
On Tue, Sep 22, 2009 at 5:57 AM, Kenton Varda ken...@google.com wrote: So Java users don't like relying on C++ code in their build process. On the other hand, most C++ users would not accept Java in their build process either. But if we wrote the C++ code generator in C++ and the Java code generator in Java, then we'd need to write and maintain a .proto parser in both languages as well. A correct parser that supports all features (e.g. custom options) is a large, complicated piece of code, and maintaining two separate implementations of it would be a huge pain. This seems to leave no options that please everyone. :/ We have a similar issue here with an Ivy-based environment, but it's not really a problem. We just put the path to protoc in a user-specific properties file (i.e. $HOME/.build.properties) and require that our build machines are set up appropriately if you want to build modules that use protoc. Since we already have to do that for other tools anyway, it's no big deal. Installing protoc just becomes another step in the list of prerequisites when preparing a build environment, much like we have to install appropriate versions of other tools needed to build (a JRE, Ant, Ivy, etc) -O --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: protoc feature question
Henner Zeller wrote: And if you do it from an ant-buildfile (often used in java projects) then it would be as well hard as ant has not really a notion of file-dependencies (yeah, ant sucks as build-tool). Perhaps you are looking for uptodate or dependset -O --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: 2.2.0 make check failures on Solaris 10 sparc32 (problems with zlib?)
Kenton Varda wrote:
Run configure --without-zlib to disable zlib support. Alternatively,
fix it and send me a patch. :)
I took a look at fixing it in gzip_stream.{cc,h} (conditionally removing
GZIP format so you'd get compile errors if you tried to use that on a
system with an old zlib), but it was going to be awkward without
breaking backwards compatibility because the default format if not
specified is GZIP.
So here's a patch that just detects old zlib versions and disables zlib
support entirely in that case.
I have approximately zero experience in writing autoconf files, so this
may be completely the wrong approach! I did test it on a couple of
systems (one with zlib 1.2.3.3, one with zlib 1.1.4) and it appeared to
do the right thing.
-O
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Index: configure.ac
===
--- configure.ac(revision 221)
+++ configure.ac(working copy)
@@ -82,14 +82,24 @@
AC_FUNC_STRTOD
AC_CHECK_FUNCS([ftruncate memset mkdir strchr strerror strtol])
+# Checks for a sufficiently recent zlib
HAVE_ZLIB=0
AS_IF([test $with_zlib != no],
[AC_SEARCH_LIBS([zlibVersion], [z],
-[AC_DEFINE([HAVE_ZLIB], [1], [Enable classes using zlib compression.])
- HAVE_ZLIB=1],
-[if test $with_zlib != check; then
- AC_MSG_FAILURE([--with-zlib was given, but test for zlib failed])
- fi])])
+[AC_MSG_CHECKING([the zlib version])
+ AC_COMPILE_IFELSE(
+ AC_LANG_PROGRAM([[
+#include zlib.h
+#if !defined(ZLIB_VERNUM) || (ZLIB_VERNUM 0x1204)
+# error zlib version too old
+#endif
+]], []),
+ [AC_MSG_RESULT([ok (1.2.0.4 or later)])
+ AC_DEFINE([HAVE_ZLIB], [1], [Enable classes using zlib compression.])
+ HAVE_ZLIB=1],
+ [AC_MSG_RESULT([too old (1.2.0.3 or earlier)])])])])
+AS_IF([test $with_zlib != check test $HAVE_ZLIB = 0],
+ [AC_MSG_FAILURE([--with-zlib was given, but test for zlib failed])])
AM_CONDITIONAL([HAVE_ZLIB], [test $HAVE_ZLIB = 1])
AS_IF([test $with_protoc != no], [
2.2.0 make check failures on Solaris 10 sparc32 (problems with zlib?)
Environment: protobuf 2.2.0 $ uname -a SunOS arson 5.10 Generic_118833-23 sun4u sparc SUNW,Sun-Fire-280R $ g++ -v Reading specs from /usr/local/lib/gcc/sparc-sun-solaris2.10/3.4.6/specs Configured with: ../configure --with-as=/usr/ccs/bin/as --with-ld=/usr/ccs/bin/ld --enable-shared --enable-languages=c,c++,f77 Thread model: posix gcc version 3.4.6 Configured with: $ ./configure --prefix=/usr/local/protobuf/2.2.0 --disable-64bit-solaris The tests start failing here: [--] 14 tests from IoTest [ RUN ] IoTest.ArrayIo [ OK ] IoTest.ArrayIo (4 ms) [ RUN ] IoTest.GzipIo google/protobuf/io/zero_copy_stream_unittest.cc:194: Failure Value of: WriteToOutput(output, str.c_str(), str.size()) Actual: false Expected: true google/protobuf/io/zero_copy_stream_unittest.cc:194: Failure Value of: WriteToOutput(output, str.c_str(), str.size()) Actual: false Expected: true google/protobuf/io/zero_copy_stream_unittest.cc:194: Failure Value of: WriteToOutput(output, str.c_str(), str.size()) Actual: false Expected: true and it all goes downhill from there. zlib is the system-installed /usr/lib/libz.so, which appears to be zlib 1.1.4. I've attached the (gzipped, since it's not short) output of make check. Any ideas? -O --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~--- make-check.log.gz Description: GNU Zip compressed data
Re: Notes on building a 64-bit install on Solaris 10 x86
Kenton Varda wrote: I'm pretty sure #1 is not protobuf's fault. I've never heard of libgcc_s which means it's probably something GCC links against implicitly. And anyway, if libwhatever.so.1 exists, there should always be a libwhatever.so symlink to it... right? I *think* so, but calling g++ directly to link things works fine with nothing special. libtool is implicated somewhere here since it likes to pull all the implicit objects out and specify them itself. -O --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
optimize_for = LITE_RUNTIME and DebugString() / ShortDebugString()
While experimenting with LITE_RUNTIME I noticed that the C++ methods
DebugString() and ShortDebugString() are only present on Message, not
MessageLite.
I understand that DebugString()/ShortDebugString() are implemented via
reflection, which isn't available in the lite runtime, but what about a
simple implementation that returns a less detailed string? It seems
reasonable that you'd get less useful debug output when using the lite
runtime. Something like this perhaps:
string MessageLite::ShortDebugString() const {
string result;
result += (lite message type ;
result += GetTypeName();
result += );
return result;
}
That would mean you wouldn't need to do so much conditional compilation
for code that can be built against either the full or lite library. I
have a fair amount of code of the form:
DEBUG(CONNECTION, Send: %s, message.ShortDebugString().c_str());
and wrapping it all in #ifdefs is fairly awkward.
-O
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: missing parseFrom(...) variants in generate Java code; and how to build your own parseFrom() equivalent
Kenton Varda wrote:
On Tue, Aug 18, 2009 at 6:55 PM, Oliver Jowett oliver.jow...@gmail.com
mailto:oliver.jow...@gmail.com wrote:
Hi
Using 2.2.0, the compiler doesn't seem to be generating Java code for
the parseFrom(byte[],int,int) and
parseFrom(byte[],int,int,ExtensionRegistryLite) static methods.
Did it ever generate those methods? I don't remember if it did, but I
definitely don't remember removing them.
I don't think so - I just tried to use them based on the documentation
saying there's a parseFrom() variant for each mergeFrom() and ran into
compile errors. I really did need to parse from a partial array there -
it was a section of a NIO buffer's backing array.
Arguably your code should look like:
message = builder.mergeFrom(array, offset, length).buildPartial();
if (!message.isInitialized()) {
throw new InvalidProtocolBufferException(...);
}
It's a little bit nicer than the try/catch...
Unfortunately, this doesn't give you the nice list of missing fields
that parseFrom() generates.
-O
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Building without RTTI support causes make check to hang (recursive call to pthread_once?)
I'm trying to build 2.2.0 without RTTI support and running into some obscure problems. The environment is: $ g++ -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.3.3-5ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) Results so far: The default configuration (just ./configure) passes make check. With CXXFLAGS=-g -DNDEBUG -DGTEST_HAS_RTTI=0 -DGOOGLE_PROTOBUF_NO_RTTI -fno-rtti PROTOBUF_OPT_FLAG=-O2: make check hangs on the CustomOptions.OptionLocations test. The stack trace looks like this: (gdb) where #0 0x2b85e53005cb in pthread_once () from /lib/libpthread.so.0 #1 0x00426431 in google::protobuf::GoogleOnceInit (once=0xa7b148, init_func=0x726bfd protobuf_unittest::protobuf_AssignDesc_google_2fprotobuf_2funittest_5fcustom_5foptions_2eproto()) at ./google/protobuf/stubs/once.h:115 #2 0x0071c3c0 in protobuf_AssignDescriptorsOnce () at google/protobuf/unittest_custom_options.pb.cc:342 #3 0x0071c4c9 in protobuf_unittest::ComplexOpt6::GetMetadata (this=0x262e320) at google/protobuf/unittest_custom_options.pb.cc:3796 #4 0x2b85e3ffb67a in google::protobuf::internal::ReflectionOps::Merge (fr...@0x262e320, to=0x268f0e0) at ./google/protobuf/message.h:311 #5 0x0071dc4a in protobuf_unittest::ComplexOpt6::MergeFrom (this=0x268f0e0, fr...@0x262e320) at google/protobuf/unittest_custom_options.pb.cc:3753 #6 0x2b85e3fa844e in google::protobuf::internal::ExtensionSet::MergeFrom (this=0x268f048, other=value optimized out) at google/protobuf/extension_set.cc:644 #7 0x2b85e3fdf191 in google::protobuf::MessageOptions::MergeFrom (this=0x268f040, fr...@0x268a530) at google/protobuf/descriptor.pb.cc:4732 #8 0x2b85e3fcebf3 in google::protobuf::DescriptorBuilder::AllocateOptionsImplgoogle::protobuf::Descriptor (this=0x7fffc6d68730, name_sco...@0x268ef40, element_na...@0x268ef40, orig_optio...@0x268a530, descriptor=0x2687820) at google/protobuf/descriptor.cc:2648 #9 0x2b85e3fc6fcb in google::protobuf::DescriptorBuilder::BuildMessage (this=0x7fffc6d68730, pro...@0x268a370, parent=0x0, result=0x2687820) at google/protobuf/descriptor.cc:2625 #10 0x2b85e3fc7a86 in google::protobuf::DescriptorBuilder::BuildFile (this=0x7fffc6d68730, pro...@0x7fffc6d687b0) at google/protobuf/descriptor.cc:2814 #11 0x2b85e3fc8c5f in google::protobuf::DescriptorPool::BuildFileFromDatabase (this=0x25b6c90, pro...@0x7fffc6d687b0) at google/protobuf/descriptor.cc:2197 #12 0x2b85e3fc9c53 in google::protobuf::DescriptorPool::TryFindFileInFallbackDatabase (this=0x25b6c90, na...@0x7fffc6d68ac0) at google/protobuf/descriptor.cc:1230 #13 0x2b85e3fc9d40 in google::protobuf::DescriptorPool::FindFileByName (this=0x25b6c90, na...@0x7fffc6d68ac0) at google/protobuf/descriptor.cc:875 #14 0x00726c3c in protobuf_unittest::protobuf_AssignDesc_google_2fprotobuf_2funittest_5fcustom_5foptions_2eproto () at google/protobuf/unittest_custom_options.pb.cc:79 #15 0x2b85e53005d3 in pthread_once () from /lib/libpthread.so.0 #16 0x00426431 in google::protobuf::GoogleOnceInit (once=0xa7b148, init_func=0x726bfd protobuf_unittest::protobuf_AssignDesc_google_2fprotobuf_2funittest_5fcustom_5foptions_2eproto()) at ./google/protobuf/stubs/once.h:115 #17 0x0071c3c0 in protobuf_AssignDescriptorsOnce () at google/protobuf/unittest_custom_options.pb.cc:342 #18 0x0071ca01 in protobuf_unittest::TestMessageWithCustomOptions::descriptor () at google/protobuf/unittest_custom_options.pb.cc:766 #19 0x00479d88 in google::protobuf::descriptor_unittest::CustomOptions_OptionLocations_Test::TestBody (this=0x262eab0) at google/protobuf/descriptor_unittest.cc:1956 #20 0x2b85e44ec225 in testing::Test::Run (this=0x262eab0) at src/gtest.cc:2135 #21 0x2b85e44f14de in testing::internal::TestInfoImpl::Run (this=0x260e980) at src/gtest.cc:2355 #22 0x2b85e44f9716 in testing::internal::TestInfoImpl::RunTest (test_info=0x260e960) at ./src/gtest-internal-inl.h:496 #23 0x2b85e44f22be in testing::internal::Vectortesting::TestInfo*::ForEachvoid (*)(testing::TestInfo*) (this=0x260eb40, functor=0x2b85e44f96f9 testing::internal::TestInfoImpl::RunTest(testing::TestInfo*)) at ./src/gtest-internal-inl.h:344 #24 0x2b85e44f13c9 in testing::internal::TestCase::Run (this=0x260eaf0) at
Re: Building without RTTI support causes make check to hang (recursive call to pthread_once?)
Oliver Jowett wrote:
#4 0x2b85e3ffb67a in google::protobuf::internal::ReflectionOps::Merge
(fr...@0x262e320, to=0x268f0e0) at ./google/protobuf/message.h:311
#5 0x0071dc4a in protobuf_unittest::ComplexOpt6::MergeFrom
(this=0x268f0e0, fr...@0x262e320) at
google/protobuf/unittest_custom_options.pb.cc:3753
#6 0x2b85e3fa844e in
google::protobuf::internal::ExtensionSet::MergeFrom (this=0x268f048,
other=value optimized out) at google/protobuf/extension_set.cc:644
#7 0x2b85e3fdf191 in google::protobuf::MessageOptions::MergeFrom
(this=0x268f040, fr...@0x268a530) at google/protobuf/descriptor.pb.cc:4732
The difference in behaviour is in this bit of generated code:
void MessageOptions::MergeFrom(const ::google::protobuf::Message from) {
Sorry, wrong bit of code. The MessageOptions::MergeFrom in the
stacktrace is actually the const MessageOptions from variant. It
looks like it's actually ComplexOpt6::MergeFrom that differs in
behaviour with RTTI off.
-O
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
