[protobuf] C++ "bytes" fields access problem
Hi. I get funny problem. In proto file I have one "bytes" field, it used for exchanging small binary array of data(~100 bytes). In java side it is ok, because of using ByteString as container. But on C++ consumer I get an error, when access to field: libprotobuf ERROR c:\projects \protobuf-2.3.0\src\google\protobuf\wire_format.cc:1059] Encountered string containing invalid UTF-8 data while parsing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. In generated binding class I have only getter with return type std::string. In this mail list i find two answers: 1) strings is ok for binary data 2) problem with zero bytes while presenting byte array as string The first answer is not Ok, because it add constraint for binary data. Second one looks like true, but I cant do anything with it because of container constrains. Any ideas how to convert byte array to string without UTF-8 check, or how to get raw byte array? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Java reflection package resolution
Hello, I am using the google protocol buffer reflection api to try to dynamically create a gui for users to fill in Messages manually. I had done this completely using Java reflection alone, but this eventually led to problems synchronizing the data, and was all around more work than google's reflection api should be. I have implemented as follows: I pass a Descriptor into the main JPanel building function. From here, it iterates over the fields, and if it is integral, it makes a JTextField, and if it is a Message, it calls the same function again. What I can not find out is the type that the fielddescriptor as I am iterating over the fields from the descriptor. It appears that one "should" be able to figure this out in several ways, but none work for me. First I thought it must be soemthing in JavaType, but that appears to only be what google protocol buffers converted the .proto into. Next I thought I could get the full package name and me a newInstance of it, but when calling fullName, it only returns the class name and no package name. Next, I tried getting the file and querying it's package name, but it returns "". Like I said above, I implemented all of this in pure java reflection, and the .proto generates the .class files correctly with fully qualified names, but the google protocol buffer reflection api (using descriptors and field descriptors) does not give me enough information. Anyone have any ideas as to why it's not giving me what I want? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] C++ "bytes" fields access problem
Can you provide a small reproduction of the problem? The generated code should not be doing any UTF8 validation. Is it possible you have a field defined elsewhere that has type 'string'? On Tue, Sep 7, 2010 at 4:40 AM, Mk wrote: > Hi. > I get funny problem. In proto file I have one "bytes" field, it used > for exchanging small binary array of data(~100 bytes). In java side it > is ok, because of using ByteString as container. But on C++ consumer I > get an error, when access to field: libprotobuf ERROR c:\projects > \protobuf-2.3.0\src\google\protobuf\wire_format.cc:1059] Encountered > string containing invalid UTF-8 data while parsing protocol buffer. > Strings must contain only UTF-8; use the 'bytes' type for raw bytes. > > In generated binding class I have only getter with return type > std::string. In this mail list i find two answers: > 1) strings is ok for binary data > 2) problem with zero bytes while presenting byte array as string > > The first answer is not Ok, because it add constraint for binary data. > Second one looks like true, but I cant do anything with it because of > container constrains. > > Any ideas how to convert byte array to string without UTF-8 check, or > how to get raw byte array? > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Java reflection package resolution
I'm confused. You mention using pure Java reflection, yet you also said you are trying to do some stuff with protobuf reflection. You should not be using Java reflection at all. To get the field type, as defined in the .proto file, use FieldDescriptor.getType() On Tue, Sep 7, 2010 at 6:29 AM, narfm...@yahoo.com wrote: > Hello, > > I am using the google protocol buffer reflection api to try to > dynamically create a gui for users to fill in Messages manually. I had > done this completely using Java reflection alone, but this eventually > led to problems synchronizing the data, and was all around more work > than google's reflection api should be. > > I have implemented as follows: I pass a Descriptor into the main > JPanel building function. From here, it iterates over the fields, and > if it is integral, it makes a JTextField, and if it is a Message, it > calls the same function again. What I can not find out is the type > that the fielddescriptor as I am iterating over the fields from the > descriptor. It appears that one "should" be able to figure this out in > several ways, but none work for me. > > First I thought it must be soemthing in JavaType, but that appears to > only be what google protocol buffers converted the .proto into. Next I > thought I could get the full package name and me a newInstance of it, > but when calling fullName, it only returns the class name and no > package name. Next, I tried getting the file and querying it's package > name, but it returns "". > > Like I said above, I implemented all of this in pure java reflection, > and the .proto generates the .class files correctly with fully > qualified names, but the google protocol buffer reflection api (using > descriptors and field descriptors) does not give me enough > information. Anyone have any ideas as to why it's not giving me what I > want? > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Announcing Lua Support
Thanks! I've added your implementation to the wiki (along with the other Lua implementation, which didn't seem to be on the list) On Sun, Sep 5, 2010 at 9:47 PM, Gregory Szorc wrote: > Oops! I didn't mean to claim my project marked the arrival of protocol > buffers to Lua. There is a prior Lua implementation at > http://code.google.com/p/protoc-gen-lua/. The major difference is most of > that one is implemented in Lua (only the low-level packing/unpacking is in > C) and mine is completely done in C/C++. > > Sorry for the follow-up spam, but I wanted to correct the false assertion > in my previous post. > > Greg > > *From:* Gregory Szorc > *Sent:* Sunday, September 05, 2010 9:30 PM > *To:* protobuf@googlegroups.com > *Subject:* Announcing Lua Support > > I'm pleased to announce the arrival of protocol buffers to Lua! > > The project lives at http://github.com/indygreg/lua-protobuf . > Documentation can be found on the wiki there. > > The plugin generates C++ code that interfaces with both the C++ code > generated by protoc and the "official" Lua C API. So, support is limited to > Lua implementations that conform to that API (official distribution on > lua.org, LuaJIT, possibly others). > > There are a number of features still absent and some known bugs and issues > (see the wiki). But, it is generally usable and has many of the features > you'd expect, so releasing I am. In the days ahead, I hope to spend time on > a test suite as well as exposing more of the C++ API to Lua. > > Comments, feedback, and suggestions (especially on how to deal with the > lua_Number conversion problem) are very welcome. > > Gregory Szorc > gregory.sz...@gmail.com > > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Doc Bug: lang guide says to use "_abstract_services"; it should be "_generic_services"
Just reporting a simple doc bug; the language guide says to use the _abstract_services options (ex: cc_abstract_services) to disable service/rpc generation, but the actual options are named _generic_services. See http://code.google.com/apis/protocolbuffers/docs/proto.html#options --Nate -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Scala protocol buffers protoc plugin
Default values for required fields do mean that: use the default if not explicitly set. This only affects the in-memory object, though - IsInitialized() will still fail if a required field is not set. However, such a proto would be usable by a client that does not do required field checking by using the Partial versions of the serialization/parsing routines. But most clients use the versions with IsInitialized() checks, so default values are not really useful for required fields in practice, aside from documentation purposes in the .proto file. On Thu, Sep 2, 2010 at 6:34 PM, Jeff Plaisance wrote: > Does default on required fields currently have a meaning? It compiles but > I'm not sure if there are cases where the default is ever usable. Maybe > required with default specified could mean use the default if not explicitly > set. This change would also allow you to get rid of unused required fields > over 2 releases. This might be too serious of an api change at this point, > but it seems like required has fallen out of favor anyway, so maybe this > wouldn't be too bad. > > > On Tue, Aug 24, 2010 at 4:47 PM, Kenton Varda wrote: > >> On Wed, Aug 18, 2010 at 7:07 AM, Jeff Plaisance >> wrote: >> >>> It seems like the issue here is that optional has been overloaded to mean >>> two different things: >>> >>> 1) Not really optional but in order to do rolling upgrades we're making >>> it optional. The default should be used if it is not set. In my opinion, >>> in this case there should be no "has" method because either its result is >>> irrelevant or it is being used to overload some other meaning on top of >>> optional. >>> 2) Optional in the sense of Option, Maybe, Nullable, empty, can be null, >>> whatever you want to call it. In my opinion this should be encapsulated in >>> the type so that the programmer is forced to handle all possible cases. The >>> has method should not be used for this because it is too easy to ignore. >>> >> >> Yes, I think you're right, and I see how it makes sense to distinguish >> these two by the presence or absence of an explicit default value. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Scala protocol buffers protoc plugin
To restate and generalize what others have said: Default values have one purpose: they define what is returned by a field's getter if the field is not set. The meaning applies equally to required and optional fields. That's it. The default value simply does not affect anything except for the getter method. In particular it is never sent over the wire. On Thu, Sep 2, 2010 at 6:34 PM, Jeff Plaisance wrote: > Maybe required with default specified could mean use the default if not > explicitly set. This change would also allow you to get rid of unused > required fields over 2 releases. If all software that compiles in your protocol is on a defined release schedule, then you can also take the approach of just changing the required field to optional, but continuing to always set it until the next release. In practice, we've found it very hard to coordinate release schedules in this way -- and too easy to *think* it's safe when it actually isn't -- so we instead just say that required fields shall remain required forever. > This might be too serious of an api change at this point, but it seems > like required has fallen out of favor anyway, so maybe this wouldn't be too > bad. Yes, behavioral changes that would affect existing .proto files are generally non-starters. > > > On Tue, Aug 24, 2010 at 4:47 PM, Kenton Varda wrote: > >> On Wed, Aug 18, 2010 at 7:07 AM, Jeff Plaisance >> wrote: >> >>> It seems like the issue here is that optional has been overloaded to mean >>> two different things: >>> >>> 1) Not really optional but in order to do rolling upgrades we're making >>> it optional. The default should be used if it is not set. In my opinion, >>> in this case there should be no "has" method because either its result is >>> irrelevant or it is being used to overload some other meaning on top of >>> optional. >>> 2) Optional in the sense of Option, Maybe, Nullable, empty, can be null, >>> whatever you want to call it. In my opinion this should be encapsulated in >>> the type so that the programmer is forced to handle all possible cases. The >>> has method should not be used for this because it is too easy to ignore. >>> >> >> Yes, I think you're right, and I see how it makes sense to distinguish >> these two by the presence or absence of an explicit default value. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Memory leak detected in protobuf
Calling the destructor is the only thing you need to do to completely free a protobuf object. Please make sure you are calling ShutdownProtobufLibrary() before your program exits, otherwise some leak detectors will report false positives. On Thu, Sep 2, 2010 at 8:03 AM, Jean-Sebastien Stoezel wrote: > Hello, > > I am observing a memory leak after including the protobufs in my > project. I would like to investigate whether this leak is due to this > library or the use I make of it. > > I am using the protobufs (latest version) with VC++, running 3 > threads. > 2 server threads are pending, waiting for connections, another thread > acts as client, connecting to the servers and sending messages to > them. > I tested this code without the protobufs and I did not notice an > increase in memory usage over time. After adding the protobufs, I see > a constant increase in memory usage. > > I instrumented the code with VC++ crtdbg library, the tool used to > detect memory leaks. The tool is reporting memory leaks which I > believe point to protobuf. I read the crtdbg documentation and if > _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the > file where the leak happened. However no file name is provided by the > tool, it could be because I statically link to the protobufs. > > Anyhow, from the server thread's context, protobufs are stored in a > vector. A packet delimiter parses data from a socket. Each packet is > parsed for a single protobuf message. > > Pseudo code: > > m_IncomingMessages is defined as > std::vector m_IncomingMessages>, > where LFLComProtocol::LFLComProtocolMsg is a protobuf message. > > > // adding a protobuf to the protobuf queue, after a packet of data was > received > // Add an element to the vector > m_IncomingMessages.resize(m_IncomingMessages.size() + 1); > // Parse the data from the new message > m_IncomingMessages[m_IncomingMessages.size() - > 1].ParseFromArray(p_Data, dataSize); > > // consuming the protobufs, FIFO style > m_IncomingMessages.erase(m_IncomingMessages.begin()); > > > I have checked that the messages are properly queued and dequeued, the > size of the vector does not increase and stay constant (usually 0 > since the message is consumed right away) > > At this point I am thinking that this way of erasing may be causing > the leak. From the STL documentation I gather that the vector::erase > method is supposed to call the destructor for each element... Is this > good enough to free all the memory allocated for the message? > > Any other idea why this memory leak is happening? > > Jean-Sebastien > > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] How to import a compiler generated .java file
If you make sure that protoc's output directory is in Eclipse's source path, Eclipse should pick it up. You may have to right-click on the project and choose "refresh" whenever you regenerate it. On Fri, Sep 3, 2010 at 2:48 PM, Sean wrote: > I have a example.proto which will generate example.java. > > Here is what I want to achieve: > > 1) I want to organize my directory like the following... > project | > --build.xml > --/src/*java (all java code I wrote, one of the java file > needs to "import com.mycompany.proto.example" to achieve its logic) > --/proto/example.proto (the proto definition file) > --/lib/*.jar (the dependent third party jar) > > 2) I want to delete the generated file each time when I "ant clean". > > 3) I do not want to manually import example.java each time I launch > eclipse. >(But eclipse can't resolve "import com.mycompany.proto.example" if > I do not do so) > > Can someone share any experience? > > Thanks, > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Java reflection package resolution
Descriptors are just descriptors, not implementations. Several implementations can have the same Descriptor. E.g. you can have a generated class instance and a DynamicMessage instance that have the same Descriptor. Therefore, you cannot instantiate a message class given just its Descriptor. I think what you're looking for is Builder.newBuilderForField(). E.g.: builder.setField(field, builder.newBuilderForField(field).setField(...).build()); Thus as long as you start out with a builder for the root message, you can construct builders for all sub-messages. You can look at com.google.protobuf.TextFormat for a good example of how to use the protobuf reflection interface. On Tue, Sep 7, 2010 at 6:29 AM, narfm...@yahoo.com wrote: > Hello, > > I am using the google protocol buffer reflection api to try to > dynamically create a gui for users to fill in Messages manually. I had > done this completely using Java reflection alone, but this eventually > led to problems synchronizing the data, and was all around more work > than google's reflection api should be. > > I have implemented as follows: I pass a Descriptor into the main > JPanel building function. From here, it iterates over the fields, and > if it is integral, it makes a JTextField, and if it is a Message, it > calls the same function again. What I can not find out is the type > that the fielddescriptor as I am iterating over the fields from the > descriptor. It appears that one "should" be able to figure this out in > several ways, but none work for me. > > First I thought it must be soemthing in JavaType, but that appears to > only be what google protocol buffers converted the .proto into. Next I > thought I could get the full package name and me a newInstance of it, > but when calling fullName, it only returns the class name and no > package name. Next, I tried getting the file and querying it's package > name, but it returns "". > > Like I said above, I implemented all of this in pure java reflection, > and the .proto generates the .class files correctly with fully > qualified names, but the google protocol buffer reflection api (using > descriptors and field descriptors) does not give me enough > information. Anyone have any ideas as to why it's not giving me what I > want? > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Doc Bug: lang guide says to use "_abstract_services"; it should be "_generic_services"
Wow, I can't believe that wasn't noticed earlier. I will fix these. Thanks for the note. On Tue, Sep 7, 2010 at 11:28 AM, Nate Jones wrote: > Just reporting a simple doc bug; the language guide says to use the > _abstract_services options (ex: cc_abstract_services) to disable > service/rpc generation, but the actual options are named > _generic_services. > > See http://code.google.com/apis/protocolbuffers/docs/proto.html#options > > --Nate > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
