[protobuf] C++ JsonStringToMessage with nested Any message
I hit a roadblock trying to parse out a nested message. I have found a
workaround, but I'd like to get to the bottom of this.
I have a .proto file like this:
```syntax = "proto3";
import "google/protobuf/descriptor.proto";
import "google/protobuf/any.proto";
package Test2;
message Subscription {
google.protobuf.FileDescriptorSet descriptor_set = 1;
string name = 2;
int32 interval = 3;
}
message AnotherMsg {
google.protobuf.FileDescriptorSet descriptor_set = 1;
string test = 2;
}
enum msg_id {
UNSPECIFIED = 0;
SUBSCRIBE = 1;
}
message WSMessage{
msg_id req_id = 1;
google.protobuf.Any msg_body = 2;
google.protobuf.FileDescriptorSet descriptor_set = 3;
}
message WSReceivedMessage {
repeated WSMessage message = 1;
google.protobuf.FileDescriptorSet descriptor_set = 2;
}```
WSMessage has an ID, and a message. The documentation indicates
JsonStringToMessage "It will use the DescriptorPool
of
the passed-in message to resolve Any types."
Now, I haven't been able to find any member data or member function of
Message that allows access to a DescriptorPool. I see access to
descriptors...
I even went down the rabbit hole of using the "Parser" with input from the
Tokenizer to read the descriptor.proto file:
```std::fstream myFile;
myFile.open("descriptor.proto");
google::protobuf::io::IstreamInputStream realFileInput();
google::protobuf::io::IstreamInputStream* myFileInput =
myCollector realCollector{};
myCollector* collector =
google::protobuf::io::Tokenizer myTkn(myFileInput, collector);
google::protobuf::compiler::Parser myParser{};
google::protobuf::FileDescriptorProto myProto;
myParser.Parse(, );```
>From this parser, I obtained my FileDescriptorProto, which I then used in a
DescriptorPool:
```google::protobuf::DescriptorPool myPool{};
myPool.BuildFile(myProto);```
Next I tried to use the DynamicMessageFactory to create a message that
would contain the whole DescriptorPool of messages in my main .proto file.
This is where I hit the roadblock. There is no single descriptor, or
converter from a descriptor pool to a descriptor. The DynamicMessageFactory
creates a message from a single descriptor.
I can just simply declare a single message. My issue is I'm looking to have
a message with a descriptor pool (or some sort of descriptor) that contains
the possibilities for my "Any" message.
The documentation seems incorrect to me. util::Status
util::JsonStringToMessage(
StringPiece input,
Message
*
message,
const JsonParseOptions
&
options)
Converts from JSON to protobuf message.
This is a simple wrapper of JsonStringToBinary(). It will use the
DescriptorPool
of
the passed-in message to resolve Any types.
There doesn't appear to be a "DescriptorPool" for a message. What am I
missing? Thanks very much for your help.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/ffa42fd2-5706-462a-879c-f0f3bbaf7894n%40googlegroups.com.
[protobuf] Using Any objects (in C++)
Everything that I've been trying has been failing completely so I'm
wondering if there is something that I missing or doing wrong.
To summarize, I have an Any message (let's call it mA) that I know contains
a certain other specific message (call it messageTypeB). If I create an
instance of messageTypeB (call it mB) and mA.UnpackTo() it fails. If I
use the mA.Is() method it also returns false. If I compare
mA.type_url() to mB.GetDescriptor()->full_name() they differ only in that
the former has "type.googleapis.com/" at the front. So it seems that they
do/should be corresponding to the same type. Yet the Is() method still
says they don't.
Just to see what would happen I tried creating a new Any object (call it
mANew) from mA and use set_type_url() to set the url to exactly the string
that I'm getting back from mB, but even then mANew.Is()
returns false.
Sometimes code speaks louder than words so here is an example of what I'm
doing that seems like it should work but isn't:
const ::vthree::protobuf::Any& anyMessage =
parametricAttribution.layer_specific_attribution();
const std::string anyUrl =
anyMessage.type_url();
// anyUrl is
"type.googleapis.com/com.here.pb.hdmap.platform.v1.core.LinkParametricAttributionList"
::com::here::pb::hdmap::platform::v1::core::LinkParametricAttributionList
coreLinkParametricAttributionList;
const std::string aName =
coreLinkParametricAttributionList.GetDescriptor()->full_name();
// aName is
"com.here.pb.hdmap.platform.v1.core.LinkParametricAttributionList"
if
(anyMessage.Is<::com::here::pb::hdmap::platform::v1::core::LinkParametricAttributionList>())
{
// It never gets in here.
}
So now I'm thoroughly baffled (and going a little crazy). When I ask the
message for a string telling me type it is, it tells me that it is of a
certain type. But if I use the Is() method with that type, it returns
false. Even if I set the type_url() string to exactly match what I get
from the descriptor of the type that I'm trying to unpack into it still
says it's not that type.
Has anybody else been using Any objects in C++ and had success? If so,
were you doing anything different than what I've shown here? Elsewhere at
my company we are using these same protobuf binary files in Java and having
no problems
so either I'm missing something here or the problem might be specific to
the C++ libraries. (For reference we're using Beta-1 but it didn't seem
like Beta-2 offered any changes that would be related.)
-Justin
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Number of issues with protobuf.
I'm not sure what version of protobuf you're using but if you're using 3
you could try using Any. I have run into a similar issue where I was using
a base message object to contain message type, message name as well as an
Any that contained the message payload. This Any is typically some other
protobuf message object that is unpacked according to the aforementioned
variables of the base message object.
You could use an enum like Mark mentioned in his response or an int to
determine how the Any is to be unpacked.
On Sunday, November 15, 2015 at 7:05:22 PM UTC-5, Jakob Bowyer wrote:
>
> I really like using protobuf but I cannot find a way to make this one case
> work.
>
> I need to define a base, lets call it Transaction. Each message could be
> implemented differently so my original implementation was
>
> message Transaction{
> oneof type {
> NTx n = 1;
> TTx tx = 2;
> }
> }
>
> message NTx {
> required string something_else;
> }
>
> message TTx {
> required string soemthing;
> }
>
> Now I need people outside of the project to extend Transaction.
> I realised that oneof can't be extended so I moved to doing
>
> message Transaction {
> enum /* ... */
>
> }
>
> This is starting to get chaotic. So I moved to
>
> message Transaction {
> required int32 tx_type = 1;
> extensions 100 to max;
> }
>
> message NTx {
> extend Transaction {
> optional int32 tx_type = 101;
> }
> optional string something = 1;
> }
>
>
> This also is horrible to work with due to how extensions work in Java.
>
>
>
> I really need some help.
>
> I need a way of a generic transaction envelope. Then I need end users to
> define their own transaction types with arbitrary data in it.
>
> And it could be two or three client modules combined together. If anyone
> could help me I would be grateful.
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Possible C++ serialization issue with repeated fields
My team and I noticed a potential bug in the serialization process, that
seems unintended.
If I defined a message structure as follows:
message X {
required Y y = 1;
}
message Y {
repeated Things things = 1;
repeated Stuff stuff = 2;
}
message Things {
required string name = 1;
}
message Stuff {
required string name = 1;
}
When I create an object of type X, serialize and deserialize it, using the
following code (for example):
X x_original;
std::string x_content;
x_original.SerializeToString(x_content);
X x_new;
x_new.ParseFromString(x_content);
The ParseFromString call will throw an error :
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
message of type testDomain.X because it is missing required fields: y
Our intended use case for this is, in our example, letting another
application know that there are no Things and no Stuff available, so I
believe this should work as written.
if I, however, change the code to :
X x_original;
Y y;
x_original.mutable_y()-CopyFrom(y);
std::string x_content;
x_original.SerializeToString(x_content);
X x_new;
x_new.ParseFromString(x_content);
It works fine, as I believe would set_allocated_y() if I used a *y.
Is this an intended 'feature' and should I be doing something different?
Thanks,
Justin
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Possible C++ serialization issue with repeated fields
It's also worth noting that doing :
x_original.IsInitialized()
will return true, even though it cannot be deserialized.
On Wednesday, June 18, 2014 12:26:10 PM UTC-4, Justin wrote:
My team and I noticed a potential bug in the serialization process, that
seems unintended.
If I defined a message structure as follows:
message X {
required Y y = 1;
}
message Y {
repeated Things things = 1;
repeated Stuff stuff = 2;
}
message Things {
required string name = 1;
}
message Stuff {
required string name = 1;
}
When I create an object of type X, serialize and deserialize it, using the
following code (for example):
X x_original;
std::string x_content;
x_original.SerializeToString(x_content);
X x_new;
x_new.ParseFromString(x_content);
The ParseFromString call will throw an error :
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
message of type testDomain.X because it is missing required fields: y
Our intended use case for this is, in our example, letting another
application know that there are no Things and no Stuff available, so I
believe this should work as written.
if I, however, change the code to :
X x_original;
Y y;
x_original.mutable_y()-CopyFrom(y);
std::string x_content;
x_original.SerializeToString(x_content);
X x_new;
x_new.ParseFromString(x_content);
It works fine, as I believe would set_allocated_y() if I used a *y.
Is this an intended 'feature' and should I be doing something different?
Thanks,
Justin
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Possible C++ serialization issue with repeated fields
On Wednesday, June 18, 2014 12:36:27 PM UTC-4, Ilia Mirkin wrote:
On Wed, Jun 18, 2014 at 12:26 PM, Justin jsca...@gmail.com javascript:
wrote:
My team and I noticed a potential bug in the serialization process, that
seems unintended.
If I defined a message structure as follows:
message X {
required Y y = 1;
}
message Y {
repeated Things things = 1;
repeated Stuff stuff = 2;
}
message Things {
required string name = 1;
}
message Stuff {
required string name = 1;
}
When I create an object of type X, serialize and deserialize it, using
the
following code (for example):
X x_original;
std::string x_content;
x_original.SerializeToString(x_content);
I believe this returns false, which indicates an error, since
x_original.y isn't set, and it's required.
This actually doesn't return false, however a call to IsInitialized() does.
In all of my testing, SerializeToString() returns true regardless of
content. For example, if I change the code to this:
bool response = x_original.SerializeToString(x_content);
std::cout Serialized: std::boolalpha response std::endl;
I get, Serialized: true
X x_new;
x_new.ParseFromString(x_content);
The ParseFromString call will throw an error :
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
message of type testDomain.X because it is missing required fields: y
Our intended use case for this is, in our example, letting another
application know that there are no Things and no Stuff available, so I
believe this should work as written.
Perhaps y should be optional then? As the proto is written, you must
supply a y.
I guess I assumed that it automatically creates the branches in the tree,
but doesn't create the leaves, but if it only creates a branch for
something that has a leaf, this functionality makes more sense to me.
if I, however, change the code to :
X x_original;
Y y;
x_original.mutable_y()-CopyFrom(y);
std::string x_content;
x_original.SerializeToString(x_content);
X x_new;
x_new.ParseFromString(x_content);
It works fine, as I believe would set_allocated_y() if I used a *y.
Is this an intended 'feature' and should I be doing something different?
Yep, it's a feature -- if a field is required, it must be present in
order to serialize and deserialize without error.
Thanks for the responses. We can work around this, but it's worth noting
that it will serialize without error as written :).
-ilia
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] C++ Code Gen for Repeated Enumerations
I noticed that if I have a message with a repeated field of an enumeration
type, such as:
message foo {
enum bar_e {
BAR_VALUE_1
}
repeated bar_e bars = 1;
}
The code generation seems to have an error in typing when accessing the
repeated field. I would expect that the line:
foo.bars()
to return me a RepeatedFieldbar_e, but it actually returns
RepeatedFieldint32. I understand that enums and int32 are convertible,
but it seems ... strange.
This is evident when I try to use the C++11 for loops, because this doesn't
work:
for (bar_e bar: foo.bars())
Instead, I have to do something like:
for (int bar_int : foo.bars())
and then static_cast it to the enumeration ... or use a standard for loop,
since foo.bar(x) returns the enum as you would expect.
Is this intended behavoir? And if so, why?
Thanks!
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
[protobuf] C++11 for each loops with RepeatedField
I assume there is a simple answer to this, but I'm having trouble finding it and my searches for similar questions have come up empty ... Perhaps it being Friday afternoon is also reducing my ability to solve this :) I want to be able to use C++11 range-based for loops on a repeated field in my message, but I can't seem to get it to compile properly. This is true for mutable and const references. So, for example: for (message_item curr_message : data.mutable_message_item()) does not work, nor can I do so without the mutable, but I would think it should. Is this because RepeatedField does not support the appropriate operators? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] C++11 for each loops with RepeatedField
Thanks! I knew it was going to be something simple ... I blame end of week exhaustion! On Friday, April 18, 2014 2:40:14 PM UTC-4, Feng Xiao wrote: On Fri, Apr 18, 2014 at 11:28 AM, Justin jsca...@gmail.com javascript:wrote: I assume there is a simple answer to this, but I'm having trouble finding it and my searches for similar questions have come up empty ... Perhaps it being Friday afternoon is also reducing my ability to solve this :) I want to be able to use C++11 range-based for loops on a repeated field in my message, but I can't seem to get it to compile properly. This is true for mutable and const references. So, for example: for (message_item curr_message : data.mutable_message_item()) Try: for (message_item curr_message : ***data.mutable_message_item()) does not work, nor can I do so without the mutable, but I would think it should. Is this because RepeatedField does not support the appropriate operators? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Has no symbols error during make install
I would like to install protocol buffer in my own user directory instead of /usr/local. I am running ./configure with the --prefix option. I have run make clean to eliminate worries of recompiling. When running make install I get the following errors: libtool: install: ranlib /home/mydir/opt/lib/libprotobuf-lite.a ranlib: file: /home/mydir/opt/lib/libprotobuf-lite.a(atomicops_internals_x86_gcc.o) has no symbols ranlib: file: /home/mydir/opt/lib/libprotobuf-lite.a(atomicops_internals_x86_msvc.o) has no symbols ../libtoll: line 1096: 57502 Bus error: 10 ranlib /home/mydir/opt/lib/libprotobuf-lite.a make[3]: *** [install-libLTLIBRARIES] Error 138 make[2]: *** [install-am] Error 2 make[1]: *** [install] Error 2 make: *** [install-recursive] Error 1 Does anyone know what could potentially be the issue here and how to resolve? Thanks -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Re: generating proto documentation
I realize this thread had been dead for several years, but I'm wondering if there is a good link somewhere to your perl script? I need to generate documentation for proto files and had decided that was probably the best approach. If there were a plugin for Visual Studio that would be fine too since Visual Studio can generate an XML file. On Thursday, May 7, 2009 5:23:42 AM UTC-6, OJW wrote: I've put up the perl script we use to generate documentation of proto files: http://dev.openstreetmap.org/~ojw/proto2wikidoc Our proto files are in quite a simplified format, so this probably won't parse your files without modification. However, it's a base to start from if anyone wants to add better parsing etc. sample output at: http://dev.openstreetmap.org/~ojw/proto2wikidoc/index.php/Myprotocol_protocol thanks to cueSim for releasing the code. Thanks to openstreetmap for (unknown to them) hosting the project page. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] Attempting to build generic Message from serialized data regardless of type (c++)
I'm receiving a serialized message over middleware (qpid), and I want to be able to instantiate a Message object in my base class which does not know the actual message type. I have a feeling this isn't actually possible, and I can easily workaround this by just passing the serialized message to the subclass instead of a Message object, but I would rather do it this way if possible. Here is the relevant code snippet: namespace GPB = google::protobuf; // Check the queue for a message qpid::messaging::Message message = amqpReceiver_.fetch(qpid::messaging::Duration::SECOND * 1); std::string msgContent = message.getContent(); // Build a generic GPB message, without caring about the type. // Step 1: Create a message factory GPB::DynamicMessageFactory factory; // Step 2: Build a descriptor based on the message content GPB::Descriptor typeDescriptor(msgContent); // Step 3: Use the factory to make us an instance of the appropriate message type GPB::Message *gpbMessage = factory.GetPrototype(typeDescriptor).New(); handleMessage(gpbMessage); This is all surrounded by a try/catch block, of course and I clean up memory after this. The handleMessage(Message *) method is a pure virtual that the subclasses implement depending on type. The qpid stuff is largely irrelevant, I just wanted to include mention of where it is coming from. My issue lies in the Step 2, where I don't actually know how to make it work. The variable msgContent contains the serialized GPB message, but this class has no way of knowing the type. Is this even possible? I am pretty new to GPB, so I'm not sure if the serialized data even has enough information in it to allow generic deserialization and I'm partially assuming that it does not, but I'm hoping it does. I have read a bunch of stuff on forums, etc. regarding FileDescriptorProto and DescriptorPool, which may be part of the answer, but I'm not getting the whole picture yet. Thanks for any help you can offer! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Attempting to build generic Message from serialized data regardless of type (c++)
Thank you for your response Ilia.
I was beginning to realize that I would have to provide some additional
information for it to work beyond just the serialized data. I hadn't,
however, thought of embedding it using the AMQP message options to make it
work. Not sure why it didn't occur to me, but it's a good idea! thanks!
On Wednesday, May 22, 2013 1:52:04 PM UTC-4, Ilia Mirkin wrote:
Is it expected that the type of the message is available at
compile/run-time, or are you trying to load it dynamically? If
dynamically, you'd have to pass the descriptor as part of the message,
which is a little annoying (since it's not just the message, but also
any messages that that message references). If statically you should
be able to load the proto by name, you can get that out of
MessageFactory::generated_factory(), otherwise you have to use one of
the dynamic ones and load proto files/descriptor pb's you get over the
wire into it.
Usually with such generic situations, you might have a wrapper which would
be
message TheData {
required string type = 1;
required bytes data = 2;
}
Or a more efficient arrangement like
varint header length
message Header {
required string type = 1;
anything else you want
}
varint data length
the bytes of the proto
This can be easily arranged for with a CodedInput/OutputStream in
Java, and I think there are similar constructs in the C++ impl.
An alternate technique, since you're using AMQP, is to attach meaning
to the channel key/name (or whatever that thing is called) and
determine the type from that.
On Wed, May 22, 2013 at 11:05 AM, Justin jsca...@gmail.com javascript:
wrote:
I'm receiving a serialized message over middleware (qpid), and I want to
be
able to instantiate a Message object in my base class which does not
know
the actual message type. I have a feeling this isn't actually possible,
and
I can easily workaround this by just passing the serialized message to
the
subclass instead of a Message object, but I would rather do it this way
if
possible.
Here is the relevant code snippet:
namespace GPB = google::protobuf;
// Check the queue for a message
qpid::messaging::Message message =
amqpReceiver_.fetch(qpid::messaging::Duration::SECOND * 1);
std::string msgContent = message.getContent();
// Build a generic GPB message, without caring about the type.
// Step 1: Create a message factory
GPB::DynamicMessageFactory factory;
// Step 2: Build a descriptor based on the message content
GPB::Descriptor typeDescriptor(msgContent);
// Step 3: Use the factory to make us an instance of the appropriate
message
type
GPB::Message *gpbMessage = factory.GetPrototype(typeDescriptor).New();
handleMessage(gpbMessage);
This is all surrounded by a try/catch block, of course and I clean up
memory
after this.
The handleMessage(Message *) method is a pure virtual that the
subclasses
implement depending on type. The qpid stuff is largely irrelevant, I
just
wanted to include mention of where it is coming from.
My issue lies in the Step 2, where I don't actually know how to make
it
work. The variable msgContent contains the serialized GPB message, but
this
class has no way of knowing the type. Is this even possible? I am pretty
new
to GPB, so I'm not sure if the serialized data even has enough
information
in it to allow generic deserialization and I'm partially assuming that
it
does not, but I'm hoping it does.
I have read a bunch of stuff on forums, etc. regarding
FileDescriptorProto
and DescriptorPool, which may be part of the answer, but I'm not getting
the
whole picture yet.
Thanks for any help you can offer!
--
You received this message because you are subscribed to the Google
Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to protobuf+u...@googlegroups.com javascript:.
To post to this group, send email to prot...@googlegroups.comjavascript:.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
[protobuf] PATCH: Leading comments droppped when using --descriptor_set_out
Greetings! The protoc compiler will write a FileDescriptorProto protocol buffer describing its input files when given the --descriptor_set_out flag. If --include_source_info is also given, then comments (and other information) for each element will also be included. However, any comments at the beginning of the file that are NOT attached to an element will be dropped. I am working on a documentation generator for Protocol Buffers, using the existing plugin infrastructure, and in many cases the leading comments in the file act as an introduction. Therefore, it's important to me that these comments show up in the FileDescriptorProto provided by the compiler. Would you accept a patch that preserves these comments? I have made the changes locally and would be glad to submit it. Justin p.s. To reproduce the issue, I'll use descriptor.proto. Running the following command puts 'descriptor.pb' in /tmp (assuming descriptor.proto is in the current directory): protoc --include_source_info --descriptor_set_out=/tmp/descriptor.pb descriptor.proto descriptor.proto begins with these lines (v2.5.0): // Protocol Buffers - Google's data interchange format // Copyright 2008 Google Inc. All rights reserved. // http://code.google.com/p/protobuf/ And that text is missing from descriptor.pb -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Best practices for proto file organization in large projects
Okay, so I got this build tree working. I had to: (1) Modify FindProtobuf.cmake (the build rule) so that I can specify the proto_path (2) Modify each .proto import to specify the file to import using the full path from the root, even if the files are parallel to the .proto source. (3) Add CMAKE_SOURCE_DIR (the source tree root) to the proto_path (4) Add CMAKE_BINARY_DIR (the build root) to the gcc_path. (5) Do a clean build. I had old generated .pb.h files in the source tree, and they were being included rather then the newly generated files that lives in the build-tree. Thanks for your help. Justin On Wednesday, June 20, 2012 6:24:23 AM UTC-7, Evan Jones wrote: On Jun 19, 2012, at 13:53 , Justin Muncaster wrote: 1Running C++ protocol buffer compiler on common/bar/bar.proto 1common/foo/foo.proto: File not found. 1bar.proto: Import common/foo/foo.proto was not found or had errors. I can fix the error by hacking FindProtobuf.cmake and passing in additional include directories, but I run into problems down the line, which leads me to think there must be a better way. Every example I see has all proto files in one folder and does not have cross-library protobuf message dependencies. This should work, and with the project you attached it does work (well, once I fixed a bad field number): Yamnuska:project ej$ protoc --cpp_out=build common/bar/bar.proto Yamnuska:project ej$ I don't know how CMake or this PROTOBUF_GENERATE_CPP rule works, but maybe you need to pass the appropriate --proto_path argument so it looks for the included .proto in the right place? Good luck, Evan -- http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/19Xfy9exiscJ. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Best practices for proto file organization in large projects
Hello, I've been using protocol buffers for a while, and I love the library, however I find that when using them in large projects I generally have to fight with the compiler to get protoc to play nicely with my build system. My projects are organized as follows: common/foo/foo.h common/foo/foo.cpp common/foo/foo.proto ... common/bar/bar.h common/bar/bar.cpp common/bar/bar.proto where bar.proto contains import common/foo/foo.proto ... and elsewhere... app1/baz.proto ... app1/fud.proto and baz.proto and fud.proto contain import common/bar/bar.proto I'm currently changing our build system to be cmake based and I'm again finding myself fighting with the build system to get the .proto to be automatically generated in a way where they build correctly. This leads me to believe that I am doing something wrong or at least not organizing files in a way that is expected. In light of that, How do you organize your proto files when you have many in common libraries? Do all .proto files live in one folder? Should one avoid import a/b/c/d/f.proto? Do you have any recommendations for how one ought one setup the cmake build system to work with proto files that are organized as they are above? Any general recommendations? Thanks, Justin -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/S24EsiM971cJ. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Recursive import problem
I have the following message structure:
===A.proto===
message A
{
optional B1 b1 = 1;
}
message B1
{
optional A a = 1;
}
Everything works as intended when the messages are in the same file.
However, in my situation there are several child messages (B2,
B3, ...) which all naturally all go in their own separate files
(B2.proto, B3.proto, ...). Unfortunately, B1 cannot go in its own file
due to a problem with a recursive import.
Is there any way to solve this? Can I forward declare messages?
Thanks,
Justin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Initial benchmarking committed to svn (r100)
On Mar 6, 1:13 am, Justin Azoff justin.az...@gmail.com wrote: I did a quick port to python(pasted at the end, hopefully it wont be garbled) well, that didn't work. I threw it up at http://bouncybouncy.net/ramblings/files/ProtoBench.py if anyone is interested. -- - Justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Performance comparison of Thrift, JSON and Protocol Buffers
On Mar 4, 10:24 pm, David Anderson d...@natulte.net wrote: I think the major point to take away from the comparison is: use the correct tool for your needs. If you need backward/forward compatibility, heterogeneous versions of software interacting and some structural validation (just structure, not talking about the higher level semantics of fields), PB/Thrift is what you need. If you don't care about the above points, by all means use json (and don't forget to get your web server to gzip traffic). - Dave I definitely agree! I have been also been looking at this from another angle: Right now JSON is faster than protobuf (at least in python), but protobuf produces smaller output. Protobuf will only get faster, but JSON can not get any smaller. Looking forward, protobuf definitely has an advantage. -- - Justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Initial benchmarking committed to svn (r100)
On Mar 5, 6:07 am, Jon Skeet sk...@pobox.com wrote: I've just committed the first pass of a benchmarking tool to trunk. Everything is under benchmarks - see the readme.txt for usage guidelines. Cool :-) I did a quick port to python(pasted at the end, hopefully it wont be garbled), my results on a 3.2ghz p4: % protoc --python_out tmp google_size.proto % touch tmp/__init__.py % python ProtoBench.py tmp.google_size_pb2.SizeMessage1 google_message1.dat Benchmarking tmp.google_size_pb2.SizeMessage1 with file google_message1.dat Serialize to string: 25388 iterations in 31.0s 0.18MB/s Deserialize from string: 19504 iterations in 30.9s 0.14MB/s with psyco: % python ProtoBench.py tmp.google_size_pb2.SizeMessage1 google_message1.dat Benchmarking tmp.google_size_pb2.SizeMessage1 with file google_message1.dat Serialize to string: 38763 iterations in 36.0s 0.23MB/s Deserialize from string: 22180 iterations in 29.6s 0.16MB/s -- - Justin import time import sys import traceback #import psyco #psyco.full () class ProtoBench: MIN_SAMPLE_TIME_S = 2 TARGET_TIME_S = 30 def runTest(self, type, filename): print Benchmarking %s with file %s % (type, filename) mod_name, class_name = type.rsplit(., 1) try : __import__ (mod_name) mod = sys.modules [mod_name] clazz = getattr(mod, class_name) self.inputData = open(filename).read () sampleMessage = clazz () sampleMessage.ParseFromString (self.inputData) except Exception, e: sys.stderr.write(Unable to get default message for %s\n % type); traceback.print_exc () return False try : self.benchmark(Serialize to string, sampleMessage.SerializeToString) self.benchmark(Deserialize from string, lambda: sampleMessage.ParseFromString (self.inputData)) except Exception, e: sys.stderr.write(Error: %s\n % e) sys.stderr.write(Detailed exception information: \n) traceback.print_exc () return False return True def benchmark(self, name, action): iterations = 1 elapsed = self.timeAction(action, iterations) while elapsed self.MIN_SAMPLE_TIME_S: iterations *= 2; elapsed = self.timeAction(action, iterations) iterations = int ((self.TARGET_TIME_S / elapsed) * iterations) elapsed = self.timeAction(action, iterations) print %s: %d iterations in %0.1fs %0.2fMB/s % ( name, iterations, elapsed, (iterations * len(self.inputData)/elapsed/ 1024/1024) ) def timeAction(self, action, iterations): start = time.clock () for _ in xrange (iterations): action () end = time.clock () return end - start if __name__ == __main__: if len(sys.argv) 3 or len(sys.argv) %2 ! =1: sys.stderr.write(Usage: ProtoBench descriptor type name input data The descriptor type name is the fully-qualified message name, e.g. benchmark.Message1 (You can specify multiple pairs of descriptor type name and input data.) \n) sys.exit (1) pairs = zip(sys.argv[1::2], sys.argv [2::2]) pb = ProtoBench () for type, filename in pairs: pb.runTest(type, filename) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Performance comparison of Thrift, JSON and Protocol Buffers
On Mar 2, 10:52 am, Stephan Richter stephan.rich...@gmail.com wrote: The outcome looks about right. In the latest version of Python, even simplejson has C extensions. A one order of magnitude difference between a pure Python versus C implementation is about right, if not too small. I would have expected a difference of 20-50 times. Besides the fact that this post is is far too short on details - i.e. Python version, OS, hardware -- I would look at it as a great motivation to get the C extensions for PB done quickly. ;-) Regards, Stephan Hi all :-) I actually posted a follow up: http://bouncybouncy.net/ramblings/posts/more_on_json_vs_thrift_and_protocol_buffers/ It turned out I didn't have the simplejson C extension installed... With that installed the speed difference was much greater. The test were run on python2.4 on a 3.2gz p4. Aside from the speed I really like Protocol Buffers, the API and docs are very well done :-) - Justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
