[protobuf] Re: Self-describing messages in Python
Hello,
in the end I managed to fix this issue using CopyFrom(fds), where fds is an
instnce of google.protobuf.descriptor_pb2.FileDescriptorSet, rather than
file.append function as show in my original code.
So in the code above we would call
my_proto_instance.descriptor_set.CopyFrom(addressbook_fds) instead of
my_proto_instance.descriptor_set.file.append(addressbook_fds).
Hope someone finds this useful.
Cheers
Jan
On Friday, April 7, 2023 at 11:38:50 PM UTC+2 Jan Machek wrote:
> Hello,
>
> I am trying to build self-describing messages using Python. I read through
> the documentation
> https://protobuf.dev/programming-guides/techniques/#self-description as
> well as through the Python API
> https://googleapis.dev/python/protobuf/latest/google/protobuf.html, but I
> am still not very sure about how to do that.
>
> I started with the example message
> syntax = "proto3"; import "google/protobuf/any.proto"; import
> "google/protobuf/descriptor.proto"; message SelfDescribingMessage { //
> Set of FileDescriptorProtos which describe the type and its dependencies.
> google.protobuf.FileDescriptorSet descriptor_set = 1; // The message and
> its type, encoded as an Any message. google.protobuf.Any message = 2; }
>
> And I would like to pass through it a simple addressbook message
>
> syntax = "proto2";
>
> package tutorial;
>
> message AddressBook {
> optional string name = 1;
> optional string number = 2;
> }
>
> The AddressBook message would be contained in the message field and the
> descriptor_set field would contain the descriptor of the proto above.
>
> However I could not do this no matter what I have tried.
>
> The closest I got was exporting a file descriptor set from the
> SelfDescribingMessage.proto given in the documentation as "protoc
> --proto_path=. --proto_path=./include
> --descriptor_set_out=./self_describing_ds --include_imports
> self_describing.proto" and then reading it
>
> with open("self_describing_ds", 'rb') as fh:
> fds = descriptor_pb2.FileDescriptorSet.FromString(fh.read())
>
> message_classes = message_factory.GetMessages(fds.file)
> my_proto_instance = message_classes["SelfDescribingMessage"]()
>
> address_book = addressbook_pb2.AddressBook()
> address_book.name = "John Doe"
> address_book.number = "123456"
>
> my_proto_instance.message.Pack(address_book)
>
> I am not able to set my_proto_instance.descriptor_set though. Extracting
> address book descriptor set using protoc and then reading it and trying to
> append it
>
> with open("addressbook_ds", 'rb') as fh:
> addressbook_fds =
> descriptor_pb2.FileDescriptorSet.FromString(fh.read())
> my_proto_instance.descriptor_set.file.append(addressbook_fds)
>
> fails on
>
> TypeError: Parameter to MergeFrom() must be instance of same class:
> expected got 'google.protobuf.descriptor_pb2.FileDescriptorProto'>.
>
> I could not get any closer.
>
> Does anyone have any simple example on sending the self-describing
> messages in Python, please?
>
> Thanks a lot in advance.
>
> Jan
>
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/e85a4c87-9c2b-4a99-8c9c-a88db37a8b6cn%40googlegroups.com.
[protobuf] Re: Self describing messages
Thanks a lot. I change this line to DynamicMessage dm=DynamicMessage.parseFrom(fileDescr.getMessageTypes().get(0), input); look like it is works. Sergey N Lukin On Jun 24, 7:59 pm, Jason Hsueh jas...@google.com wrote: What are you still missing? The code you've written ought to work after changing to pass the correct FileDescriptor object to DynamicMessage. On Fri, Jun 24, 2011 at 12:27 AM, slookin sloo...@gmail.com wrote: Yes, in generally it is not Self Describing, but couldn't implement real self describing, becase the message size. But I'm able upload proto descripter to reciver, so I want to parse incomming message using proto descripter. Could you help me, how I can do it? On Jun 23, 8:12 pm, Jason Hsueh jas...@google.com wrote: Oh, I missed that you were reading the FileDescriptorSet from a separate file, not the same stream. This isn't exactly self describing since when you transmit the message you assume the recipient knows what type the message is, and has access to the FileDescriptorSet. Seehttp:// code.google.com/apis/protocolbuffers/docs/techniques.html#self... On Thu, Jun 23, 2011 at 12:31 AM, slookin sloo...@gmail.com wrote: Thanks Jason, It is all my code (link to real java class - http://test.look-in.net/pf/pf.zip. Could you give me links with examples or explain why i should serializes the FileDescriptorSet? On Jun 22, 8:35 pm, Jason Hsueh jas...@google.com wrote: On Wed, Jun 22, 2011 at 3:30 AM, slookin sloo...@gmail.com wrote: I trying to develop flexible server for reciving message, i'm not able to create java classes for each message type, but I can upload proto descripter on server. Of course client (sender) will use generated java classes for prepare messages: My code (sender): Person.Builder person = Person.newBuilder(); person.setId(Integer.valueOf(42)); person.setEmail(test_em...@gmail.com); person.setName(Viktor Villari); Person p=person.build(); FileOutputStream fstream = new FileOutputStream (message.pf); CodedOutputStream outSream = CodedOutputStream.newInstance(fstream); p.writeTo(outSream); outSream.flush(); System.out.println(sent); Reciver (address.descriptor.proto - grenerated via descriptor.proto): FileInputStream input = new FileInputStream (address.descriptor.proto); DescriptorProtos.FileDescriptorSet fdsProto=DescriptorProtos.FileDescriptorSet.parseFrom(input); input = new FileInputStream (message.pf); // System.out.println(fds.getFile(0).getMessageType(0).getName()); System.out.println(File name = +fdsProto.getFile(0).getName()); // System.out.println(Message type = +fdsProto.getFile(0).getMessageType(0).getName()); // System.out.println(Field info = +fdsProto.getFile(0).getMessageType(0).getField(0).getName()+ +fdsProto.getFile(0).getMessageType(0).getField(0).getType()); FileDescriptor fileDescr=FileDescriptor.buildFrom(fdsProto.getFile(0), new FileDescriptor[0]); System.out.println(Message type = +fileDescr.getMessageTypes().get(0).getName()); System.out.println(Field info = +fileDescr.getMessageTypes().get(0).getFields().get(0).getName()+ type= +fileDescr.getMessageTypes().get(0).getFields().get(0).getType()); DynamicMessage dm=DynamicMessage.parseFrom(fdsProto.getDescriptor(), input); On this line, you are passing the descriptor for the FileDescriptorSet. You should be using fileDescr to get at the Person message type. (Note that your description scheme does not indicate what message type is actually used. But I also don't see the code snippet that serializes the FileDescriptorSet, or for that matter delimit between the FileDescriptorSet and the serialized Person data, so maybe you are doing something elsewhere) System.out.println(DynamicMessage to string \n+dm.toString()); // problem line System.out.println(Person.name = + dm.getField(fileDescr.getMessageTypes().get(0).getFields().get(0))); // Output: File name = addressbook.proto Message type = Person Field info = name type= STRING DynamicMessage to string 2: Viktor Villari 3: 42 4: test_em...@gmail.com Exception in thread main java.lang.IllegalArgumentException: FieldDescriptor does not match message type. at com.google.protobuf.DynamicMessage.verifyContainingType(DynamicMessage.java: 242) at com.google.protobuf.DynamicMessage.getField(DynamicMessage.java: 160)
Re: [protobuf] Re: Self describing messages
What are you still missing? The code you've written ought to work after changing to pass the correct FileDescriptor object to DynamicMessage. On Fri, Jun 24, 2011 at 12:27 AM, slookin sloo...@gmail.com wrote: Yes, in generally it is not Self Describing, but couldn't implement real self describing, becase the message size. But I'm able upload proto descripter to reciver, so I want to parse incomming message using proto descripter. Could you help me, how I can do it? On Jun 23, 8:12 pm, Jason Hsueh jas...@google.com wrote: Oh, I missed that you were reading the FileDescriptorSet from a separate file, not the same stream. This isn't exactly self describing since when you transmit the message you assume the recipient knows what type the message is, and has access to the FileDescriptorSet. Seehttp:// code.google.com/apis/protocolbuffers/docs/techniques.html#self... On Thu, Jun 23, 2011 at 12:31 AM, slookin sloo...@gmail.com wrote: Thanks Jason, It is all my code (link to real java class - http://test.look-in.net/pf/pf.zip. Could you give me links with examples or explain why i should serializes the FileDescriptorSet? On Jun 22, 8:35 pm, Jason Hsueh jas...@google.com wrote: On Wed, Jun 22, 2011 at 3:30 AM, slookin sloo...@gmail.com wrote: I trying to develop flexible server for reciving message, i'm not able to create java classes for each message type, but I can upload proto descripter on server. Of course client (sender) will use generated java classes for prepare messages: My code (sender): Person.Builder person = Person.newBuilder(); person.setId(Integer.valueOf(42)); person.setEmail(test_em...@gmail.com); person.setName(Viktor Villari); Person p=person.build(); FileOutputStream fstream = new FileOutputStream (message.pf); CodedOutputStream outSream = CodedOutputStream.newInstance(fstream); p.writeTo(outSream); outSream.flush(); System.out.println(sent); Reciver (address.descriptor.proto - grenerated via descriptor.proto): FileInputStream input = new FileInputStream (address.descriptor.proto); DescriptorProtos.FileDescriptorSet fdsProto=DescriptorProtos.FileDescriptorSet.parseFrom(input); input = new FileInputStream (message.pf); // System.out.println(fds.getFile(0).getMessageType(0).getName()); System.out.println(File name = +fdsProto.getFile(0).getName()); // System.out.println(Message type = +fdsProto.getFile(0).getMessageType(0).getName()); // System.out.println(Field info = +fdsProto.getFile(0).getMessageType(0).getField(0).getName()+ +fdsProto.getFile(0).getMessageType(0).getField(0).getType()); FileDescriptor fileDescr=FileDescriptor.buildFrom(fdsProto.getFile(0), new FileDescriptor[0]); System.out.println(Message type = +fileDescr.getMessageTypes().get(0).getName()); System.out.println(Field info = +fileDescr.getMessageTypes().get(0).getFields().get(0).getName()+ type= +fileDescr.getMessageTypes().get(0).getFields().get(0).getType()); DynamicMessage dm=DynamicMessage.parseFrom(fdsProto.getDescriptor(), input); On this line, you are passing the descriptor for the FileDescriptorSet. You should be using fileDescr to get at the Person message type. (Note that your description scheme does not indicate what message type is actually used. But I also don't see the code snippet that serializes the FileDescriptorSet, or for that matter delimit between the FileDescriptorSet and the serialized Person data, so maybe you are doing something elsewhere) System.out.println(DynamicMessage to string \n+dm.toString()); // problem line System.out.println(Person.name = + dm.getField(fileDescr.getMessageTypes().get(0).getFields().get(0))); // Output: File name = addressbook.proto Message type = Person Field info = name type= STRING DynamicMessage to string 2: Viktor Villari 3: 42 4: test_em...@gmail.com Exception in thread main java.lang.IllegalArgumentException: FieldDescriptor does not match message type. at com.google.protobuf.DynamicMessage.verifyContainingType(DynamicMessage.java: 242) at com.google.protobuf.DynamicMessage.getField(DynamicMessage.java: 160) at net.lookin.protobuf.Test.main(Test.java:64) Why I recive exception? How I can access to specific field value in DynamicMessage? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group,
[protobuf] Re: Self describing messages
Thanks Jason, It is all my code (link to real java class - http://test.look-in.net/pf/pf.zip. Could you give me links with examples or explain why i should serializes the FileDescriptorSet? On Jun 22, 8:35 pm, Jason Hsueh jas...@google.com wrote: On Wed, Jun 22, 2011 at 3:30 AM, slookin sloo...@gmail.com wrote: I trying to develop flexible server for reciving message, i'm not able to create java classes for each message type, but I can upload proto descripter on server. Of course client (sender) will use generated java classes for prepare messages: My code (sender): Person.Builder person = Person.newBuilder(); person.setId(Integer.valueOf(42)); person.setEmail(test_em...@gmail.com); person.setName(Viktor Villari); Person p=person.build(); FileOutputStream fstream = new FileOutputStream (message.pf); CodedOutputStream outSream = CodedOutputStream.newInstance(fstream); p.writeTo(outSream); outSream.flush(); System.out.println(sent); Reciver (address.descriptor.proto - grenerated via descriptor.proto): FileInputStream input = new FileInputStream (address.descriptor.proto); DescriptorProtos.FileDescriptorSet fdsProto=DescriptorProtos.FileDescriptorSet.parseFrom(input); input = new FileInputStream (message.pf); // System.out.println(fds.getFile(0).getMessageType(0).getName()); System.out.println(File name = +fdsProto.getFile(0).getName()); // System.out.println(Message type = +fdsProto.getFile(0).getMessageType(0).getName()); // System.out.println(Field info = +fdsProto.getFile(0).getMessageType(0).getField(0).getName()+ +fdsProto.getFile(0).getMessageType(0).getField(0).getType()); FileDescriptor fileDescr=FileDescriptor.buildFrom(fdsProto.getFile(0), new FileDescriptor[0]); System.out.println(Message type = +fileDescr.getMessageTypes().get(0).getName()); System.out.println(Field info = +fileDescr.getMessageTypes().get(0).getFields().get(0).getName()+ type= +fileDescr.getMessageTypes().get(0).getFields().get(0).getType()); DynamicMessage dm=DynamicMessage.parseFrom(fdsProto.getDescriptor(), input); On this line, you are passing the descriptor for the FileDescriptorSet. You should be using fileDescr to get at the Person message type. (Note that your description scheme does not indicate what message type is actually used. But I also don't see the code snippet that serializes the FileDescriptorSet, or for that matter delimit between the FileDescriptorSet and the serialized Person data, so maybe you are doing something elsewhere) System.out.println(DynamicMessage to string \n+dm.toString()); // problem line System.out.println(Person.name = + dm.getField(fileDescr.getMessageTypes().get(0).getFields().get(0))); // Output: File name = addressbook.proto Message type = Person Field info = name type= STRING DynamicMessage to string 2: Viktor Villari 3: 42 4: test_em...@gmail.com Exception in thread main java.lang.IllegalArgumentException: FieldDescriptor does not match message type. at com.google.protobuf.DynamicMessage.verifyContainingType(DynamicMessage.java: 242) at com.google.protobuf.DynamicMessage.getField(DynamicMessage.java: 160) at net.lookin.protobuf.Test.main(Test.java:64) Why I recive exception? How I can access to specific field value in DynamicMessage? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Self describing messages
Oh, I missed that you were reading the FileDescriptorSet from a separate file, not the same stream. This isn't exactly self describing since when you transmit the message you assume the recipient knows what type the message is, and has access to the FileDescriptorSet. See http://code.google.com/apis/protocolbuffers/docs/techniques.html#self-description On Thu, Jun 23, 2011 at 12:31 AM, slookin sloo...@gmail.com wrote: Thanks Jason, It is all my code (link to real java class - http://test.look-in.net/pf/pf.zip. Could you give me links with examples or explain why i should serializes the FileDescriptorSet? On Jun 22, 8:35 pm, Jason Hsueh jas...@google.com wrote: On Wed, Jun 22, 2011 at 3:30 AM, slookin sloo...@gmail.com wrote: I trying to develop flexible server for reciving message, i'm not able to create java classes for each message type, but I can upload proto descripter on server. Of course client (sender) will use generated java classes for prepare messages: My code (sender): Person.Builder person = Person.newBuilder(); person.setId(Integer.valueOf(42)); person.setEmail(test_em...@gmail.com); person.setName(Viktor Villari); Person p=person.build(); FileOutputStream fstream = new FileOutputStream (message.pf); CodedOutputStream outSream = CodedOutputStream.newInstance(fstream); p.writeTo(outSream); outSream.flush(); System.out.println(sent); Reciver (address.descriptor.proto - grenerated via descriptor.proto): FileInputStream input = new FileInputStream (address.descriptor.proto); DescriptorProtos.FileDescriptorSet fdsProto=DescriptorProtos.FileDescriptorSet.parseFrom(input); input = new FileInputStream (message.pf); // System.out.println(fds.getFile(0).getMessageType(0).getName()); System.out.println(File name = +fdsProto.getFile(0).getName()); // System.out.println(Message type = +fdsProto.getFile(0).getMessageType(0).getName()); // System.out.println(Field info = +fdsProto.getFile(0).getMessageType(0).getField(0).getName()+ +fdsProto.getFile(0).getMessageType(0).getField(0).getType()); FileDescriptor fileDescr=FileDescriptor.buildFrom(fdsProto.getFile(0), new FileDescriptor[0]); System.out.println(Message type = +fileDescr.getMessageTypes().get(0).getName()); System.out.println(Field info = +fileDescr.getMessageTypes().get(0).getFields().get(0).getName()+ type= +fileDescr.getMessageTypes().get(0).getFields().get(0).getType()); DynamicMessage dm=DynamicMessage.parseFrom(fdsProto.getDescriptor(), input); On this line, you are passing the descriptor for the FileDescriptorSet. You should be using fileDescr to get at the Person message type. (Note that your description scheme does not indicate what message type is actually used. But I also don't see the code snippet that serializes the FileDescriptorSet, or for that matter delimit between the FileDescriptorSet and the serialized Person data, so maybe you are doing something elsewhere) System.out.println(DynamicMessage to string \n+dm.toString()); // problem line System.out.println(Person.name = + dm.getField(fileDescr.getMessageTypes().get(0).getFields().get(0))); // Output: File name = addressbook.proto Message type = Person Field info = name type= STRING DynamicMessage to string 2: Viktor Villari 3: 42 4: test_em...@gmail.com Exception in thread main java.lang.IllegalArgumentException: FieldDescriptor does not match message type. at com.google.protobuf.DynamicMessage.verifyContainingType(DynamicMessage.java: 242) at com.google.protobuf.DynamicMessage.getField(DynamicMessage.java: 160) at net.lookin.protobuf.Test.main(Test.java:64) Why I recive exception? How I can access to specific field value in DynamicMessage? -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to
Re: [protobuf] Re: Self describing messages
On Mon, Jul 26, 2010 at 7:54 AM, David dudeh...@yahoo.com wrote: I don't want to send the whole enchilada on the wire - just the descriptor for the message being sent. The whole enchilada is needed if your message refers to any other message types. If not, you can just build a dummy FileDescriptorProto containing your DescriptorProto on the receiving end, then call FileDescriptor.buildFrom() on that. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Self describing messages
I don't want to send the whole enchilada on the wire - just the descriptor for the message being sent. As I showed above, I know how to get the descriptor on the wire, but the other side (getting the descriptor off the wire and parsing the message using it) is unclear to me. On Jul 24, 2:29 pm, Christopher Smith cbsm...@gmail.com wrote: You probably need a FileDescriptorSet, which has the whole enchilada. --Chris On Jul 23, 2010, at 12:08 PM, David dudeh...@yahoo.com wrote: I'm trying to send a self-describing message from a JMS producer to a consumer. On the producer I have something like: Descriptor desc = builder.getDescriptorForType(); DescriptorProto proto = desc.toProto(); Then I send the bytes of the proto and the bytes of the built object (from builder). On the consumer side, I can't figure out how to take the DescriptorProto I get on the wire and make a Descriptor out of it which I can use to parse the bytes of the object using DynamicMessage. I've seen one mention of using FileDescriptorProto, but I'm not working with those... just DescriptorProto. Can someone point me in the right direction? Thanks, -David- -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Self describing messages
I don't want to send the whole enchilada on the wire - just the
descriptor for the message being sent.
As I showed above, I know how to get the descriptor on the wire, but
the other side (getting the descriptor off the wire and parsing the
message using it) is unclear to me.
You'll need to bundle it up into a FileDescriptorProto (on the
receiving side), which is the only unit at which the system can turn
descriptors into actual messages. The code looks something like this:
DescriptorProto proto = whatever;
FileDescriptorProto fileProto =
FileDescriptorProto.newBuilder().addMessageType(proto).build();
FileDescriptor file = FileDescriptor.buildFrom(fileProto, new
FileDescriptor[] {});
Descriptor descriptor = file.getMessageTypes().get(0);
Note that this won't work if you use submessages, because submessage
references in DescriptorProtos include their package, and this way of
doing it doesn't specify a package, so it won't be able to find the
submessage definition (unless you're using the default package, of
course). To fix that, you can either pass the package name along with
the DescriptorProto, or you can munge the DescriptorProto on the
sending side to include the package in its name, resulting in code
like the following.
DescriptorProto proto = whatever;
String name = proto.getName();
int dotIndex = name.lastIndexOf('.');
String protoName = name.substring(dotIndex + 1);
String packageName = (dotIndex == -1) ? : name.substring(0, dotIndex);
DescriptorProto modified =
DescriptorProto.newBuilder(proto).setName(protoName).build();
FileDescriptorProto fileProto =
FileDescriptorProto.newBuilder().addMessageType(modified).setPackage(packageName).build();
FileDescriptor file = FileDescriptor.buildFrom(fileProto, new
FileDescriptor[] {});
Descriptor descriptor = file.getMessageTypes().get(0);
Once you have the descriptor, you just need you use one of the
DynamicMessage.parseFrom() methods to parse the message.
- Adam
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
