Re: [protobuf] Parse a .proto file
Ok… I finally figured out the work around for this. I use a separate .proto
file that contains my custom options.
package com.lithum.pbnj;
import "google/protobuf/descriptor.proto";
option java_package = "com.lithium.pbnj";
message LiOptions {
optional bool isPii = 1 [default = false];
optional bool isEmail = 2 [default = false];
optional bool isIpAddress = 3 [default = false];
}
extend google.protobuf.FieldOptions {
optional LiOptions li_opts = 50101;
}
Then I compile this .proto into a .java and can use it. When a message uses
this extension, I can figure out which fields use my options, I use the
following code:
Descriptors.FileDescriptor fieldOptionsDesc =
DescriptorProtos.FieldOptions.getDescriptor().getFile();
Descriptors.FileDescriptor extensionsDesc =
Extensions.getDescriptor().getFile();
Descriptors.FileDescriptor[] files = new
Descriptors.FileDescriptor[]{fieldOptionsDesc, extensionsDesc};
DescriptorProtos.FileDescriptorSet set =
DescriptorProtos.FileDescriptorSet.parseFrom(
PBnJ.class.getResourceAsStream("/messages.desc"));
DescriptorProtos.FileDescriptorProto messages = set.getFile(0);
Descriptors.FileDescriptor fileDesc =
Descriptors.FileDescriptor.buildFrom(messages, files);
Descriptors.Descriptor md =
fileDesc.findMessageTypeByName("MessagePublish");
Set piiFields = Sets.newHashSet();
for (Descriptors.FieldDescriptor fieldDescriptor : md.getFields()) {
DescriptorProtos.FieldOptions options =
fieldDescriptor.getOptions();
UnknownFieldSet.Field field =
options.getUnknownFields().asMap().get(Extensions.LI_OPTS_FIELD_NUMBER);
if (field != null) {
Extensions.LiOptions liOptions =
Extensions.LiOptions.parseFrom(field.getLengthDelimitedList().get(0).toByteArray());
if (liOptions.getIsEmail() || liOptions.getIsIpAddress() ||
liOptions.getIsPii()) {
piiFields.add(fieldDescriptor);
System.out.println(fieldDescriptor.toProto());
}
}
}
On Saturday, November 1, 2014 4:26:35 AM UTC-7, Oliver wrote:
On 1 November 2014 02:24, Pradeep Gollakota > wrote:
> > Confirmed... When I replaced the "md" variable with the compiled
> Descriptor,
> > it worked. I didn't think I was mixing the descriptors, e.g. the
> > MessagePublish message is one that is produced via the compiled API and
> > parsed using the DynamicMessage API. The isPii extension has been
> refactored
> > into a separate proto that is precompiled into my codebase. I.E. the
> > descriptor for MessagePublish should be loaded dynamically and the
> > descriptor for the FieldOption I'm defining won't be loaded dynamically.
> As
> > far as I can tell, there shouldn't be any mixing of the descriptor
> pools,
> > though I may be wrong.
>
> This is exactly where the problem is, though - you have:
>
> MessagePublish descriptor D1 (from encoded descriptorset) references
> extension field F1 (from encoded descriptorset)
> Message descriptor D2 (from pregenerated code) references extension
> field F2 (from pregenerated code)
>
> So if you have a message built from D1 then it thinks it has a field
> F1; when you ask if it has extension F2 it says "no!" even though
> they're really the same thing.
>
> > Any thoughts on how I can proceed with this project?
>
> It seems like a flaw in the API .. In the case I ran into, I could
> work around it as the processing code only wanted to work with
> non-option extensions when it had the precompiled code for the
> extension, so for those well-known message types I'd just look up the
> descriptor to use from the precompiled set rather than using the
> in-stream descriptor set (the message format included the descriptors
> inline). That doesn't really work here though.
>
> Can you inspect the options using field descriptors from the encoded
> descriptorset, rather than using Messages.pii from the pregenerated
> code?
>
> Oliver
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
On 1 November 2014 02:24, Pradeep Gollakota wrote: > Confirmed... When I replaced the "md" variable with the compiled Descriptor, > it worked. I didn't think I was mixing the descriptors, e.g. the > MessagePublish message is one that is produced via the compiled API and > parsed using the DynamicMessage API. The isPii extension has been refactored > into a separate proto that is precompiled into my codebase. I.E. the > descriptor for MessagePublish should be loaded dynamically and the > descriptor for the FieldOption I'm defining won't be loaded dynamically. As > far as I can tell, there shouldn't be any mixing of the descriptor pools, > though I may be wrong. This is exactly where the problem is, though - you have: MessagePublish descriptor D1 (from encoded descriptorset) references extension field F1 (from encoded descriptorset) Message descriptor D2 (from pregenerated code) references extension field F2 (from pregenerated code) So if you have a message built from D1 then it thinks it has a field F1; when you ask if it has extension F2 it says "no!" even though they're really the same thing. > Any thoughts on how I can proceed with this project? It seems like a flaw in the API .. In the case I ran into, I could work around it as the processing code only wanted to work with non-option extensions when it had the precompiled code for the extension, so for those well-known message types I'd just look up the descriptor to use from the precompiled set rather than using the in-stream descriptor set (the message format included the descriptors inline). That doesn't really work here though. Can you inspect the options using field descriptors from the encoded descriptorset, rather than using Messages.pii from the pregenerated code? Oliver -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Confirmed... When I replaced the "md" variable with the compiled Descriptor, it worked. I didn't think I was mixing the descriptors, e.g. the MessagePublish message is one that is produced via the compiled API and parsed using the DynamicMessage API. The isPii extension has been refactored into a separate proto that is precompiled into my codebase. I.E. the descriptor for MessagePublish should be loaded dynamically and the descriptor for the FieldOption I'm defining won't be loaded dynamically. As far as I can tell, there shouldn't be any mixing of the descriptor pools, though I may be wrong. Any thoughts on how I can proceed with this project? On Fri Oct 31 2014 at 7:01:55 PM Oliver Jowett wrote: > You may be running into issues where the set of descriptors associated > with your parsed DynamicMessage (i.e. the ones you parsed at runtime) > do not match the set of descriptors from your pregenerated code (which > will be using their own descriptor pool). IIRC they're looked up by > identity, so even if they have the same structure they won't match if > loaded separately. It's a bit of a wart in the API - I'm not sure what > the right way to do this is, if you try to mix pregenerated code & > dynamically loaded descriptors, all sorts of things break. > > Oliver > > > On 1 November 2014 00:48, Pradeep Gollakota wrote: > > Not really... one of the use cases I'm trying to solve for is an > > anonymization use case. We will have several app's writing protobuf > records > > and the data will pass through an anonymization layer. The anonymizer > > inspects the schema's for all incoming data and will transform the pii > > fields. Since I will be defining the custom options that will be used by > the > > app dev's, I will have precompiled classes available for reference just > like > > the code shows. > > > > So what I'm trying to figure out is, using the DynamicMessage API and > having > > parsed a Descriptor, how do I find all the fields which have been > annotated > > with the (isPii = true) option. > > > > On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: > >> > >> On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota > >> wrote: > >> > Boolean extension = > >> > fieldDescriptor.getOptions().getExtension(Messages.isPii); > >> > >> Shouldn't this use some sort of API that doesn't use the Messages class > at > >> all? > >> > >> -ilia > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Protocol Buffers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to protobuf+unsubscr...@googlegroups.com. > > To post to this group, send email to protobuf@googlegroups.com. > > Visit this group at http://groups.google.com/group/protobuf. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
You may be running into issues where the set of descriptors associated with your parsed DynamicMessage (i.e. the ones you parsed at runtime) do not match the set of descriptors from your pregenerated code (which will be using their own descriptor pool). IIRC they're looked up by identity, so even if they have the same structure they won't match if loaded separately. It's a bit of a wart in the API - I'm not sure what the right way to do this is, if you try to mix pregenerated code & dynamically loaded descriptors, all sorts of things break. Oliver On 1 November 2014 00:48, Pradeep Gollakota wrote: > Not really... one of the use cases I'm trying to solve for is an > anonymization use case. We will have several app's writing protobuf records > and the data will pass through an anonymization layer. The anonymizer > inspects the schema's for all incoming data and will transform the pii > fields. Since I will be defining the custom options that will be used by the > app dev's, I will have precompiled classes available for reference just like > the code shows. > > So what I'm trying to figure out is, using the DynamicMessage API and having > parsed a Descriptor, how do I find all the fields which have been annotated > with the (isPii = true) option. > > On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: >> >> On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota >> wrote: >> > Boolean extension = >> > fieldDescriptor.getOptions().getExtension(Messages.isPii); >> >> Shouldn't this use some sort of API that doesn't use the Messages class at >> all? >> >> -ilia > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Not really... one of the use cases I'm trying to solve for is an anonymization use case. We will have several app's writing protobuf records and the data will pass through an anonymization layer. The anonymizer inspects the schema's for all incoming data and will transform the pii fields. Since I will be defining the custom options that will be used by the app dev's, I will have precompiled classes available for reference just like the code shows. So what I'm trying to figure out is, using the DynamicMessage API and having parsed a Descriptor, how do I find all the fields which have been annotated with the (isPii = true) option. On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: > > On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota > wrote: > > Boolean extension = > > fieldDescriptor.getOptions().getExtension(Messages.isPii); > > Shouldn't this use some sort of API that doesn't use the Messages class at > all? > > -ilia > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota wrote: > Boolean extension = > fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Ok… so I was finally able to parse a dynamic message and it looks good. It
looks like it was just a user error on my part… after a little bit of
digging around, I found the right APIs to call. Now my code looks like:
Descriptors.FileDescriptor fieldOptionsDesc =
DescriptorProtos.FieldOptions.getDescriptor().getFile();
DescriptorProtos.FileDescriptorSet set =
DescriptorProtos.FileDescriptorSet.parseFrom(
PBnJ.class.getResourceAsStream("/messages.desc"));
Descriptors.Descriptor md =
Descriptors.FileDescriptor.buildFrom(set.getFile(0),
new
Descriptors.FileDescriptor[]{fieldOptionsDesc}).findMessageTypeByName("MessagePublish");
Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail("he...@example.com")
.setMessageAuthorUid(1)
.setMessageContent("hello world!")
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(md,
event.toByteArray());
// Parse worked!
for (Descriptors.FieldDescriptor fieldDescriptor : md.getFields()) {
Boolean extension =
fieldDescriptor.getOptions().getExtension(Messages.isPii);
System.out.println(fieldDescriptor.getName() + " isPii = " +
extension);
}
The output is:
uuid isPii = false
timestamp isPii = false
message_uid isPii = false
message_content isPii = false
message_author_uid isPii = false
email isPii = false
For some reason, this is incorrectly showing “isPii = false” for the email
field when it should be “isPii = true” (as it is in the .proto file). Any
thoughts on this?
Thanks again all!
On Friday, October 31, 2014 2:18:44 PM UTC-7, Ilia Mirkin wrote:
At no point are you specifying that you want to use the
> "MessagePublish" descriptor, so you must still be using the API
> incorrectly...
>
> On Fri, Oct 31, 2014 at 5:10 PM, Pradeep Gollakota > wrote:
> > Ok… awesome… I do have the .proto’s ahead of time, so I can have them
> > compiled to the .desc files and store those.
> >
> > Here’s my .proto file:
> >
> > package com.lithum.pbnj;
> >
> > import "google/protobuf/descriptor.proto";
> >
> > option java_package = "com.lithium.pbnj";
> >
> > extend google.protobuf.FieldOptions {
> > optional bool isPii = 50101;
> > }
> >
> > message MessagePublish {
> > required string uuid = 1;
> > required int64 timestamp = 2;
> > required int64 message_uid = 3;
> > required string message_content = 4;
> > required int64 message_author_uid = 5;
> > optional string email = 6 [(isPii) = true];
> > }
> >
> > I compiled this .proto file into a .desc file using the command you gave
> me.
> > I’m now trying to parse a DynamicMessage from the .desc file. Here’s the
> > code I have so far.
> >
> > DescriptorProtos.FileDescriptorSet descriptorSet =
> >
> DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream("/messages.desc"));
>
>
> > Descriptors.Descriptor desc =
> > descriptorSet.getFile(0).getDescriptorForType();
> >
> > Messages.MessagePublish event =
> Messages.MessagePublish.newBuilder()
> > .setUuid(UUID.randomUUID().toString())
> > .setTimestamp(System.currentTimeMillis())
> > .setEmail("he...@example.com ")
> > .setMessageAuthorUid(1)
> > .setMessageContent("hello world!")
> > .setMessageUid(1)
> > .build();
> >
> > DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
> > event.toByteArray());
> >
> > The final line in the above code is throwing the following exception:
> >
> > Exception in thread "main"
> > com.google.protobuf.InvalidProtocolBufferException: Protocol message
> > end-group tag did not match expected tag.
> > at
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
>
>
> > at
> >
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
>
>
> > at
> >
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
> > at
> >
> com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
>
>
> > at
> >
> com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
>
>
> > at
> >
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
>
>
> > at
> >
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
>
>
> > at
> >
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
>
>
> > at
> >
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(Ab
Re: [protobuf] Parse a .proto file
At no point are you specifying that you want to use the
"MessagePublish" descriptor, so you must still be using the API
incorrectly...
On Fri, Oct 31, 2014 at 5:10 PM, Pradeep Gollakota wrote:
> Ok… awesome… I do have the .proto’s ahead of time, so I can have them
> compiled to the .desc files and store those.
>
> Here’s my .proto file:
>
> package com.lithum.pbnj;
>
> import "google/protobuf/descriptor.proto";
>
> option java_package = "com.lithium.pbnj";
>
> extend google.protobuf.FieldOptions {
> optional bool isPii = 50101;
> }
>
> message MessagePublish {
> required string uuid = 1;
> required int64 timestamp = 2;
> required int64 message_uid = 3;
> required string message_content = 4;
> required int64 message_author_uid = 5;
> optional string email = 6 [(isPii) = true];
> }
>
> I compiled this .proto file into a .desc file using the command you gave me.
> I’m now trying to parse a DynamicMessage from the .desc file. Here’s the
> code I have so far.
>
> DescriptorProtos.FileDescriptorSet descriptorSet =
> DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream("/messages.desc"));
> Descriptors.Descriptor desc =
> descriptorSet.getFile(0).getDescriptorForType();
>
> Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
> .setUuid(UUID.randomUUID().toString())
> .setTimestamp(System.currentTimeMillis())
> .setEmail("he...@example.com")
> .setMessageAuthorUid(1)
> .setMessageContent("hello world!")
> .setMessageUid(1)
> .build();
>
> DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
> event.toByteArray());
>
> The final line in the above code is throwing the following exception:
>
> Exception in thread "main"
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag.
> at
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
> at
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
> at
> com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
> at
> com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:171)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:412)
> at com.google.protobuf.DynamicMessage.parseFrom(DynamicMessage.java:119)
> at com.lithium.pbnj.PBnJ.main(PBnJ.java:36)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
> On Friday, October 31, 2014 11:39:27 AM UTC-7, Oliver wrote:
>>
>> Basically, you can't do that in pure Java - the compiler is a C++
>> binary, there is no Java version.
>>
>> Still, working with the output of --descriptor_set_out is probably the
>> way to go here. If you have the .proto file ahead of time, you can
>> pregenerate the descriptor output at build time and store it instead
>> of the .proto file. If you don't have the .proto file ahead of time
>> (and you can't redesign - this is not a good design) then you could
>> run the compiler at runtime and read the output. Either way, now you
>> have a parsed version of the message format as a protobuf-encoded
>> message that you can read into your Java program and extract the
>> Descriptors you need.
>>
>> If you're looking at a selfdescribing message format, then I'd go with
>> using the parsed descriptors as your format description, not the text
>> .proto file.
>>
>> Oliver
>>
>>
>> On 31 October 2014 17:56, Pradeep Gollakota wrote:
>> > Hi Oliver,
>> >
>> > Thanks for the response! I guess my question wasn't quite clear. In my
>> > java
>> > code I have a string which contains the content of a .proto file. Given
>> > this
>> > string, how can I create an instance of a Descriptor class s
Re: [protobuf] Parse a .proto file
Ok… awesome… I do have the .proto’s ahead of time, so I can have them
compiled to the .desc files and store those.
Here’s my .proto file:
package com.lithum.pbnj;
import "google/protobuf/descriptor.proto";
option java_package = "com.lithium.pbnj";
extend google.protobuf.FieldOptions {
optional bool isPii = 50101;
}
message MessagePublish {
required string uuid = 1;
required int64 timestamp = 2;
required int64 message_uid = 3;
required string message_content = 4;
required int64 message_author_uid = 5;
optional string email = 6 [(isPii) = true];
}
I compiled this .proto file into a .desc file using the command you gave
me. I’m now trying to parse a DynamicMessage from the .desc file. Here’s
the code I have so far.
DescriptorProtos.FileDescriptorSet descriptorSet =
DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream("/messages.desc"));
Descriptors.Descriptor desc =
descriptorSet.getFile(0).getDescriptorForType();
Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail("he...@example.com")
.setMessageAuthorUid(1)
.setMessageContent("hello world!")
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
event.toByteArray());
The final line in the above code is throwing the following exception:
Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException:
Protocol message end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
at
com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
at
com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:171)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:412)
at com.google.protobuf.DynamicMessage.parseFrom(DynamicMessage.java:119)
at com.lithium.pbnj.PBnJ.main(PBnJ.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
On Friday, October 31, 2014 11:39:27 AM UTC-7, Oliver wrote:
Basically, you can't do that in pure Java - the compiler is a C++
> binary, there is no Java version.
>
> Still, working with the output of --descriptor_set_out is probably the
> way to go here. If you have the .proto file ahead of time, you can
> pregenerate the descriptor output at build time and store it instead
> of the .proto file. If you don't have the .proto file ahead of time
> (and you can't redesign - this is not a good design) then you could
> run the compiler at runtime and read the output. Either way, now you
> have a parsed version of the message format as a protobuf-encoded
> message that you can read into your Java program and extract the
> Descriptors you need.
>
> If you're looking at a selfdescribing message format, then I'd go with
> using the parsed descriptors as your format description, not the text
> .proto file.
>
> Oliver
>
>
> On 31 October 2014 17:56, Pradeep Gollakota > wrote:
> > Hi Oliver,
> >
> > Thanks for the response! I guess my question wasn't quite clear. In my
> java
> > code I have a string which contains the content of a .proto file. Given
> this
> > string, how can I create an instance of a Descriptor class so I can do
> > DynamicMessage parsing.
> >
> > Thanks!
> > - Pradeep
> >
> > On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote:
> >>
> >> On 30 October 2014 02:53, Pradeep Gollakota
> wrote:
> >>
> >> > I have a use case where I need to parse messages without having the
> >> > corresponding precompiled classes in Java.
Re: [protobuf] Parse a .proto file
Basically, you can't do that in pure Java - the compiler is a C++ binary, there is no Java version. Still, working with the output of --descriptor_set_out is probably the way to go here. If you have the .proto file ahead of time, you can pregenerate the descriptor output at build time and store it instead of the .proto file. If you don't have the .proto file ahead of time (and you can't redesign - this is not a good design) then you could run the compiler at runtime and read the output. Either way, now you have a parsed version of the message format as a protobuf-encoded message that you can read into your Java program and extract the Descriptors you need. If you're looking at a selfdescribing message format, then I'd go with using the parsed descriptors as your format description, not the text .proto file. Oliver On 31 October 2014 17:56, Pradeep Gollakota wrote: > Hi Oliver, > > Thanks for the response! I guess my question wasn't quite clear. In my java > code I have a string which contains the content of a .proto file. Given this > string, how can I create an instance of a Descriptor class so I can do > DynamicMessage parsing. > > Thanks! > - Pradeep > > On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote: >> >> On 30 October 2014 02:53, Pradeep Gollakota wrote: >> >> > I have a use case where I need to parse messages without having the >> > corresponding precompiled classes in Java. So the DynamicMessage seems >> > to be >> > the correct fit, but I'm not sure how I can generate the DescriptorSet >> > from >> > the ".proto" definition. >> >> protoc --descriptor_set_out=FILE ? >> >> Oliver > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Hi Oliver, Thanks for the response! I guess my question wasn't quite clear. In my java code I have a string which contains the content of a .proto file. Given this string, how can I create an instance of a Descriptor class so I can do DynamicMessage parsing. Thanks! - Pradeep On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote: > > On 30 October 2014 02:53, Pradeep Gollakota > wrote: > > > I have a use case where I need to parse messages without having the > > corresponding precompiled classes in Java. So the DynamicMessage seems > to be > > the correct fit, but I'm not sure how I can generate the DescriptorSet > from > > the ".proto" definition. > > protoc --descriptor_set_out=FILE ? > > Oliver > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
On 30 October 2014 02:53, Pradeep Gollakota wrote: > I have a use case where I need to parse messages without having the > corresponding precompiled classes in Java. So the DynamicMessage seems to be > the correct fit, but I'm not sure how I can generate the DescriptorSet from > the ".proto" definition. protoc --descriptor_set_out=FILE ? Oliver -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Parse a .proto file
Hi Protobuf gurus, I'm trying to parse a .proto file in Java to use with DynamicMessages. Is this possible or does it have to be compiled to a "descriptor set" file first before this can be done? I have a use case where I need to parse messages without having the corresponding precompiled classes in Java. So the DynamicMessage seems to be the correct fit, but I'm not sure how I can generate the DescriptorSet from the ".proto" definition. Thanks in advance, Pradeep -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
