[protobuf] Re: Issue 549 in protobuf: Cannot install to anywhere except /usr/local/lib
Comment #2 on issue 549 by arglanir: Cannot install to anywhere except /usr/local/lib https://code.google.com/p/protobuf/issues/detail?id=549 I have found the missing option at https://www.mail-archive.com/protobuf@googlegroups.com/msg02998.html. So if someone needs it, you should use: ./configure --prefix=/home/sliverdragon37 --disable-shared -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Hi Oliver, Thanks for the response! I guess my question wasn't quite clear. In my java code I have a string which contains the content of a .proto file. Given this string, how can I create an instance of a Descriptor class so I can do DynamicMessage parsing. Thanks! - Pradeep On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote: On 30 October 2014 02:53, Pradeep Gollakota prade...@gmail.com javascript: wrote: I have a use case where I need to parse messages without having the corresponding precompiled classes in Java. So the DynamicMessage seems to be the correct fit, but I'm not sure how I can generate the DescriptorSet from the .proto definition. protoc --descriptor_set_out=FILE ? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Nano protobuf for android
I can not find the protobuf build tool for android (nano protobuf). Any one has exprienced with this please help me, thanks! -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Issue with data loss when using CodedOutputStream
Hello, please help with the following problem.
I use CodedOutputStream to write a message's length and message itself to a
file.
When I later read from generated file, it turns out that file contains less
data.
I provide source code for simple test: write 1000 messages to a file and
then read this file.
Output is the following:
[aagenosov@black build]$ bin/test_coded_stream
written 1000 records, total bytes: 16384
test successfull
total length: 13890, consumed bytes: 13890, messages count: 594
I tried to forcedly flush buffers, but there was no effect.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
#include iostream
#include sstream
#include fstream
#include string
#include memory
#include fcntl.h
#include errno.h
#include google/protobuf/io/coded_stream.h
#include google/protobuf/io/zero_copy_stream_impl.h
static const uint32_t records_limit = 1000;
int main()
{
using namespace google::protobuf::io;
const char * file_path = /tmp/coded_stream_test.log;
{
int file_handle = open(file_path,
O_WRONLY | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH
);
if (file_handle == -1)
{
std::cerr can't create file file_path , errno: errno std::endl;
return -1;
}
std::auto_ptrFileOutputStream file_stream;
std::auto_ptrCodedOutputStream coded_output;
file_stream.reset(new FileOutputStream(file_handle));
file_stream-SetCloseOnDelete(true);
coded_output.reset(new CodedOutputStream(file_stream.get()));
uint32_t idx = 0;
for (; idx records_limit; ++idx)
{
std::ostringstream data;
data iteration idx;
const std::string str = data.str();
int coded_size = CodedOutputStream::VarintSize32(str.length()) + str.length();
uint8_t * buffer = coded_output-GetDirectBufferForNBytesAndAdvance(coded_size);
if (buffer != NULL)
{
buffer = CodedOutputStream::WriteVarint32ToArray(str.length(), buffer);
buffer = CodedOutputStream::WriteRawToArray(str.c_str(), str.length(), buffer);
}
else
{
coded_output-WriteVarint32(str.length());
coded_output-WriteString(str);
}
if (coded_output-HadError())
{
std::cerr error on iteration idx std::endl;
break;
}
}
/*coded_output.reset();
if (!file_stream-Close())
{
std::cerr error file_stream-GetErrno() on close file std::endl;
}*/
std::cout written idx records, total bytes: file_stream-ByteCount() std::endl;
}
std::fstream input_file(file_path, std::ios_base::in | std::ios_base::binary);
input_file.seekg(0, input_file.end);
size_t total_length = input_file.tellg();
input_file.seekg(0, input_file.beg);
std::auto_ptrZeroCopyInputStream raw_input(new IstreamInputStream(input_file));
std::auto_ptrCodedInputStream coded_input(new CodedInputStream(raw_input.get()));
uint32_t message_count = 0;
int error = 0;
while (!coded_input-ExpectAtEnd() static_castsize_t(raw_input-ByteCount()) != total_length)
{
uint32_t msg_len;
if (!coded_input-ReadVarint32(msg_len))
{
std::cerr failed to get length of next message std::endl;
error = -1;
break;
}
CodedInputStream::Limit limit = coded_input-PushLimit(msg_len);
std::string buf;
if (!coded_input-ReadString(buf, msg_len))
{
std::cerr failed to parse message std::endl;
error = -1;
break;
}
message_count++;
//std::cout buf std::endl;
coded_input-PopLimit(limit);
}
if (error == 0)
{
std::cout test successfull std::endl;
}
std::cout total length: total_length
, consumed bytes: raw_input-ByteCount()
, messages count: message_count
std::endl;
input_file.close();
return error;
}
Re: [protobuf] Parse a .proto file
Basically, you can't do that in pure Java - the compiler is a C++ binary, there is no Java version. Still, working with the output of --descriptor_set_out is probably the way to go here. If you have the .proto file ahead of time, you can pregenerate the descriptor output at build time and store it instead of the .proto file. If you don't have the .proto file ahead of time (and you can't redesign - this is not a good design) then you could run the compiler at runtime and read the output. Either way, now you have a parsed version of the message format as a protobuf-encoded message that you can read into your Java program and extract the Descriptors you need. If you're looking at a selfdescribing message format, then I'd go with using the parsed descriptors as your format description, not the text .proto file. Oliver On 31 October 2014 17:56, Pradeep Gollakota pradeep...@gmail.com wrote: Hi Oliver, Thanks for the response! I guess my question wasn't quite clear. In my java code I have a string which contains the content of a .proto file. Given this string, how can I create an instance of a Descriptor class so I can do DynamicMessage parsing. Thanks! - Pradeep On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote: On 30 October 2014 02:53, Pradeep Gollakota prade...@gmail.com wrote: I have a use case where I need to parse messages without having the corresponding precompiled classes in Java. So the DynamicMessage seems to be the correct fit, but I'm not sure how I can generate the DescriptorSet from the .proto definition. protoc --descriptor_set_out=FILE ? Oliver -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Ok… awesome… I do have the .proto’s ahead of time, so I can have them
compiled to the .desc files and store those.
Here’s my .proto file:
package com.lithum.pbnj;
import google/protobuf/descriptor.proto;
option java_package = com.lithium.pbnj;
extend google.protobuf.FieldOptions {
optional bool isPii = 50101;
}
message MessagePublish {
required string uuid = 1;
required int64 timestamp = 2;
required int64 message_uid = 3;
required string message_content = 4;
required int64 message_author_uid = 5;
optional string email = 6 [(isPii) = true];
}
I compiled this .proto file into a .desc file using the command you gave
me. I’m now trying to parse a DynamicMessage from the .desc file. Here’s
the code I have so far.
DescriptorProtos.FileDescriptorSet descriptorSet =
DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream(/messages.desc));
Descriptors.Descriptor desc =
descriptorSet.getFile(0).getDescriptorForType();
Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail(he...@example.com)
.setMessageAuthorUid(1)
.setMessageContent(hello world!)
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
event.toByteArray());
The final line in the above code is throwing the following exception:
Exception in thread main com.google.protobuf.InvalidProtocolBufferException:
Protocol message end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
at
com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
at
com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:171)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:412)
at com.google.protobuf.DynamicMessage.parseFrom(DynamicMessage.java:119)
at com.lithium.pbnj.PBnJ.main(PBnJ.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
On Friday, October 31, 2014 11:39:27 AM UTC-7, Oliver wrote:
Basically, you can't do that in pure Java - the compiler is a C++
binary, there is no Java version.
Still, working with the output of --descriptor_set_out is probably the
way to go here. If you have the .proto file ahead of time, you can
pregenerate the descriptor output at build time and store it instead
of the .proto file. If you don't have the .proto file ahead of time
(and you can't redesign - this is not a good design) then you could
run the compiler at runtime and read the output. Either way, now you
have a parsed version of the message format as a protobuf-encoded
message that you can read into your Java program and extract the
Descriptors you need.
If you're looking at a selfdescribing message format, then I'd go with
using the parsed descriptors as your format description, not the text
.proto file.
Oliver
On 31 October 2014 17:56, Pradeep Gollakota prade...@gmail.com
javascript: wrote:
Hi Oliver,
Thanks for the response! I guess my question wasn't quite clear. In my
java
code I have a string which contains the content of a .proto file. Given
this
string, how can I create an instance of a Descriptor class so I can do
DynamicMessage parsing.
Thanks!
- Pradeep
On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote:
On 30 October 2014 02:53, Pradeep Gollakota prade...@gmail.com
wrote:
I have a use case where I need to parse messages without having the
corresponding precompiled classes in Java. So the DynamicMessage
Re: [protobuf] Parse a .proto file
At no point are you specifying that you want to use the
MessagePublish descriptor, so you must still be using the API
incorrectly...
On Fri, Oct 31, 2014 at 5:10 PM, Pradeep Gollakota pradeep...@gmail.com wrote:
Ok… awesome… I do have the .proto’s ahead of time, so I can have them
compiled to the .desc files and store those.
Here’s my .proto file:
package com.lithum.pbnj;
import google/protobuf/descriptor.proto;
option java_package = com.lithium.pbnj;
extend google.protobuf.FieldOptions {
optional bool isPii = 50101;
}
message MessagePublish {
required string uuid = 1;
required int64 timestamp = 2;
required int64 message_uid = 3;
required string message_content = 4;
required int64 message_author_uid = 5;
optional string email = 6 [(isPii) = true];
}
I compiled this .proto file into a .desc file using the command you gave me.
I’m now trying to parse a DynamicMessage from the .desc file. Here’s the
code I have so far.
DescriptorProtos.FileDescriptorSet descriptorSet =
DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream(/messages.desc));
Descriptors.Descriptor desc =
descriptorSet.getFile(0).getDescriptorForType();
Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail(he...@example.com)
.setMessageAuthorUid(1)
.setMessageContent(hello world!)
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
event.toByteArray());
The final line in the above code is throwing the following exception:
Exception in thread main
com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
at
com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
at
com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:171)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:412)
at com.google.protobuf.DynamicMessage.parseFrom(DynamicMessage.java:119)
at com.lithium.pbnj.PBnJ.main(PBnJ.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
On Friday, October 31, 2014 11:39:27 AM UTC-7, Oliver wrote:
Basically, you can't do that in pure Java - the compiler is a C++
binary, there is no Java version.
Still, working with the output of --descriptor_set_out is probably the
way to go here. If you have the .proto file ahead of time, you can
pregenerate the descriptor output at build time and store it instead
of the .proto file. If you don't have the .proto file ahead of time
(and you can't redesign - this is not a good design) then you could
run the compiler at runtime and read the output. Either way, now you
have a parsed version of the message format as a protobuf-encoded
message that you can read into your Java program and extract the
Descriptors you need.
If you're looking at a selfdescribing message format, then I'd go with
using the parsed descriptors as your format description, not the text
.proto file.
Oliver
On 31 October 2014 17:56, Pradeep Gollakota prade...@gmail.com wrote:
Hi Oliver,
Thanks for the response! I guess my question wasn't quite clear. In my
java
code I have a string which contains the content of a .proto file. Given
this
string, how can I create an instance of a Descriptor class so I can do
DynamicMessage parsing.
Thanks!
- Pradeep
On Thursday, October 30, 2014 2:41:19 PM UTC-7, Oliver wrote:
Re: [protobuf] Parse a .proto file
Ok… so I was finally able to parse a dynamic message and it looks good. It
looks like it was just a user error on my part… after a little bit of
digging around, I found the right APIs to call. Now my code looks like:
Descriptors.FileDescriptor fieldOptionsDesc =
DescriptorProtos.FieldOptions.getDescriptor().getFile();
DescriptorProtos.FileDescriptorSet set =
DescriptorProtos.FileDescriptorSet.parseFrom(
PBnJ.class.getResourceAsStream(/messages.desc));
Descriptors.Descriptor md =
Descriptors.FileDescriptor.buildFrom(set.getFile(0),
new
Descriptors.FileDescriptor[]{fieldOptionsDesc}).findMessageTypeByName(MessagePublish);
Messages.MessagePublish event = Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail(he...@example.com)
.setMessageAuthorUid(1)
.setMessageContent(hello world!)
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(md,
event.toByteArray());
// Parse worked!
for (Descriptors.FieldDescriptor fieldDescriptor : md.getFields()) {
Boolean extension =
fieldDescriptor.getOptions().getExtension(Messages.isPii);
System.out.println(fieldDescriptor.getName() + isPii = +
extension);
}
The output is:
uuid isPii = false
timestamp isPii = false
message_uid isPii = false
message_content isPii = false
message_author_uid isPii = false
email isPii = false
For some reason, this is incorrectly showing “isPii = false” for the email
field when it should be “isPii = true” (as it is in the .proto file). Any
thoughts on this?
Thanks again all!
On Friday, October 31, 2014 2:18:44 PM UTC-7, Ilia Mirkin wrote:
At no point are you specifying that you want to use the
MessagePublish descriptor, so you must still be using the API
incorrectly...
On Fri, Oct 31, 2014 at 5:10 PM, Pradeep Gollakota prade...@gmail.com
javascript: wrote:
Ok… awesome… I do have the .proto’s ahead of time, so I can have them
compiled to the .desc files and store those.
Here’s my .proto file:
package com.lithum.pbnj;
import google/protobuf/descriptor.proto;
option java_package = com.lithium.pbnj;
extend google.protobuf.FieldOptions {
optional bool isPii = 50101;
}
message MessagePublish {
required string uuid = 1;
required int64 timestamp = 2;
required int64 message_uid = 3;
required string message_content = 4;
required int64 message_author_uid = 5;
optional string email = 6 [(isPii) = true];
}
I compiled this .proto file into a .desc file using the command you gave
me.
I’m now trying to parse a DynamicMessage from the .desc file. Here’s the
code I have so far.
DescriptorProtos.FileDescriptorSet descriptorSet =
DescriptorProtos.FileDescriptorSet.parseFrom(PBnJ.class.getResourceAsStream(/messages.desc));
Descriptors.Descriptor desc =
descriptorSet.getFile(0).getDescriptorForType();
Messages.MessagePublish event =
Messages.MessagePublish.newBuilder()
.setUuid(UUID.randomUUID().toString())
.setTimestamp(System.currentTimeMillis())
.setEmail(he...@example.com javascript:)
.setMessageAuthorUid(1)
.setMessageContent(hello world!)
.setMessageUid(1)
.build();
DynamicMessage dynamicMessage = DynamicMessage.parseFrom(desc,
event.toByteArray());
The final line in the above code is throwing the following exception:
Exception in thread main
com.google.protobuf.InvalidProtocolBufferException: Protocol message
end-group tag did not match expected tag.
at
com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at
com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:174)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:478)
at
com.google.protobuf.MessageReflection$BuilderAdapter.parseMessage(MessageReflection.java:482)
at
com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:780)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:336)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:318)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:229)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:180)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:419)
at
Re: [protobuf] Parse a .proto file
On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota pradeep...@gmail.com wrote: Boolean extension = fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Not really... one of the use cases I'm trying to solve for is an anonymization use case. We will have several app's writing protobuf records and the data will pass through an anonymization layer. The anonymizer inspects the schema's for all incoming data and will transform the pii fields. Since I will be defining the custom options that will be used by the app dev's, I will have precompiled classes available for reference just like the code shows. So what I'm trying to figure out is, using the DynamicMessage API and having parsed a Descriptor, how do I find all the fields which have been annotated with the (isPii = true) option. On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota prade...@gmail.com javascript: wrote: Boolean extension = fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
You may be running into issues where the set of descriptors associated with your parsed DynamicMessage (i.e. the ones you parsed at runtime) do not match the set of descriptors from your pregenerated code (which will be using their own descriptor pool). IIRC they're looked up by identity, so even if they have the same structure they won't match if loaded separately. It's a bit of a wart in the API - I'm not sure what the right way to do this is, if you try to mix pregenerated code dynamically loaded descriptors, all sorts of things break. Oliver On 1 November 2014 00:48, Pradeep Gollakota pradeep...@gmail.com wrote: Not really... one of the use cases I'm trying to solve for is an anonymization use case. We will have several app's writing protobuf records and the data will pass through an anonymization layer. The anonymizer inspects the schema's for all incoming data and will transform the pii fields. Since I will be defining the custom options that will be used by the app dev's, I will have precompiled classes available for reference just like the code shows. So what I'm trying to figure out is, using the DynamicMessage API and having parsed a Descriptor, how do I find all the fields which have been annotated with the (isPii = true) option. On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota prade...@gmail.com wrote: Boolean extension = fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Parse a .proto file
Confirmed... When I replaced the md variable with the compiled Descriptor, it worked. I didn't think I was mixing the descriptors, e.g. the MessagePublish message is one that is produced via the compiled API and parsed using the DynamicMessage API. The isPii extension has been refactored into a separate proto that is precompiled into my codebase. I.E. the descriptor for MessagePublish should be loaded dynamically and the descriptor for the FieldOption I'm defining won't be loaded dynamically. As far as I can tell, there shouldn't be any mixing of the descriptor pools, though I may be wrong. Any thoughts on how I can proceed with this project? On Fri Oct 31 2014 at 7:01:55 PM Oliver Jowett oliver.jow...@gmail.com wrote: You may be running into issues where the set of descriptors associated with your parsed DynamicMessage (i.e. the ones you parsed at runtime) do not match the set of descriptors from your pregenerated code (which will be using their own descriptor pool). IIRC they're looked up by identity, so even if they have the same structure they won't match if loaded separately. It's a bit of a wart in the API - I'm not sure what the right way to do this is, if you try to mix pregenerated code dynamically loaded descriptors, all sorts of things break. Oliver On 1 November 2014 00:48, Pradeep Gollakota pradeep...@gmail.com wrote: Not really... one of the use cases I'm trying to solve for is an anonymization use case. We will have several app's writing protobuf records and the data will pass through an anonymization layer. The anonymizer inspects the schema's for all incoming data and will transform the pii fields. Since I will be defining the custom options that will be used by the app dev's, I will have precompiled classes available for reference just like the code shows. So what I'm trying to figure out is, using the DynamicMessage API and having parsed a Descriptor, how do I find all the fields which have been annotated with the (isPii = true) option. On Friday, October 31, 2014 3:25:51 PM UTC-7, Ilia Mirkin wrote: On Fri, Oct 31, 2014 at 6:18 PM, Pradeep Gollakota prade...@gmail.com wrote: Boolean extension = fieldDescriptor.getOptions().getExtension(Messages.isPii); Shouldn't this use some sort of API that doesn't use the Messages class at all? -ilia -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
