subject:"Re\: \[protobuf\] Dynamic\/run\-time decoding"

Re: [protobuf] Dynamic/run-time decoding

2016-05-18 Thread osuciu

At least in Java, you can do everything fully dynamically at runtime. The 
native API can get quite confusing (between all the 
protos/descriptors/files...). I distilled following wrapper to make it 
easier:
https://github.com/os72/protobuf-dynamic


On Friday, April 1, 2016 at 4:32:29 PM UTC-7, Josh Haberman wrote:
>
> Hi Mike,
>
> Python indeed has descriptor classes that you can use to build proto 
> definitions at runtime. However it does not have a parser for .proto files 
> built in. So if you want to load .proto files at runtime, you'll need to 
> use protoc to build a descriptor (for example, you could shell out to 
> protoc).
>
> Josh
>
> On Monday, March 21, 2016 at 10:43:24 AM UTC-7, Mike Trienis wrote:
>>
>> Hi,
>>
>> Do you know if it's possible do dynamic decoding at run-time using the 
>> Python API? 
>>
>>- 
>>https://developers.google.com/protocol-buffers/docs/reference/python/
>>
>> Indeed there is a descriptor.FileDescriptor, however I'm not sure what 
>> the other classes correspond to?
>>
>> Thanks, Mike. 
>>
>> On Tuesday, August 12, 2014 at 10:36:13 AM UTC-7, Feng Xiao wrote:
>>>
>>> Protobuf supports creating message types dynamically at runtime and use 
>>> them for parsing/serialization/etc.
>>>
>>> First you need to build up a DescriptorPool 
>>> 
>>>  
>>> that contains all types that you may want to use. There are two approaches 
>>> to construct this pool. One is to call DescriptorPool::BuildFile() directly 
>>> with parsed proto files. For example:
>>>   // Convert .proto files into parsed FileDescriptorProto
>>>   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
>>> FileInputStream stream(filename);
>>> google::protobuf::io::Tokenizer tokenizer();
>>> google::protobuf::compiler::Parser parser;
>>> return parser.Parse(, result);
>>>   }
>>>   // Build the descriptor pool
>>>   DescriptorPool pool;
>>>   for (string filename : proto_files) {
>>> FileDescriptorProto proto;
>>> ParseProtoFile(filename, );
>>> pool.BuildFile(proto);
>>>   }
>>>
>>> After you have the pool, you can query for a type by its name. For 
>>> example, DescriptorPool::FindMessageTypeByName().
>>>
>>> Then to actually parse/serialize/use message types in the pool, you need 
>>> to construct message objects around them. DynamicMessage 
>>> 
>>>  
>>> is used for that:
>>>   // Suppose you want to parse a message type with a specific type name.
>>>   Descriptor* descriptor = 
>>> pool.FindMessageTypeByName(message_type_to_parse);
>>>   DynamicMessageFactory factory;
>>>   unique_ptr message = factory.GetPrototype(descriptor)->New();
>>>   // Use the message object for parsing/etc.
>>>   message->ParseFromString(input_data);
>>>   // Access a specific field in the message
>>>   FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);
>>>   switch (field->type()) {
>>> case TYPE_INT32: message->GetReflection()->GetInt32(*message, 
>>> field); break;
>>> ...
>>>   }
>>>
>>> On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský  
>>> wrote:
>>>
 Hi,

 I have very specific problem. I have data and proto file available and 
 my application should take both and based on external configuration 
 determine how to interpret data (many different types/messages in proto). 
 Yet that can be determine only during run. My question is if there is any 
 support for that, I mean that I will be able to parse proto and decode 
 data 
 based on content of interpret intermediate structures.

 I have been trying to analyze this possibility directly from codes but 
 not with much success. I would be glad for any guidance.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2016-04-01 Thread 'Josh Haberman' via Protocol Buffers

Hi Mike,

Python indeed has descriptor classes that you can use to build proto 
definitions at runtime. However it does not have a parser for .proto files 
built in. So if you want to load .proto files at runtime, you'll need to 
use protoc to build a descriptor (for example, you could shell out to 
protoc).

Josh

On Monday, March 21, 2016 at 10:43:24 AM UTC-7, Mike Trienis wrote:
>
> Hi,
>
> Do you know if it's possible do dynamic decoding at run-time using the 
> Python API? 
>
>- https://developers.google.com/protocol-buffers/docs/reference/python/
>
> Indeed there is a descriptor.FileDescriptor, however I'm not sure what the 
> other classes correspond to?
>
> Thanks, Mike. 
>
> On Tuesday, August 12, 2014 at 10:36:13 AM UTC-7, Feng Xiao wrote:
>>
>> Protobuf supports creating message types dynamically at runtime and use 
>> them for parsing/serialization/etc.
>>
>> First you need to build up a DescriptorPool 
>> 
>>  
>> that contains all types that you may want to use. There are two approaches 
>> to construct this pool. One is to call DescriptorPool::BuildFile() directly 
>> with parsed proto files. For example:
>>   // Convert .proto files into parsed FileDescriptorProto
>>   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
>> FileInputStream stream(filename);
>> google::protobuf::io::Tokenizer tokenizer();
>> google::protobuf::compiler::Parser parser;
>> return parser.Parse(, result);
>>   }
>>   // Build the descriptor pool
>>   DescriptorPool pool;
>>   for (string filename : proto_files) {
>> FileDescriptorProto proto;
>> ParseProtoFile(filename, );
>> pool.BuildFile(proto);
>>   }
>>
>> After you have the pool, you can query for a type by its name. For 
>> example, DescriptorPool::FindMessageTypeByName().
>>
>> Then to actually parse/serialize/use message types in the pool, you need 
>> to construct message objects around them. DynamicMessage 
>> 
>>  
>> is used for that:
>>   // Suppose you want to parse a message type with a specific type name.
>>   Descriptor* descriptor = 
>> pool.FindMessageTypeByName(message_type_to_parse);
>>   DynamicMessageFactory factory;
>>   unique_ptr message = factory.GetPrototype(descriptor)->New();
>>   // Use the message object for parsing/etc.
>>   message->ParseFromString(input_data);
>>   // Access a specific field in the message
>>   FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);
>>   switch (field->type()) {
>> case TYPE_INT32: message->GetReflection()->GetInt32(*message, field); 
>> break;
>> ...
>>   }
>>
>> On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský  
>> wrote:
>>
>>> Hi,
>>>
>>> I have very specific problem. I have data and proto file available and 
>>> my application should take both and based on external configuration 
>>> determine how to interpret data (many different types/messages in proto). 
>>> Yet that can be determine only during run. My question is if there is any 
>>> support for that, I mean that I will be able to parse proto and decode data 
>>> based on content of interpret intermediate structures.
>>>
>>> I have been trying to analyze this possibility directly from codes but 
>>> not with much success. I would be glad for any guidance.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2016-03-21 Thread Mike Trienis

Hi,

Do you know if it's possible do dynamic decoding at run-time using the 
Python API? 

   - https://developers.google.com/protocol-buffers/docs/reference/python/
   
Indeed there is a descriptor.FileDescriptor, however I'm not sure what the 
other classes correspond to?

Thanks, Mike. 

On Tuesday, August 12, 2014 at 10:36:13 AM UTC-7, Feng Xiao wrote:
>
> Protobuf supports creating message types dynamically at runtime and use 
> them for parsing/serialization/etc.
>
> First you need to build up a DescriptorPool 
> 
>  
> that contains all types that you may want to use. There are two approaches 
> to construct this pool. One is to call DescriptorPool::BuildFile() directly 
> with parsed proto files. For example:
>   // Convert .proto files into parsed FileDescriptorProto
>   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
> FileInputStream stream(filename);
> google::protobuf::io::Tokenizer tokenizer();
> google::protobuf::compiler::Parser parser;
> return parser.Parse(, result);
>   }
>   // Build the descriptor pool
>   DescriptorPool pool;
>   for (string filename : proto_files) {
> FileDescriptorProto proto;
> ParseProtoFile(filename, );
> pool.BuildFile(proto);
>   }
>
> After you have the pool, you can query for a type by its name. For 
> example, DescriptorPool::FindMessageTypeByName().
>
> Then to actually parse/serialize/use message types in the pool, you need 
> to construct message objects around them. DynamicMessage 
> 
>  
> is used for that:
>   // Suppose you want to parse a message type with a specific type name.
>   Descriptor* descriptor = 
> pool.FindMessageTypeByName(message_type_to_parse);
>   DynamicMessageFactory factory;
>   unique_ptr message = factory.GetPrototype(descriptor)->New();
>   // Use the message object for parsing/etc.
>   message->ParseFromString(input_data);
>   // Access a specific field in the message
>   FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);
>   switch (field->type()) {
> case TYPE_INT32: message->GetReflection()->GetInt32(*message, field); 
> break;
> ...
>   }
>
> On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský  > wrote:
>
>> Hi,
>>
>> I have very specific problem. I have data and proto file available and my 
>> application should take both and based on external configuration determine 
>> how to interpret data (many different types/messages in proto). Yet that 
>> can be determine only during run. My question is if there is any support 
>> for that, I mean that I will be able to parse proto and decode data based 
>> on content of interpret intermediate structures.
>>
>> I have been trying to analyze this possibility directly from codes but 
>> not with much success. I would be glad for any guidance.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-07-20 Thread Jan Kyjovský

Hello,

have you any luck finding where there may be problem?

On Monday, 15 June 2015 08:27:30 UTC+2, Jan Kyjovský wrote:

 It has grown a bit since I started so I will attach zipped whole project.

 On Friday, 12 June 2015 08:47:59 UTC+2, 肖锋 wrote:



 On Wednesday, June 10, 2015 at 9:49:03 PM UTC-7, Jan Kyjovský wrote:

 I did as you told me. And tried to compile my proto files with protoc. 
 Well it eat them up without any problem so I guess that clear that there is 
 no problem with them but in that case why is it causing such error to me.

 As I gave some hint before problem is happening when I am trying to do 
 following with phone.proto which is being added as second into descriptor 
 database

 compiler::Parser parser;

 parser.RecordErrorsTo(errCollector);
 bool bV = parser.Parse(tokenizer, result);
 std::string strErr = errCollector.GetErrorAndWarnings();

 return bV;

 Parse returns false and error collector holds following error:
 1:1: ERROR: Multiple package definitions.

 Do you have nay idea what that could meant?

 Could you post the whole content of your code? It would be easier for me 
 to figure out what's wrong if I can run the code myself.
  


 On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:



 On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 I have remade code so I now add into descriptor database all 
 protofiles specified by directory. There is but little problem with error 
 handling. From nature how I am handling errors it seems that message is 
 possible to decode but there is one error in error collector which is 
 bothering me:

 1:1: ERROR: Multiple package definitions.

 *My protos are defined as follows:*
 *addressbook_fragmented.proto:*
 import phone.proto;

 package tutorial;

 option java_package = com.example.tutorial;
 option java_outer_classname = AddressBookProtos;

 message Person {
   required string name = 1;
   required int32 id = 2;// Unique ID number for this person.
   optional string email = 3;

 //  enum PhoneType {
 //MOBILE = 0;
 //HOME = 1;
 //WORK = 2;
 //  }

   message PhoneNumber {
 required string number = 1;
 optional PhoneType type = 2 [default = HOME];
   }

   repeated PhoneNumber phone = 4;
   optional int32 age = 5;
 }

 // Our address book file is just one of these.
 message AddressBook {
   repeated Person person = 1;
 }

 *phone.proto:*
 package tutorial;

 enum PhoneType {
   MOBILE = 0;
   HOME = 1;
   WORK = 2;
 }

 Even though there is error in error collector I can ignore it and 
 decode message successfully. Yet if I try to remove this package info 
 from 
 phone.proto itis not detecting any error but FindMessageTypeByName fails 
 to 
 find data type. I am suspecting since not all references for person data 
 type I am asking for are resolved it fails.

 So I am confused how exactly should be these package keywords be used?

 I don't see a problem in your use of package. You could try to run 
 protoc on these proto files and see if it reports any error. I suspect 
 it's 
 not the .proto file's problem.
  


 On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:



 On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 HI,

 I somehow understand what you mean but still I lack in experience 
 with it. I tried to switch to descriptor database but still I got some 
 problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string 
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor 
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = 
 m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!, 
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream

Re: [protobuf] Dynamic/run-time decoding

2015-06-12 Thread 肖锋



On Wednesday, June 10, 2015 at 9:49:03 PM UTC-7, Jan Kyjovský wrote:

 I did as you told me. And tried to compile my proto files with protoc. 
 Well it eat them up without any problem so I guess that clear that there is 
 no problem with them but in that case why is it causing such error to me.

 As I gave some hint before problem is happening when I am trying to do 
 following with phone.proto which is being added as second into descriptor 
 database

 compiler::Parser parser;

 parser.RecordErrorsTo(errCollector);
 bool bV = parser.Parse(tokenizer, result);
 std::string strErr = errCollector.GetErrorAndWarnings();

 return bV;

 Parse returns false and error collector holds following error:
 1:1: ERROR: Multiple package definitions.

 Do you have nay idea what that could meant?

Could you post the whole content of your code? It would be easier for me to 
figure out what's wrong if I can run the code myself.
 


 On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:



 On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 I have remade code so I now add into descriptor database all protofiles 
 specified by directory. There is but little problem with error handling. 
 From nature how I am handling errors it seems that message is possible to 
 decode but there is one error in error collector which is bothering me:

 1:1: ERROR: Multiple package definitions.

 *My protos are defined as follows:*
 *addressbook_fragmented.proto:*
 import phone.proto;

 package tutorial;

 option java_package = com.example.tutorial;
 option java_outer_classname = AddressBookProtos;

 message Person {
   required string name = 1;
   required int32 id = 2;// Unique ID number for this person.
   optional string email = 3;

 //  enum PhoneType {
 //MOBILE = 0;
 //HOME = 1;
 //WORK = 2;
 //  }

   message PhoneNumber {
 required string number = 1;
 optional PhoneType type = 2 [default = HOME];
   }

   repeated PhoneNumber phone = 4;
   optional int32 age = 5;
 }

 // Our address book file is just one of these.
 message AddressBook {
   repeated Person person = 1;
 }

 *phone.proto:*
 package tutorial;

 enum PhoneType {
   MOBILE = 0;
   HOME = 1;
   WORK = 2;
 }

 Even though there is error in error collector I can ignore it and decode 
 message successfully. Yet if I try to remove this package info from 
 phone.proto itis not detecting any error but FindMessageTypeByName fails to 
 find data type. I am suspecting since not all references for person data 
 type I am asking for are resolved it fails.

 So I am confused how exactly should be these package keywords be used?

 I don't see a problem in your use of package. You could try to run 
 protoc on these proto files and see if it reports any error. I suspect it's 
 not the .proto file's problem.
  


 On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:



 On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 HI,

 I somehow understand what you mean but still I lack in experience with 
 it. I tried to switch to descriptor database but still I got some 
 problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string 
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor 
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = 
 m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!, 
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 result-set_name(m_strProtoFile);

 compiler::Parser parser;

 return parser.Parse(tokenizer, result);
 }

 I have used SimpleDescriptorDatabase since it

Re: [protobuf] Dynamic/run-time decoding

2015-06-10 Thread Jan Kyjovský

I did as you told me. And tried to compile my proto files with protoc. Well 
it eat them up without any problem so I guess that clear that there is no 
problem with them but in that case why is it causing such error to me.

As I gave some hint before problem is happening when I am trying to do 
following with phone.proto which is being added as second into descriptor 
database

compiler::Parser parser;

parser.RecordErrorsTo(errCollector);
bool bV = parser.Parse(tokenizer, result);
std::string strErr = errCollector.GetErrorAndWarnings();

return bV;

Parse returns false and error collector holds following error:
1:1: ERROR: Multiple package definitions.

Do you have nay idea what that could meant?

On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:



 On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 I have remade code so I now add into descriptor database all protofiles 
 specified by directory. There is but little problem with error handling. 
 From nature how I am handling errors it seems that message is possible to 
 decode but there is one error in error collector which is bothering me:

 1:1: ERROR: Multiple package definitions.

 *My protos are defined as follows:*
 *addressbook_fragmented.proto:*
 import phone.proto;

 package tutorial;

 option java_package = com.example.tutorial;
 option java_outer_classname = AddressBookProtos;

 message Person {
   required string name = 1;
   required int32 id = 2;// Unique ID number for this person.
   optional string email = 3;

 //  enum PhoneType {
 //MOBILE = 0;
 //HOME = 1;
 //WORK = 2;
 //  }

   message PhoneNumber {
 required string number = 1;
 optional PhoneType type = 2 [default = HOME];
   }

   repeated PhoneNumber phone = 4;
   optional int32 age = 5;
 }

 // Our address book file is just one of these.
 message AddressBook {
   repeated Person person = 1;
 }

 *phone.proto:*
 package tutorial;

 enum PhoneType {
   MOBILE = 0;
   HOME = 1;
   WORK = 2;
 }

 Even though there is error in error collector I can ignore it and decode 
 message successfully. Yet if I try to remove this package info from 
 phone.proto itis not detecting any error but FindMessageTypeByName fails to 
 find data type. I am suspecting since not all references for person data 
 type I am asking for are resolved it fails.

 So I am confused how exactly should be these package keywords be used?

 I don't see a problem in your use of package. You could try to run 
 protoc on these proto files and see if it reports any error. I suspect it's 
 not the .proto file's problem.
  


 On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:



 On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 HI,

 I somehow understand what you mean but still I lack in experience with 
 it. I tried to switch to descriptor database but still I got some problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string 
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor 
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!, 
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 result-set_name(m_strProtoFile);

 compiler::Parser parser;

 return parser.Parse(tokenizer, result);
 }

 I have used SimpleDescriptorDatabase since it looked like it has 
 already implemented and its covering what I will be requiring from it (or 
 at least I think it does)

 If you use SimpleDescriptorDatabase, you need to add all .proto files 
 into the

Re: [protobuf] Dynamic/run-time decoding

2015-06-09 Thread 'Feng Xiao' via Protocol Buffers

On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 I have remade code so I now add into descriptor database all protofiles
 specified by directory. There is but little problem with error handling.
 From nature how I am handling errors it seems that message is possible to
 decode but there is one error in error collector which is bothering me:

 1:1: ERROR: Multiple package definitions.

 *My protos are defined as follows:*
 *addressbook_fragmented.proto:*
 import phone.proto;

 package tutorial;

 option java_package = com.example.tutorial;
 option java_outer_classname = AddressBookProtos;

 message Person {
   required string name = 1;
   required int32 id = 2;// Unique ID number for this person.
   optional string email = 3;

 //  enum PhoneType {
 //MOBILE = 0;
 //HOME = 1;
 //WORK = 2;
 //  }

   message PhoneNumber {
 required string number = 1;
 optional PhoneType type = 2 [default = HOME];
   }

   repeated PhoneNumber phone = 4;
   optional int32 age = 5;
 }

 // Our address book file is just one of these.
 message AddressBook {
   repeated Person person = 1;
 }

 *phone.proto:*
 package tutorial;

 enum PhoneType {
   MOBILE = 0;
   HOME = 1;
   WORK = 2;
 }

 Even though there is error in error collector I can ignore it and decode
 message successfully. Yet if I try to remove this package info from
 phone.proto itis not detecting any error but FindMessageTypeByName fails to
 find data type. I am suspecting since not all references for person data
 type I am asking for are resolved it fails.

 So I am confused how exactly should be these package keywords be used?

I don't see a problem in your use of package. You could try to run protoc
on these proto files and see if it reports any error. I suspect it's not
the .proto file's problem.



 On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:



 On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com
 wrote:

 HI,

 I somehow understand what you mean but still I lack in experience with
 it. I tried to switch to descriptor database but still I got some problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!,
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 result-set_name(m_strProtoFile);

 compiler::Parser parser;

 return parser.Parse(tokenizer, result);
 }

 I have used SimpleDescriptorDatabase since it looked like it has already
 implemented and its covering what I will be requiring from it (or at least
 I think it does)

 If you use SimpleDescriptorDatabase, you need to add all .proto files
 into the database explicitly. For example, if you have 2 .proto files
 foo.proto and bar.proto, where bar.proto imports foo.proto, you need to add
 both foo.proto and bar.proto to SimpleDescriptorDatabase. If you already
 have a list of all proto files, using SimpleDescriptorDatabase should be
 fine.



 Everything is fine until I try to get descriptor by
 FindMessageTypeByName from pool. It returns null. Am I committing some
 steps?

 I don't see anything obviously wrong in your code.






 On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:



 On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com
 wrote:

 Hi again,

 As I have stated before I am done with decoding, but now I am solving
 different type of problem. As I have mentioned before imports may prove
 problematic to our implementation.

 Let me

Re: [protobuf] Dynamic/run-time decoding

2015-06-08 Thread Jan Kyjovský

I have remade code so I now add into descriptor database all protofiles 
specified by directory. There is but little problem with error handling. 
From nature how I am handling errors it seems that message is possible to 
decode but there is one error in error collector which is bothering me:

1:1: ERROR: Multiple package definitions.

*My protos are defined as follows:*
*addressbook_fragmented.proto:*
import phone.proto;

package tutorial;

option java_package = com.example.tutorial;
option java_outer_classname = AddressBookProtos;

message Person {
  required string name = 1;
  required int32 id = 2;// Unique ID number for this person.
  optional string email = 3;

//  enum PhoneType {
//MOBILE = 0;
//HOME = 1;
//WORK = 2;
//  }

  message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
  optional int32 age = 5;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person person = 1;
}

*phone.proto:*
package tutorial;

enum PhoneType {
  MOBILE = 0;
  HOME = 1;
  WORK = 2;
}

Even though there is error in error collector I can ignore it and decode 
message successfully. Yet if I try to remove this package info from 
phone.proto itis not detecting any error but FindMessageTypeByName fails to 
find data type. I am suspecting since not all references for person data 
type I am asking for are resolved it fails.

So I am confused how exactly should be these package keywords be used?

On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:



 On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 HI,

 I somehow understand what you mean but still I lack in experience with 
 it. I tried to switch to descriptor database but still I got some problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string 
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor 
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!, 
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 result-set_name(m_strProtoFile);

 compiler::Parser parser;

 return parser.Parse(tokenizer, result);
 }

 I have used SimpleDescriptorDatabase since it looked like it has already 
 implemented and its covering what I will be requiring from it (or at least 
 I think it does)

 If you use SimpleDescriptorDatabase, you need to add all .proto files into 
 the database explicitly. For example, if you have 2 .proto files foo.proto 
 and bar.proto, where bar.proto imports foo.proto, you need to add both 
 foo.proto and bar.proto to SimpleDescriptorDatabase. If you already have a 
 list of all proto files, using SimpleDescriptorDatabase should be fine. 
  


 Everything is fine until I try to get descriptor by FindMessageTypeByName 
 from pool. It returns null. Am I committing some steps?

 I don't see anything obviously wrong in your code.
  

  



 On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:



 On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com 
 wrote:

 Hi again,

 As I have stated before I am done with decoding, but now I am solving 
 different type of problem. As I have mentioned before imports may prove 
 problematic to our implementation.

 Let me describe use-case how it will be used. There will be 
 configuration stating for which data (message number, specific field) will 
 be applied which proto file and structure (data type). Therefore there is 
 no knowledge about other proto files up until now. In

Re: [protobuf] Dynamic/run-time decoding

2015-06-03 Thread Jan Kyjovský

HI,

I somehow understand what you mean but still I lack in experience with it. 
I tried to switch to descriptor database but still I got some problems.

C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;

// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}

C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}

bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string 
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor 
*)m_Pool-FindMessageTypeByName(strType);
Message *message = m_MessageFactory.GetPrototype(descriptor)-New();

if (descriptor == NULL)
{
char szError[256];

sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);

return false;
}

bool bFlag = message-ParseFromString(strWireFormat);

if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}

return true;
}

bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
  int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

if (file_descriptor == -1)
{
char szError[256];

sprintf(szError, Invalid proto file \%s\!, 
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);

return false;
}

C_ParserErrorCollector errCollector;

io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
result-set_name(m_strProtoFile);

compiler::Parser parser;

return parser.Parse(tokenizer, result);
}

I have used SimpleDescriptorDatabase since it looked like it has already 
implemented and its covering what I will be requiring from it (or at least 
I think it does)

Everything is fine until I try to get descriptor by FindMessageTypeByName 
from pool. It returns null. Am I committing some steps?


On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:



 On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi again,

 As I have stated before I am done with decoding, but now I am solving 
 different type of problem. As I have mentioned before imports may prove 
 problematic to our implementation.

 Let me describe use-case how it will be used. There will be configuration 
 stating for which data (message number, specific field) will be applied 
 which proto file and structure (data type). Therefore there is no knowledge 
 about other proto files up until now. In run time data are fetched and 
 shipped to decoder. In current implementation is specified proto file 
 loaded and specific type used for decoding. That much is clear and works. 
 But if some parts of structures are imported. It will fail. That much is 
 clear that using just one file descriptor will be not enough as you have 
 mentioned earlier.

 That leads to your proposition (posted earlier) to use 
 DescriptorDatabase. I have looked on implementation of classes derived from 
 DscriptorDatabase. I am more or less able to provide directory for 
 SourceTree, but that doesn't answer question about import. I mean since I 
 am aware of only one proto file. Is import done automatically or other 
 files has to be also included manually into database?


 Another question I have is bout order of operations. You mentioned that I 
 have to first call FindFileByName() before callingFindMessageTypeByName(). 
 That may be problem since I am not aware in which proto file may be located.

 Also I have noticed in code note that proto files with types used in 
 other proto files have to be loaded first. So is there any way how to 
 ensure right order of loading?

 Most of the work is done by DescriptorPool. Basically when you call 
 FindFileByName() on a .proto file that is not yet loaded into the pool, 
 DescriptorPool will call FindFileByName() on its underlying 
 DescriptorDatabase to get the FileDescriptorProto for file. Then 
 DescriptorPool will check the imports of this FileDescriptorPool. If it 
 finds unsatisfied imports (i.e., files imported but not yet in the pool), 
 it will issue FindFileByName() to the DescriptorDatabase again for these 
 files.

 So basically DescriptorDatabase only needs to know how to load a single 
 .proto file, while DescriptorPool will take care of the imports and 
 ordering.

 Depending on what DescriptorDatabase you are using, you may or may not 
 need to call FindFileByName() first. If the DescriptorDatabase has 
 implement FindFileContainingSymbol, then calling 
 DescriptorPool::FindMessageTypeByName() directly will be able to find the 
 file using this method. If not (the case with 
 SourceTreeDescriptorDatabase),

Re: [protobuf] Dynamic/run-time decoding

2015-06-03 Thread 'Feng Xiao' via Protocol Buffers

On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 HI,

 I somehow understand what you mean but still I lack in experience with it.
 I tried to switch to descriptor database but still I got some problems.

 C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
 {
 m_strProtoFile = strFile;

 // Build the descriptor pool
 m_bConstructed = ParseProtoFile(m_ProtoFileDescr);

 if (m_bConstructed)
 {
 m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
 m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
 }
 }

 C_ProtoDecoder::~C_ProtoDecoder()
 {
 delete m_Pool;
 }

 bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
 strWireFormat, std::string strType)
 {
 Descriptor *descriptor = (Descriptor
 *)m_Pool-FindMessageTypeByName(strType);
 Message *message = m_MessageFactory.GetPrototype(descriptor)-New();

 if (descriptor == NULL)
 {
 char szError[256];

 sprintf(szError, Unknown data type %s!, strType.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 bool bFlag = message-ParseFromString(strWireFormat);

 if (!bFlag)
 {
 m_InternalPrinter.AddErrorMessage(Encoding error!);
 }
 else
 {
 m_Transformator.MorphToStructs(*message);
 }

 return true;
 }

 bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
 {
   int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

 if (file_descriptor == -1)
 {
 char szError[256];

 sprintf(szError, Invalid proto file \%s\!,
 m_strProtoFile.c_str());
 m_InternalPrinter.AddErrorMessage(szError);

 return false;
 }

 C_ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 result-set_name(m_strProtoFile);

 compiler::Parser parser;

 return parser.Parse(tokenizer, result);
 }

 I have used SimpleDescriptorDatabase since it looked like it has already
 implemented and its covering what I will be requiring from it (or at least
 I think it does)

If you use SimpleDescriptorDatabase, you need to add all .proto files into
the database explicitly. For example, if you have 2 .proto files foo.proto
and bar.proto, where bar.proto imports foo.proto, you need to add both
foo.proto and bar.proto to SimpleDescriptorDatabase. If you already have a
list of all proto files, using SimpleDescriptorDatabase should be fine.



 Everything is fine until I try to get descriptor by FindMessageTypeByName
 from pool. It returns null. Am I committing some steps?

I don't see anything obviously wrong in your code.






 On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:



 On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com wrote:

 Hi again,

 As I have stated before I am done with decoding, but now I am solving
 different type of problem. As I have mentioned before imports may prove
 problematic to our implementation.

 Let me describe use-case how it will be used. There will be
 configuration stating for which data (message number, specific field) will
 be applied which proto file and structure (data type). Therefore there is
 no knowledge about other proto files up until now. In run time data are
 fetched and shipped to decoder. In current implementation is specified
 proto file loaded and specific type used for decoding. That much is clear
 and works. But if some parts of structures are imported. It will fail. That
 much is clear that using just one file descriptor will be not enough as you
 have mentioned earlier.

 That leads to your proposition (posted earlier) to use
 DescriptorDatabase. I have looked on implementation of classes derived from
 DscriptorDatabase. I am more or less able to provide directory for
 SourceTree, but that doesn't answer question about import. I mean since I
 am aware of only one proto file. Is import done automatically or other
 files has to be also included manually into database?


 Another question I have is bout order of operations. You mentioned that
 I have to first call FindFileByName() before
 callingFindMessageTypeByName(). That may be problem since I am not aware in
 which proto file may be located.

 Also I have noticed in code note that proto files with types used in
 other proto files have to be loaded first. So is there any way how to
 ensure right order of loading?

 Most of the work is done by DescriptorPool. Basically when you call
 FindFileByName() on a .proto file that is not yet loaded into the pool,
 DescriptorPool will call FindFileByName() on its underlying
 DescriptorDatabase to get the FileDescriptorProto for file. Then
 DescriptorPool will check the imports of this FileDescriptorPool. If it
 finds unsatisfied imports (i.e., files imported but not yet in the pool),
 it will issue FindFileByName() to the DescriptorDatabase again for these

Re: [protobuf] Dynamic/run-time decoding

2015-06-02 Thread 'Feng Xiao' via Protocol Buffers

On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.kyjov...@tieto.com wrote:

 Hi again,

 As I have stated before I am done with decoding, but now I am solving
 different type of problem. As I have mentioned before imports may prove
 problematic to our implementation.

 Let me describe use-case how it will be used. There will be configuration
 stating for which data (message number, specific field) will be applied
 which proto file and structure (data type). Therefore there is no knowledge
 about other proto files up until now. In run time data are fetched and
 shipped to decoder. In current implementation is specified proto file
 loaded and specific type used for decoding. That much is clear and works.
 But if some parts of structures are imported. It will fail. That much is
 clear that using just one file descriptor will be not enough as you have
 mentioned earlier.

 That leads to your proposition (posted earlier) to use DescriptorDatabase.
 I have looked on implementation of classes derived from DscriptorDatabase.
 I am more or less able to provide directory for SourceTree, but that
 doesn't answer question about import. I mean since I am aware of only one
 proto file. Is import done automatically or other files has to be also
 included manually into database?


 Another question I have is bout order of operations. You mentioned that I
 have to first call FindFileByName() before callingFindMessageTypeByName().
 That may be problem since I am not aware in which proto file may be located.

 Also I have noticed in code note that proto files with types used in other
 proto files have to be loaded first. So is there any way how to ensure
 right order of loading?

Most of the work is done by DescriptorPool. Basically when you call
FindFileByName() on a .proto file that is not yet loaded into the pool,
DescriptorPool will call FindFileByName() on its underlying
DescriptorDatabase to get the FileDescriptorProto for file. Then
DescriptorPool will check the imports of this FileDescriptorPool. If it
finds unsatisfied imports (i.e., files imported but not yet in the pool),
it will issue FindFileByName() to the DescriptorDatabase again for these
files.

So basically DescriptorDatabase only needs to know how to load a single
.proto file, while DescriptorPool will take care of the imports and
ordering.

Depending on what DescriptorDatabase you are using, you may or may not need
to call FindFileByName() first. If the DescriptorDatabase has implement
FindFileContainingSymbol, then calling
DescriptorPool::FindMessageTypeByName() directly will be able to find the
file using this method. If not (the case with
SourceTreeDescriptorDatabase), FindFileByName() must be called first. As
DescriptorPool will be able to find imports by itself, you don't need to
call FindFileByName() for every file, but just the file that you actually
need to use.



 I hope my question are not too strange but I my knowledge about google
 protocol buffers are a bit limited.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-06-01 Thread Jan Kyjovský

Hi again,

As I have stated before I am done with decoding, but now I am solving 
different type of problem. As I have mentioned before imports may prove 
problematic to our implementation.

Let me describe use-case how it will be used. There will be configuration 
stating for which data (message number, specific field) will be applied 
which proto file and structure (data type). Therefore there is no knowledge 
about other proto files up until now. In run time data are fetched and 
shipped to decoder. In current implementation is specified proto file 
loaded and specific type used for decoding. That much is clear and works. 
But if some parts of structures are imported. It will fail. That much is 
clear that using just one file descriptor will be not enough as you have 
mentioned earlier.

That leads to your proposition (posted earlier) to use DescriptorDatabase. 
I have looked on implementation of classes derived from DscriptorDatabase. 
I am more or less able to provide directory for SourceTree, but that 
doesn't answer question about import. I mean since I am aware of only one 
proto file. Is import done automatically or other files has to be also 
included manually into database?

Another question I have is bout order of operations. You mentioned that I 
have to first call FindFileByName() before callingFindMessageTypeByName(). 
That may be problem since I am not aware in which proto file may be located.

Also I have noticed in code note that proto files with types used in other 
proto files have to be loaded first. So is there any way how to ensure 
right order of loading?

I hope my question are not too strange but I my knowledge about google 
protocol buffers are a bit limited.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-28 Thread 'Feng Xiao' via Protocol Buffers

On Thu, May 28, 2015 at 12:02 AM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 Hi,

 yes thank you for that hint. I was able to get enough references to go
 through decoded data. I am more or less efficiently able to get values (in
 case of ENUM also original value not only symbolic meaning) but now I would
 require some auxiliary information which I am not sure are contained in
 these objects. I will need to get access to current byte offset (from data
 start) and size of element.

No, it's not possible to get the byte offset with protobuf API. If you need
that, you probably have to write your own parsing method to parse the data.


 Although size is something I can probably somehow derive. Is there any
 possibility to that?

 On Thursday, 7 May 2015 19:44:22 UTC+2, Feng Xiao wrote:



 On Thu, May 7, 2015 at 2:29 AM, Jan Kyjovský jan.ky...@tieto.com wrote:

 Hi,

 sorry for asking for advice so soon again.

 I have progressed a bit futher and I am now trying to display decoded
 data. For that purpose I have prepared some data and tried to decode them
 but I have encountered problem with repeated structures and with nested
 messages. I dont know how to get count of repeats in case of repeated
 fields so its problematic to address them. In case of nested messages I
 have experiencing difficulties how to get message.

 Here is function I have made so far. Note that I am suspecting that it
 will probably need some more tweaking so it can be called recursively (work
 still in progress). For now I am satisfied just by displaying data on first
 level.

 void ExpandSubMessage(Message *message, Descriptor *descriptor)
 {
 for (int i = 0; i  descriptor-field_count(); i++)
 {
 const FieldDescriptor* field = descriptor-field(i);
 switch (field-type())
 {
 case FieldDescriptor::TYPE_INT32:
 {
 int nVal = message-GetReflection()-GetInt32(*message,
 field);
 printf(%s = %d\n, field-name().c_str(), nVal);

 break;
 }
 case FieldDescriptor::TYPE_STRING:
 {
 std::string strVal =
 message-GetReflection()-GetString(*message, field);
 printf(%s = %s\n, field-name().c_str(),
 strVal.c_str());

 break;
 }
 //case FieldDescriptor::TYPE_ENUM:
 message-GetReflection()-GetEnum(*message, field); break;
 case FieldDescriptor::TYPE_MESSAGE:
 {
 Message *messVal;
 if (field-is_repeated())
 {

 message-GetReflection()-GetRepeatedMessage(*message, field, 1);
 }
 else
 {
 message-GetReflection()-GetMessage(*messVal,
 field);
 }
 printf(%s = %s\n, field-name().c_str(),
 /*messVal.DebugString().c_str()*/Test);

 break;
 }
 }
 }
 }

 I have tried to look for details in specification on your pages but
 didn't find anything that would answer my question.

 May I once again ask for your assistance?

 What you are trying to do looks similar to what the TextFormat class
 does. You can refer to that class as an example to learn how to use
 protobuf reflection to traverse a message tree:

 https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1449



  --
 You received this message because you are subscribed to the Google
 Groups Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-28 Thread Jan Kyjovský

Never mind then. We can provide results in two formats. Offsets and sizes 
are requirement of one of these and after giving it some thought I get the 
feeling that it would be better to use d second format (formatted text), 
also it would require much less work.

Thank you very much four your help. This time it is really end of the line. 
I am able to decode data in run-time and that's what was desired from the 
start.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-28 Thread Jan Kyjovský

Hi,

yes thank you for that hint. I was able to get enough references to go 
through decoded data. I am more or less efficiently able to get values (in 
case of ENUM also original value not only symbolic meaning) but now I would 
require some auxiliary information which I am not sure are contained in 
these objects. I will need to get access to current byte offset (from data 
start) and size of element. Although size is something I can probably 
somehow derive. Is there any possibility to that?

On Thursday, 7 May 2015 19:44:22 UTC+2, Feng Xiao wrote:



 On Thu, May 7, 2015 at 2:29 AM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi,

 sorry for asking for advice so soon again.

 I have progressed a bit futher and I am now trying to display decoded 
 data. For that purpose I have prepared some data and tried to decode them 
 but I have encountered problem with repeated structures and with nested 
 messages. I dont know how to get count of repeats in case of repeated 
 fields so its problematic to address them. In case of nested messages I 
 have experiencing difficulties how to get message.

 Here is function I have made so far. Note that I am suspecting that it 
 will probably need some more tweaking so it can be called recursively (work 
 still in progress). For now I am satisfied just by displaying data on first 
 level.

 void ExpandSubMessage(Message *message, Descriptor *descriptor)
 {
 for (int i = 0; i  descriptor-field_count(); i++)
 {
 const FieldDescriptor* field = descriptor-field(i);
 switch (field-type())
 {
 case FieldDescriptor::TYPE_INT32:
 {
 int nVal = message-GetReflection()-GetInt32(*message, 
 field);
 printf(%s = %d\n, field-name().c_str(), nVal);

 break;
 }
 case FieldDescriptor::TYPE_STRING:
 {
 std::string strVal = 
 message-GetReflection()-GetString(*message, field);
 printf(%s = %s\n, field-name().c_str(), 
 strVal.c_str());

 break;
 }
 //case FieldDescriptor::TYPE_ENUM: 
 message-GetReflection()-GetEnum(*message, field); break;
 case FieldDescriptor::TYPE_MESSAGE:
 {
 Message *messVal;
 if (field-is_repeated())
 {
 
 message-GetReflection()-GetRepeatedMessage(*message, field, 1);
 }
 else
 {
 message-GetReflection()-GetMessage(*messVal, field);
 }
 printf(%s = %s\n, field-name().c_str(), 
 /*messVal.DebugString().c_str()*/Test);

 break;
 }
 }
 }
 }

 I have tried to look for details in specification on your pages but 
 didn't find anything that would answer my question.

 May I once again ask for your assistance?

 What you are trying to do looks similar to what the TextFormat class does. 
 You can refer to that class as an example to learn how to use protobuf 
 reflection to traverse a message tree:

 https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1449
  


  -- 
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to protobuf+u...@googlegroups.com javascript:.
 To post to this group, send email to prot...@googlegroups.com 
 javascript:.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-07 Thread Jan Kyjovský

Hi,

sorry for asking for advice so soon again.

I have progressed a bit futher and I am now trying to display decoded data. 
For that purpose I have prepared some data and tried to decode them but I 
have encountered problem with repeated structures and with nested messages. 
I dont know how to get count of repeats in case of repeated fields so its 
problematic to address them. In case of nested messages I have experiencing 
difficulties how to get message.

Here is function I have made so far. Note that I am suspecting that it will 
probably need some more tweaking so it can be called recursively (work 
still in progress). For now I am satisfied just by displaying data on first 
level.

void ExpandSubMessage(Message *message, Descriptor *descriptor)
{
for (int i = 0; i  descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal = message-GetReflection()-GetInt32(*message, 
field);
printf(%s = %d\n, field-name().c_str(), nVal);

break;
}
case FieldDescriptor::TYPE_STRING:
{
std::string strVal = 
message-GetReflection()-GetString(*message, field);
printf(%s = %s\n, field-name().c_str(), strVal.c_str());

break;
}
//case FieldDescriptor::TYPE_ENUM: 
message-GetReflection()-GetEnum(*message, field); break;
case FieldDescriptor::TYPE_MESSAGE:
{
Message *messVal;
if (field-is_repeated())
{
message-GetReflection()-GetRepeatedMessage(*message, 
field, 1);
}
else
{
message-GetReflection()-GetMessage(*messVal, field);
}
printf(%s = %s\n, field-name().c_str(), 
/*messVal.DebugString().c_str()*/Test);

break;
}
}
}
}

I have tried to look for details in specification on your pages but didn't 
find anything that would answer my question.

May I once again ask for your assistance?

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-07 Thread 'Feng Xiao' via Protocol Buffers

On Thu, May 7, 2015 at 2:29 AM, Jan Kyjovský jan.kyjov...@tieto.com wrote:

 Hi,

 sorry for asking for advice so soon again.

 I have progressed a bit futher and I am now trying to display decoded
 data. For that purpose I have prepared some data and tried to decode them
 but I have encountered problem with repeated structures and with nested
 messages. I dont know how to get count of repeats in case of repeated
 fields so its problematic to address them. In case of nested messages I
 have experiencing difficulties how to get message.

 Here is function I have made so far. Note that I am suspecting that it
 will probably need some more tweaking so it can be called recursively (work
 still in progress). For now I am satisfied just by displaying data on first
 level.

 void ExpandSubMessage(Message *message, Descriptor *descriptor)
 {
 for (int i = 0; i  descriptor-field_count(); i++)
 {
 const FieldDescriptor* field = descriptor-field(i);
 switch (field-type())
 {
 case FieldDescriptor::TYPE_INT32:
 {
 int nVal = message-GetReflection()-GetInt32(*message,
 field);
 printf(%s = %d\n, field-name().c_str(), nVal);

 break;
 }
 case FieldDescriptor::TYPE_STRING:
 {
 std::string strVal =
 message-GetReflection()-GetString(*message, field);
 printf(%s = %s\n, field-name().c_str(), strVal.c_str());

 break;
 }
 //case FieldDescriptor::TYPE_ENUM:
 message-GetReflection()-GetEnum(*message, field); break;
 case FieldDescriptor::TYPE_MESSAGE:
 {
 Message *messVal;
 if (field-is_repeated())
 {
 message-GetReflection()-GetRepeatedMessage(*message,
 field, 1);
 }
 else
 {
 message-GetReflection()-GetMessage(*messVal, field);
 }
 printf(%s = %s\n, field-name().c_str(),
 /*messVal.DebugString().c_str()*/Test);

 break;
 }
 }
 }
 }

 I have tried to look for details in specification on your pages but didn't
 find anything that would answer my question.

 May I once again ask for your assistance?

What you are trying to do looks similar to what the TextFormat class does.
You can refer to that class as an example to learn how to use protobuf
reflection to traverse a message tree:
https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1449



  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-06 Thread Jan Kyjovský

Hi,

This is most likely last post. After converting these example data to 
binary and making sure that that parsing was successful (return true), I am 
finally getting decoded data. From this point on I can finish what was 
requested.

Once again thank you for all your advices and patience.

On Tuesday, 5 May 2015 19:35:03 UTC+2, Feng Xiao wrote:



 On Mon, May 4, 2015 at 11:36 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi,

 so we are somehow back in game. And now I am getting familiar what I have 
 done more then half a year ago.

 So for now I got some basic structure of code that will later be as a 
 starting point for real implementation. I am trying to do some simple 
 experiments to see how it works. Though I got problem that even if I get 
 descriptor I am not getting any data. I have used example from encoding 
 specification since I needed something really simple. 

 ...
 const Descriptor* descriptor = 
 pool.FindMessageTypeByName(tutorial.Test1);
 if (descriptor == NULL)
 {
 return;
 }
 DynamicMessageFactory factory;
 Message *message = factory.GetPrototype(descriptor)-New();
 // Use the message object for parsing/etc.
 std::string input_data = 089601;

 ParseFromString() expects protobuf binary format data (stored in a C++ 
 string), which I believe 089601 is not.
  


 message-ParseFromString(input_data);

 You could check the return value of ParseFromString() to see whether the 
 parsing is a success.
  

 // Access a specific field in the message

 for (int i = 0; i  descriptor-field_count(); i++)
 {
 const FieldDescriptor* field = descriptor-field(i);
 switch (field-type())
 {
 case FieldDescriptor::TYPE_INT32:
 {
 int nVal = 
 message-GetReflection()-GetInt32(*message, field);
 printf(%d\n, nVal);

 break;
 }
 ...

 But from reflection I am getting that value is 0. So either I have 
 wrongly set input data considering that I misunderstand format or there is 
 something else. Do you have any idea what may be cause?

 Thank you in advance for your reply.

 -- 
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to protobuf+u...@googlegroups.com javascript:.
 To post to this group, send email to prot...@googlegroups.com 
 javascript:.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-05 Thread Jan Kyjovský

Hi,

so we are somehow back in game. And now I am getting familiar what I have 
done more then half a year ago.

So for now I got some basic structure of code that will later be as a 
starting point for real implementation. I am trying to do some simple 
experiments to see how it works. Though I got problem that even if I get 
descriptor I am not getting any data. I have used example from encoding 
specification since I needed something really simple. 

...
const Descriptor* descriptor = 
pool.FindMessageTypeByName(tutorial.Test1);
if (descriptor == NULL)
{
return;
}
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
std::string input_data = 089601;
message-ParseFromString(input_data);
// Access a specific field in the message

for (int i = 0; i  descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal = message-GetReflection()-GetInt32(*message, 
field);
printf(%d\n, nVal);

break;
}
...

But from reflection I am getting that value is 0. So either I have wrongly 
set input data considering that I misunderstand format or there is 
something else. Do you have any idea what may be cause?

Thank you in advance for your reply.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2015-05-05 Thread 'Feng Xiao' via Protocol Buffers

On Mon, May 4, 2015 at 11:36 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 Hi,

 so we are somehow back in game. And now I am getting familiar what I have
 done more then half a year ago.

 So for now I got some basic structure of code that will later be as a
 starting point for real implementation. I am trying to do some simple
 experiments to see how it works. Though I got problem that even if I get
 descriptor I am not getting any data. I have used example from encoding
 specification since I needed something really simple.

 ...
 const Descriptor* descriptor =
 pool.FindMessageTypeByName(tutorial.Test1);
 if (descriptor == NULL)
 {
 return;
 }
 DynamicMessageFactory factory;
 Message *message = factory.GetPrototype(descriptor)-New();
 // Use the message object for parsing/etc.
 std::string input_data = 089601;

ParseFromString() expects protobuf binary format data (stored in a C++
string), which I believe 089601 is not.



 message-ParseFromString(input_data);

You could check the return value of ParseFromString() to see whether the
parsing is a success.


 // Access a specific field in the message

 for (int i = 0; i  descriptor-field_count(); i++)
 {
 const FieldDescriptor* field = descriptor-field(i);
 switch (field-type())
 {
 case FieldDescriptor::TYPE_INT32:
 {
 int nVal =
 message-GetReflection()-GetInt32(*message, field);
 printf(%d\n, nVal);

 break;
 }
 ...

 But from reflection I am getting that value is 0. So either I have wrongly
 set input data considering that I misunderstand format or there is
 something else. Do you have any idea what may be cause?

 Thank you in advance for your reply.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2014-10-15 Thread Jan Kyjovský

Hi,

due to organization changes and budget costs this task is put on hold.
Thank you for all your guidance up until now. If any further help is needed
regarding this issue in the future me or someone else will continue with
this thread.

On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote:

On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:

Hi,

Thank you that was one thing I tried but was not successful up until now.
Also one more issue was that i had to ask for type by full name. It took me
some time to figure it our.

Now I got different sort of questions. Since I can now progress further I
am finding new problems. As you suggested i will be using ParseFromString
method. Although I am not entirely sure what this string is. Is it
represented by hexdump or is it some other format?

ParseFromString() accepts protobuf wire-format data (which is the output
of protobuf serialization routines).

I am also wondering about imports. If one proto file imports other do I
have to take that consideration and manually handle that file as well or is
it done automatically while processing file I have provided? In my
implementation all proto files are in one folder with possible imports
between each other.

You can use the DescriptorDatabase implementation we use for protocol
compiler:

https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76

Basically rather than adding individual files to the pool, you provide a
DescriptorDatabase to the pool and the pool will use this
DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of
this approach is that, you need to call DescriptorPool::FindFileByName() on
the .proto files before you can use DescriptorPool::FindMessageTypeByName()
to search for types.

Thank you in advance for your response.

--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+u...@googlegroups.com javascript:.
To post to this group, send email to prot...@googlegroups.com
javascript:.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2014-10-15 Thread Jan Kyjovský

Hi,

due to organization changes and budget cuts this task is put on hold. Thank
you for all your guidance up until now. If any further help is needed
regarding this issue in the future me or someone else will continue with
this thread.

On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote:

On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:

Hi,

Thank you that was one thing I tried but was not successful up until now.
Also one more issue was that i had to ask for type by full name. It took me
some time to figure it our.

ParseFromString() accepts protobuf wire-format data (which is the output
of protobuf serialization routines).

You can use the DescriptorDatabase implementation we use for protocol
compiler:

https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76

Thank you in advance for your response.

--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2014-09-11 Thread 'Feng Xiao' via Protocol Buffers

On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

Hi,

Thank you that was one thing I tried but was not successful up until now.
Also one more issue was that i had to ask for type by full name. It took me
some time to figure it our.

ParseFromString() accepts protobuf wire-format data (which is the output of
protobuf serialization routines).

You can use the DescriptorDatabase implementation we use for protocol
compiler:
https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76

Thank you in advance for your response.

--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2014-09-10 Thread Jan Kyjovský

Variable proto contains some information about structure although 
proto.name is blank (don't know why).
BuildFile returns NULL as well as FindMessageTypeByName. Whey I went deeper 
in debugging I noticed that BuildFile is trying to look up for that file I 
provide in its tables but these are empty as well.

On Tuesday, 9 September 2014 22:35:05 UTC+2, Feng Xiao wrote:



 On Mon, Sep 8, 2014 at 10:32 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi,

 sorry for delay. I've been experimenting with that eve since I could 
 return to this issue. Mostly about composition of project since whenever I 
 try separate your sources from projects and use them individually I§ve been 
 getting tons of errors (linking one included). In the end I get along with 
 static library linked in my project. But putting that aside.

 I haven't get as far as to use all your example. I am stuck and fact that 
 when asking pool for specific message I am getting empty default descriptor 
 as result. I am attaching my code so far could you please advise where I am 
 making mistake?

 Its just short program where I was trying to figure out how to use it.

 #include io.h
 #include fcntl.h
 #include stdio.h
 #include string
 #include vector
 #include descriptor.h
 #include descriptor.pb.h
 #include zero_copy_stream_impl.h
 #include tokenizer.h
 #include parser.h
 #include substitute.h
 #include dynamic_message.h

 using namespace google::protobuf;

 class ParserErrorCollector : public io::ErrorCollector
 {
 public:
 ParserErrorCollector() {}
 ~ParserErrorCollector() {}

 std::string text_;

 // implements ErrorCollector -
 void AddError(int line, int column, const std::string message)
 {
 strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column + 1, 
 message);
 }

 void AddWarning(int line, int column, const std::string message)
 {
 AddError(line, column, WARNING: + message);
 }
 };

 bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);

 void main()
 {
 std::string strFile = 
 C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto;

 // Build the descriptor pool
 DescriptorPool pool;

 FileDescriptorProto proto;
 ParseProtoFile(strFile, proto); 

 pool.BuildFile(proto);

 Have you checked the return value of these two method calls?
  


 // Suppose you want to parse a message type with a specific type name.
 const Descriptor* descriptor = pool.FindMessageTypeByName(AddressBook);

 Does this return NULL or non-NULL? If it's the later, it means you have 
 successfully constructed the descriptor for AddressBook. 
  

 DynamicMessageFactory factory;
 Message *message = factory.GetPrototype(descriptor)-New();
 }

 // Convert .proto files into parsed FileDescriptorProto
 bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
 {
   int file_descriptor = open(strFileName.c_str(), O_RDONLY);

 ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 compiler::Parser parser;

 return parser.Parse(tokenizer, result);

 return true;
 }


 On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:

 Protobuf supports creating message types dynamically at runtime and use 
 them for parsing/serialization/etc.

 First you need to build up a DescriptorPool 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
  
 that contains all types that you may want to use. There are two approaches 
 to construct this pool. One is to call DescriptorPool::BuildFile() directly 
 with parsed proto files. For example:
   // Convert .proto files into parsed FileDescriptorProto
   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
 FileInputStream stream(filename);
 google::protobuf::io::Tokenizer tokenizer(stream);
 google::protobuf::compiler::Parser parser;
 return parser.Parse(tokenizer, result);
   }
   // Build the descriptor pool
   DescriptorPool pool;
   for (string filename : proto_files) {
 FileDescriptorProto proto;
 ParseProtoFile(filename, proto);
 pool.BuildFile(proto);
   }

 After you have the pool, you can query for a type by its name. For 
 example, DescriptorPool::FindMessageTypeByName().

 Then to actually parse/serialize/use message types in the pool, you need 
 to construct message objects around them. DynamicMessage 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
  
 is used for that:
   // Suppose you want to parse a message type with a specific type name.
   Descriptor* descriptor = pool.FindMessageTypeByName(
 message_type_to_parse);
   DynamicMessageFactory factory;
   unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
   // Use the message object for parsing/etc.
   message-ParseFromString(input_data);
   // Access a specific field in the message
   FieldDescriptor* field =

Re: [protobuf] Dynamic/run-time decoding

2014-09-10 Thread 'Feng Xiao' via Protocol Buffers

On Tue, Sep 9, 2014 at 11:03 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 Variable proto contains some information about structure although
 proto.name is blank (don't know why).

You need to set the proto.name yourself:
https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.cc#L147

I guess this is the reason why it doesn't work. Sorry that I missed this in
my code example.


 BuildFile returns NULL as well as FindMessageTypeByName. Whey I went
 deeper in debugging I noticed that BuildFile is trying to look up for that
 file I provide in its tables but these are empty as well.

 On Tuesday, 9 September 2014 22:35:05 UTC+2, Feng Xiao wrote:



 On Mon, Sep 8, 2014 at 10:32 PM, Jan Kyjovský jan.ky...@tieto.com
 wrote:

 Hi,

 sorry for delay. I've been experimenting with that eve since I could
 return to this issue. Mostly about composition of project since whenever I
 try separate your sources from projects and use them individually I§ve been
 getting tons of errors (linking one included). In the end I get along with
 static library linked in my project. But putting that aside.

 I haven't get as far as to use all your example. I am stuck and fact
 that when asking pool for specific message I am getting empty default
 descriptor as result. I am attaching my code so far could you please advise
 where I am making mistake?

 Its just short program where I was trying to figure out how to use it.

 #include io.h
 #include fcntl.h
 #include stdio.h
 #include string
 #include vector
 #include descriptor.h
 #include descriptor.pb.h
 #include zero_copy_stream_impl.h
 #include tokenizer.h
 #include parser.h
 #include substitute.h
 #include dynamic_message.h

 using namespace google::protobuf;

 class ParserErrorCollector : public io::ErrorCollector
 {
 public:
 ParserErrorCollector() {}
 ~ParserErrorCollector() {}

 std::string text_;

 // implements ErrorCollector -
 void AddError(int line, int column, const std::string message)
 {
 strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column +
 1, message);
 }

 void AddWarning(int line, int column, const std::string message)
 {
 AddError(line, column, WARNING: + message);
 }
 };

 bool ParseProtoFile(std::string strFileName, FileDescriptorProto*
 result);

 void main()
 {
 std::string strFile = C:\\Users\\kyjovjan\\Desktop\
 \ProtoExperiment\\addressbook.proto;

 // Build the descriptor pool
 DescriptorPool pool;

 FileDescriptorProto proto;
 ParseProtoFile(strFile, proto);

 pool.BuildFile(proto);

 Have you checked the return value of these two method calls?



 // Suppose you want to parse a message type with a specific type name.
 const Descriptor* descriptor = pool.FindMessageTypeByName(
 AddressBook);

 Does this return NULL or non-NULL? If it's the later, it means you have
 successfully constructed the descriptor for AddressBook.


 DynamicMessageFactory factory;
 Message *message = factory.GetPrototype(descriptor)-New();
 }

 // Convert .proto files into parsed FileDescriptorProto
 bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
 {
   int file_descriptor = open(strFileName.c_str(), O_RDONLY);

 ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 compiler::Parser parser;

 return parser.Parse(tokenizer, result);

 return true;
 }


 On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:

 Protobuf supports creating message types dynamically at runtime and use
 them for parsing/serialization/etc.

 First you need to build up a DescriptorPool
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
 that contains all types that you may want to use. There are two approaches
 to construct this pool. One is to call DescriptorPool::BuildFile() directly
 with parsed proto files. For example:
   // Convert .proto files into parsed FileDescriptorProto
   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
 FileInputStream stream(filename);
 google::protobuf::io::Tokenizer tokenizer(stream);
 google::protobuf::compiler::Parser parser;
 return parser.Parse(tokenizer, result);
   }
   // Build the descriptor pool
   DescriptorPool pool;
   for (string filename : proto_files) {
 FileDescriptorProto proto;
 ParseProtoFile(filename, proto);
 pool.BuildFile(proto);
   }

 After you have the pool, you can query for a type by its name. For
 example, DescriptorPool::FindMessageTypeByName().

 Then to actually parse/serialize/use message types in the pool, you
 need to construct message objects around them. DynamicMessage
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
 is used for that:
   // Suppose you want to parse a message type with a specific type name.
   Descriptor* descriptor = pool.FindMessageTypeByName(mes

Re: [protobuf] Dynamic/run-time decoding

2014-09-09 Thread 'Feng Xiao' via Protocol Buffers

On Mon, Sep 8, 2014 at 10:32 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 Hi,

 sorry for delay. I've been experimenting with that eve since I could
 return to this issue. Mostly about composition of project since whenever I
 try separate your sources from projects and use them individually I§ve been
 getting tons of errors (linking one included). In the end I get along with
 static library linked in my project. But putting that aside.

 I haven't get as far as to use all your example. I am stuck and fact that
 when asking pool for specific message I am getting empty default descriptor
 as result. I am attaching my code so far could you please advise where I am
 making mistake?

 Its just short program where I was trying to figure out how to use it.

 #include io.h
 #include fcntl.h
 #include stdio.h
 #include string
 #include vector
 #include descriptor.h
 #include descriptor.pb.h
 #include zero_copy_stream_impl.h
 #include tokenizer.h
 #include parser.h
 #include substitute.h
 #include dynamic_message.h

 using namespace google::protobuf;

 class ParserErrorCollector : public io::ErrorCollector
 {
 public:
 ParserErrorCollector() {}
 ~ParserErrorCollector() {}

 std::string text_;

 // implements ErrorCollector -
 void AddError(int line, int column, const std::string message)
 {
 strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column + 1,
 message);
 }

 void AddWarning(int line, int column, const std::string message)
 {
 AddError(line, column, WARNING: + message);
 }
 };

 bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);

 void main()
 {
 std::string strFile =
 C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto;

 // Build the descriptor pool
 DescriptorPool pool;

 FileDescriptorProto proto;
 ParseProtoFile(strFile, proto);

pool.BuildFile(proto);

Have you checked the return value of these two method calls?



 // Suppose you want to parse a message type with a specific type name.
 const Descriptor* descriptor = pool.FindMessageTypeByName(AddressBook);

Does this return NULL or non-NULL? If it's the later, it means you have
successfully constructed the descriptor for AddressBook.


 DynamicMessageFactory factory;
 Message *message = factory.GetPrototype(descriptor)-New();
 }

 // Convert .proto files into parsed FileDescriptorProto
 bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
 {
   int file_descriptor = open(strFileName.c_str(), O_RDONLY);

 ParserErrorCollector errCollector;

 io::FileInputStream stream(file_descriptor);
 stream.SetCloseOnDelete(true);
 io::Tokenizer tokenizer(stream, errCollector);
 compiler::Parser parser;

 return parser.Parse(tokenizer, result);

 return true;
 }


 On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:

 Protobuf supports creating message types dynamically at runtime and use
 them for parsing/serialization/etc.

 First you need to build up a DescriptorPool
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
 that contains all types that you may want to use. There are two approaches
 to construct this pool. One is to call DescriptorPool::BuildFile() directly
 with parsed proto files. For example:
   // Convert .proto files into parsed FileDescriptorProto
   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
 FileInputStream stream(filename);
 google::protobuf::io::Tokenizer tokenizer(stream);
 google::protobuf::compiler::Parser parser;
 return parser.Parse(tokenizer, result);
   }
   // Build the descriptor pool
   DescriptorPool pool;
   for (string filename : proto_files) {
 FileDescriptorProto proto;
 ParseProtoFile(filename, proto);
 pool.BuildFile(proto);
   }

 After you have the pool, you can query for a type by its name. For
 example, DescriptorPool::FindMessageTypeByName().

 Then to actually parse/serialize/use message types in the pool, you need
 to construct message objects around them. DynamicMessage
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
 is used for that:
   // Suppose you want to parse a message type with a specific type name.
   Descriptor* descriptor = pool.FindMessageTypeByName(
 message_type_to_parse);
   DynamicMessageFactory factory;
   unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
   // Use the message object for parsing/etc.
   message-ParseFromString(input_data);
   // Access a specific field in the message
   FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
   switch (field-type()) {
 case TYPE_INT32: message-GetReflection()-GetInt32(*message,
 field); break;
 ...
   }

 On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.ky...@tieto.com
 wrote:

 Hi,

 I have very specific problem. I have data and proto file available and
 my application should take both and based on external configuration
 determine how to interpret data

Re: [protobuf] Dynamic/run-time decoding

2014-09-08 Thread Jan Kyjovský

Hi,

sorry for delay. I've been experimenting with that eve since I could return 
to this issue. Mostly about composition of project since whenever I try 
separate your sources from projects and use them individually I§ve been 
getting tons of errors (linking one included). In the end I get along with 
static library linked in my project. But putting that aside.

I haven't get as far as to use all your example. I am stuck and fact that 
when asking pool for specific message I am getting empty default descriptor 
as result. I am attaching my code so far could you please advise where I am 
making mistake?

Its just short program where I was trying to figure out how to use it.

#include io.h
#include fcntl.h
#include stdio.h
#include string
#include vector
#include descriptor.h
#include descriptor.pb.h
#include zero_copy_stream_impl.h
#include tokenizer.h
#include parser.h
#include substitute.h
#include dynamic_message.h

using namespace google::protobuf;

class ParserErrorCollector : public io::ErrorCollector
{
public:
ParserErrorCollector() {}
~ParserErrorCollector() {}

std::string text_;

// implements ErrorCollector -
void AddError(int line, int column, const std::string message)
{
strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column + 1, 
message);
}

void AddWarning(int line, int column, const std::string message)
{
AddError(line, column, WARNING: + message);
}
};

bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);

void main()
{
std::string strFile = 
C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto;

// Build the descriptor pool
DescriptorPool pool;

FileDescriptorProto proto;
ParseProtoFile(strFile, proto);
pool.BuildFile(proto);

// Suppose you want to parse a message type with a specific type name.
const Descriptor* descriptor = pool.FindMessageTypeByName(AddressBook);
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
}

// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
{
  int file_descriptor = open(strFileName.c_str(), O_RDONLY);

ParserErrorCollector errCollector;

io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
compiler::Parser parser;

return parser.Parse(tokenizer, result);

return true;
}


On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:

 Protobuf supports creating message types dynamically at runtime and use 
 them for parsing/serialization/etc.

 First you need to build up a DescriptorPool 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
  
 that contains all types that you may want to use. There are two approaches 
 to construct this pool. One is to call DescriptorPool::BuildFile() directly 
 with parsed proto files. For example:
   // Convert .proto files into parsed FileDescriptorProto
   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
 FileInputStream stream(filename);
 google::protobuf::io::Tokenizer tokenizer(stream);
 google::protobuf::compiler::Parser parser;
 return parser.Parse(tokenizer, result);
   }
   // Build the descriptor pool
   DescriptorPool pool;
   for (string filename : proto_files) {
 FileDescriptorProto proto;
 ParseProtoFile(filename, proto);
 pool.BuildFile(proto);
   }

 After you have the pool, you can query for a type by its name. For 
 example, DescriptorPool::FindMessageTypeByName().

 Then to actually parse/serialize/use message types in the pool, you need 
 to construct message objects around them. DynamicMessage 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
  
 is used for that:
   // Suppose you want to parse a message type with a specific type name.
   Descriptor* descriptor = 
 pool.FindMessageTypeByName(message_type_to_parse);
   DynamicMessageFactory factory;
   unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
   // Use the message object for parsing/etc.
   message-ParseFromString(input_data);
   // Access a specific field in the message
   FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
   switch (field-type()) {
 case TYPE_INT32: message-GetReflection()-GetInt32(*message, field); 
 break;
 ...
   }

 On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi,

 I have very specific problem. I have data and proto file available and my 
 application should take both and based on external configuration determine 
 how to interpret data (many different types/messages in proto). Yet that 
 can be determine only during run. My question is if there is any support 
 for that, I mean that I will be able to parse proto and decode data based 
 on content of interpret intermediate structures.

 I have been trying to analyze this possibility

Re: [protobuf] Dynamic/run-time decoding

2014-08-12 Thread 'Feng Xiao' via Protocol Buffers

Protobuf supports creating message types dynamically at runtime and use
them for parsing/serialization/etc.

First you need to build up a DescriptorPool
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
that contains all types that you may want to use. There are two approaches
to construct this pool. One is to call DescriptorPool::BuildFile() directly
with parsed proto files. For example:
  // Convert .proto files into parsed FileDescriptorProto
  bool ParseProtoFile(string filename, FileDescriptorProto* result) {
FileInputStream stream(filename);
google::protobuf::io::Tokenizer tokenizer(stream);
google::protobuf::compiler::Parser parser;
return parser.Parse(tokenizer, result);
  }
  // Build the descriptor pool
  DescriptorPool pool;
  for (string filename : proto_files) {
FileDescriptorProto proto;
ParseProtoFile(filename, proto);
pool.BuildFile(proto);
  }

After you have the pool, you can query for a type by its name. For example,
DescriptorPool::FindMessageTypeByName().

Then to actually parse/serialize/use message types in the pool, you need to
construct message objects around them. DynamicMessage
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
is used for that:
  // Suppose you want to parse a message type with a specific type name.
  Descriptor* descriptor =
pool.FindMessageTypeByName(message_type_to_parse);
  DynamicMessageFactory factory;
  unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
  // Use the message object for parsing/etc.
  message-ParseFromString(input_data);
  // Access a specific field in the message
  FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
  switch (field-type()) {
case TYPE_INT32: message-GetReflection()-GetInt32(*message, field);
break;
...
  }

On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.kyjov...@tieto.com
wrote:

 Hi,

 I have very specific problem. I have data and proto file available and my
 application should take both and based on external configuration determine
 how to interpret data (many different types/messages in proto). Yet that
 can be determine only during run. My question is if there is any support
 for that, I mean that I will be able to parse proto and decode data based
 on content of interpret intermediate structures.

 I have been trying to analyze this possibility directly from codes but not
 with much success. I would be glad for any guidance.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

2014-08-12 Thread Jan Kyjovský

Thank you for advice. I'll try it as soon as I am done with other task 
currently at hand.

On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:

 Protobuf supports creating message types dynamically at runtime and use 
 them for parsing/serialization/etc.

 First you need to build up a DescriptorPool 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
  
 that contains all types that you may want to use. There are two approaches 
 to construct this pool. One is to call DescriptorPool::BuildFile() directly 
 with parsed proto files. For example:
   // Convert .proto files into parsed FileDescriptorProto
   bool ParseProtoFile(string filename, FileDescriptorProto* result) {
 FileInputStream stream(filename);
 google::protobuf::io::Tokenizer tokenizer(stream);
 google::protobuf::compiler::Parser parser;
 return parser.Parse(tokenizer, result);
   }
   // Build the descriptor pool
   DescriptorPool pool;
   for (string filename : proto_files) {
 FileDescriptorProto proto;
 ParseProtoFile(filename, proto);
 pool.BuildFile(proto);
   }

 After you have the pool, you can query for a type by its name. For 
 example, DescriptorPool::FindMessageTypeByName().

 Then to actually parse/serialize/use message types in the pool, you need 
 to construct message objects around them. DynamicMessage 
 https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
  
 is used for that:
   // Suppose you want to parse a message type with a specific type name.
   Descriptor* descriptor = 
 pool.FindMessageTypeByName(message_type_to_parse);
   DynamicMessageFactory factory;
   unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
   // Use the message object for parsing/etc.
   message-ParseFromString(input_data);
   // Access a specific field in the message
   FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
   switch (field-type()) {
 case TYPE_INT32: message-GetReflection()-GetInt32(*message, field); 
 break;
 ...
   }

 On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.ky...@tieto.com 
 javascript: wrote:

 Hi,

 I have very specific problem. I have data and proto file available and my 
 application should take both and based on external configuration determine 
 how to interpret data (many different types/messages in proto). Yet that 
 can be determine only during run. My question is if there is any support 
 for that, I mean that I will be able to parse proto and decode data based 
 on content of interpret intermediate structures.

 I have been trying to analyze this possibility directly from codes but 
 not with much success. I would be glad for any guidance.

 -- 
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to protobuf+u...@googlegroups.com javascript:.
 To post to this group, send email to prot...@googlegroups.com 
 javascript:.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

Re: [protobuf] Dynamic/run-time decoding

29 matches

Site Navigation

Mail list logo

Footer information