Re: [protobuf] Dynamic/run-time decoding
Hello,
have you any luck finding where there may be problem?
On Monday, 15 June 2015 08:27:30 UTC+2, Jan Kyjovský wrote:
It has grown a bit since I started so I will attach zipped whole project.
On Friday, 12 June 2015 08:47:59 UTC+2, 肖锋 wrote:
On Wednesday, June 10, 2015 at 9:49:03 PM UTC-7, Jan Kyjovský wrote:
I did as you told me. And tried to compile my proto files with protoc.
Well it eat them up without any problem so I guess that clear that there is
no problem with them but in that case why is it causing such error to me.
As I gave some hint before problem is happening when I am trying to do
following with phone.proto which is being added as second into descriptor
database
compiler::Parser parser;
parser.RecordErrorsTo(errCollector);
bool bV = parser.Parse(tokenizer, result);
std::string strErr = errCollector.GetErrorAndWarnings();
return bV;
Parse returns false and error collector holds following error:
1:1: ERROR: Multiple package definitions.
Do you have nay idea what that could meant?
Could you post the whole content of your code? It would be easier for me
to figure out what's wrong if I can run the code myself.
On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:
On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com
wrote:
I have remade code so I now add into descriptor database all
protofiles specified by directory. There is but little problem with error
handling. From nature how I am handling errors it seems that message is
possible to decode but there is one error in error collector which is
bothering me:
1:1: ERROR: Multiple package definitions.
*My protos are defined as follows:*
*addressbook_fragmented.proto:*
import phone.proto;
package tutorial;
option java_package = com.example.tutorial;
option java_outer_classname = AddressBookProtos;
message Person {
required string name = 1;
required int32 id = 2;// Unique ID number for this person.
optional string email = 3;
// enum PhoneType {
//MOBILE = 0;
//HOME = 1;
//WORK = 2;
// }
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
optional int32 age = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person person = 1;
}
*phone.proto:*
package tutorial;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
Even though there is error in error collector I can ignore it and
decode message successfully. Yet if I try to remove this package info
from
phone.proto itis not detecting any error but FindMessageTypeByName fails
to
find data type. I am suspecting since not all references for person data
type I am asking for are resolved it fails.
So I am confused how exactly should be these package keywords be used?
I don't see a problem in your use of package. You could try to run
protoc on these proto files and see if it reports any error. I suspect
it's
not the .proto file's problem.
On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:
On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com
wrote:
HI,
I somehow understand what you mean but still I lack in experience
with it. I tried to switch to descriptor database but still I got some
problems.
C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;
// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);
if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}
C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}
bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor
*)m_Pool-FindMessageTypeByName(strType);
Message *message =
m_MessageFactory.GetPrototype(descriptor)-New();
if (descriptor == NULL)
{
char szError[256];
sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
bool bFlag = message-ParseFromString(strWireFormat);
if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}
return true;
}
bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);
if (file_descriptor == -1)
{
char szError[256];
sprintf(szError, Invalid proto file \%s\!,
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
C_ParserErrorCollector errCollector;
io::FileInputStream stream
Re: [protobuf] Dynamic/run-time decoding
I did as you told me. And tried to compile my proto files with protoc. Well
it eat them up without any problem so I guess that clear that there is no
problem with them but in that case why is it causing such error to me.
As I gave some hint before problem is happening when I am trying to do
following with phone.proto which is being added as second into descriptor
database
compiler::Parser parser;
parser.RecordErrorsTo(errCollector);
bool bV = parser.Parse(tokenizer, result);
std::string strErr = errCollector.GetErrorAndWarnings();
return bV;
Parse returns false and error collector holds following error:
1:1: ERROR: Multiple package definitions.
Do you have nay idea what that could meant?
On Tuesday, 9 June 2015 20:31:38 UTC+2, Feng Xiao wrote:
On Mon, Jun 8, 2015 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
I have remade code so I now add into descriptor database all protofiles
specified by directory. There is but little problem with error handling.
From nature how I am handling errors it seems that message is possible to
decode but there is one error in error collector which is bothering me:
1:1: ERROR: Multiple package definitions.
*My protos are defined as follows:*
*addressbook_fragmented.proto:*
import phone.proto;
package tutorial;
option java_package = com.example.tutorial;
option java_outer_classname = AddressBookProtos;
message Person {
required string name = 1;
required int32 id = 2;// Unique ID number for this person.
optional string email = 3;
// enum PhoneType {
//MOBILE = 0;
//HOME = 1;
//WORK = 2;
// }
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
optional int32 age = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person person = 1;
}
*phone.proto:*
package tutorial;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
Even though there is error in error collector I can ignore it and decode
message successfully. Yet if I try to remove this package info from
phone.proto itis not detecting any error but FindMessageTypeByName fails to
find data type. I am suspecting since not all references for person data
type I am asking for are resolved it fails.
So I am confused how exactly should be these package keywords be used?
I don't see a problem in your use of package. You could try to run
protoc on these proto files and see if it reports any error. I suspect it's
not the .proto file's problem.
On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:
On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com
wrote:
HI,
I somehow understand what you mean but still I lack in experience with
it. I tried to switch to descriptor database but still I got some problems.
C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;
// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);
if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}
C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}
bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor
*)m_Pool-FindMessageTypeByName(strType);
Message *message = m_MessageFactory.GetPrototype(descriptor)-New();
if (descriptor == NULL)
{
char szError[256];
sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
bool bFlag = message-ParseFromString(strWireFormat);
if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}
return true;
}
bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);
if (file_descriptor == -1)
{
char szError[256];
sprintf(szError, Invalid proto file \%s\!,
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
C_ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
result-set_name(m_strProtoFile);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
I have used SimpleDescriptorDatabase since it looked like it has
already implemented and its covering what I will be requiring from it (or
at least I think it does)
If you use SimpleDescriptorDatabase, you need to add all .proto files
Re: [protobuf] Dynamic/run-time decoding
I have remade code so I now add into descriptor database all protofiles
specified by directory. There is but little problem with error handling.
From nature how I am handling errors it seems that message is possible to
decode but there is one error in error collector which is bothering me:
1:1: ERROR: Multiple package definitions.
*My protos are defined as follows:*
*addressbook_fragmented.proto:*
import phone.proto;
package tutorial;
option java_package = com.example.tutorial;
option java_outer_classname = AddressBookProtos;
message Person {
required string name = 1;
required int32 id = 2;// Unique ID number for this person.
optional string email = 3;
// enum PhoneType {
//MOBILE = 0;
//HOME = 1;
//WORK = 2;
// }
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
optional int32 age = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person person = 1;
}
*phone.proto:*
package tutorial;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
Even though there is error in error collector I can ignore it and decode
message successfully. Yet if I try to remove this package info from
phone.proto itis not detecting any error but FindMessageTypeByName fails to
find data type. I am suspecting since not all references for person data
type I am asking for are resolved it fails.
So I am confused how exactly should be these package keywords be used?
On Wednesday, 3 June 2015 19:05:59 UTC+2, Feng Xiao wrote:
On Wed, Jun 3, 2015 at 12:40 AM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
HI,
I somehow understand what you mean but still I lack in experience with
it. I tried to switch to descriptor database but still I got some problems.
C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;
// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);
if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}
C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}
bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor
*)m_Pool-FindMessageTypeByName(strType);
Message *message = m_MessageFactory.GetPrototype(descriptor)-New();
if (descriptor == NULL)
{
char szError[256];
sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
bool bFlag = message-ParseFromString(strWireFormat);
if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}
return true;
}
bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);
if (file_descriptor == -1)
{
char szError[256];
sprintf(szError, Invalid proto file \%s\!,
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
C_ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
result-set_name(m_strProtoFile);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
I have used SimpleDescriptorDatabase since it looked like it has already
implemented and its covering what I will be requiring from it (or at least
I think it does)
If you use SimpleDescriptorDatabase, you need to add all .proto files into
the database explicitly. For example, if you have 2 .proto files foo.proto
and bar.proto, where bar.proto imports foo.proto, you need to add both
foo.proto and bar.proto to SimpleDescriptorDatabase. If you already have a
list of all proto files, using SimpleDescriptorDatabase should be fine.
Everything is fine until I try to get descriptor by FindMessageTypeByName
from pool. It returns null. Am I committing some steps?
I don't see anything obviously wrong in your code.
On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:
On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com
wrote:
Hi again,
As I have stated before I am done with decoding, but now I am solving
different type of problem. As I have mentioned before imports may prove
problematic to our implementation.
Let me describe use-case how it will be used. There will be
configuration stating for which data (message number, specific field) will
be applied which proto file and structure (data type). Therefore there is
no knowledge about other proto files up until now
Re: [protobuf] Dynamic/run-time decoding
HI,
I somehow understand what you mean but still I lack in experience with it.
I tried to switch to descriptor database but still I got some problems.
C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
m_strProtoFile = strFile;
// Build the descriptor pool
m_bConstructed = ParseProtoFile(m_ProtoFileDescr);
if (m_bConstructed)
{
m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
m_Pool = new DescriptorPool(m_ProtoDescrDatabase);
}
}
C_ProtoDecoder::~C_ProtoDecoder()
{
delete m_Pool;
}
bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string
strWireFormat, std::string strType)
{
Descriptor *descriptor = (Descriptor
*)m_Pool-FindMessageTypeByName(strType);
Message *message = m_MessageFactory.GetPrototype(descriptor)-New();
if (descriptor == NULL)
{
char szError[256];
sprintf(szError, Unknown data type %s!, strType.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
bool bFlag = message-ParseFromString(strWireFormat);
if (!bFlag)
{
m_InternalPrinter.AddErrorMessage(Encoding error!);
}
else
{
m_Transformator.MorphToStructs(*message);
}
return true;
}
bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);
if (file_descriptor == -1)
{
char szError[256];
sprintf(szError, Invalid proto file \%s\!,
m_strProtoFile.c_str());
m_InternalPrinter.AddErrorMessage(szError);
return false;
}
C_ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
result-set_name(m_strProtoFile);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
I have used SimpleDescriptorDatabase since it looked like it has already
implemented and its covering what I will be requiring from it (or at least
I think it does)
Everything is fine until I try to get descriptor by FindMessageTypeByName
from pool. It returns null. Am I committing some steps?
On Tuesday, 2 June 2015 19:49:53 UTC+2, Feng Xiao wrote:
On Mon, Jun 1, 2015 at 9:55 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi again,
As I have stated before I am done with decoding, but now I am solving
different type of problem. As I have mentioned before imports may prove
problematic to our implementation.
Let me describe use-case how it will be used. There will be configuration
stating for which data (message number, specific field) will be applied
which proto file and structure (data type). Therefore there is no knowledge
about other proto files up until now. In run time data are fetched and
shipped to decoder. In current implementation is specified proto file
loaded and specific type used for decoding. That much is clear and works.
But if some parts of structures are imported. It will fail. That much is
clear that using just one file descriptor will be not enough as you have
mentioned earlier.
That leads to your proposition (posted earlier) to use
DescriptorDatabase. I have looked on implementation of classes derived from
DscriptorDatabase. I am more or less able to provide directory for
SourceTree, but that doesn't answer question about import. I mean since I
am aware of only one proto file. Is import done automatically or other
files has to be also included manually into database?
Another question I have is bout order of operations. You mentioned that I
have to first call FindFileByName() before callingFindMessageTypeByName().
That may be problem since I am not aware in which proto file may be located.
Also I have noticed in code note that proto files with types used in
other proto files have to be loaded first. So is there any way how to
ensure right order of loading?
Most of the work is done by DescriptorPool. Basically when you call
FindFileByName() on a .proto file that is not yet loaded into the pool,
DescriptorPool will call FindFileByName() on its underlying
DescriptorDatabase to get the FileDescriptorProto for file. Then
DescriptorPool will check the imports of this FileDescriptorPool. If it
finds unsatisfied imports (i.e., files imported but not yet in the pool),
it will issue FindFileByName() to the DescriptorDatabase again for these
files.
So basically DescriptorDatabase only needs to know how to load a single
.proto file, while DescriptorPool will take care of the imports and
ordering.
Depending on what DescriptorDatabase you are using, you may or may not
need to call FindFileByName() first. If the DescriptorDatabase has
implement FindFileContainingSymbol, then calling
DescriptorPool::FindMessageTypeByName() directly will be able to find the
file using this method. If not (the case with
SourceTreeDescriptorDatabase
Re: [protobuf] Dynamic/run-time decoding
Hi again, As I have stated before I am done with decoding, but now I am solving different type of problem. As I have mentioned before imports may prove problematic to our implementation. Let me describe use-case how it will be used. There will be configuration stating for which data (message number, specific field) will be applied which proto file and structure (data type). Therefore there is no knowledge about other proto files up until now. In run time data are fetched and shipped to decoder. In current implementation is specified proto file loaded and specific type used for decoding. That much is clear and works. But if some parts of structures are imported. It will fail. That much is clear that using just one file descriptor will be not enough as you have mentioned earlier. That leads to your proposition (posted earlier) to use DescriptorDatabase. I have looked on implementation of classes derived from DscriptorDatabase. I am more or less able to provide directory for SourceTree, but that doesn't answer question about import. I mean since I am aware of only one proto file. Is import done automatically or other files has to be also included manually into database? Another question I have is bout order of operations. You mentioned that I have to first call FindFileByName() before callingFindMessageTypeByName(). That may be problem since I am not aware in which proto file may be located. Also I have noticed in code note that proto files with types used in other proto files have to be loaded first. So is there any way how to ensure right order of loading? I hope my question are not too strange but I my knowledge about google protocol buffers are a bit limited. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Never mind then. We can provide results in two formats. Offsets and sizes are requirement of one of these and after giving it some thought I get the feeling that it would be better to use d second format (formatted text), also it would require much less work. Thank you very much four your help. This time it is really end of the line. I am able to decode data in run-time and that's what was desired from the start. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi,
yes thank you for that hint. I was able to get enough references to go
through decoded data. I am more or less efficiently able to get values (in
case of ENUM also original value not only symbolic meaning) but now I would
require some auxiliary information which I am not sure are contained in
these objects. I will need to get access to current byte offset (from data
start) and size of element. Although size is something I can probably
somehow derive. Is there any possibility to that?
On Thursday, 7 May 2015 19:44:22 UTC+2, Feng Xiao wrote:
On Thu, May 7, 2015 at 2:29 AM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi,
sorry for asking for advice so soon again.
I have progressed a bit futher and I am now trying to display decoded
data. For that purpose I have prepared some data and tried to decode them
but I have encountered problem with repeated structures and with nested
messages. I dont know how to get count of repeats in case of repeated
fields so its problematic to address them. In case of nested messages I
have experiencing difficulties how to get message.
Here is function I have made so far. Note that I am suspecting that it
will probably need some more tweaking so it can be called recursively (work
still in progress). For now I am satisfied just by displaying data on first
level.
void ExpandSubMessage(Message *message, Descriptor *descriptor)
{
for (int i = 0; i descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal = message-GetReflection()-GetInt32(*message,
field);
printf(%s = %d\n, field-name().c_str(), nVal);
break;
}
case FieldDescriptor::TYPE_STRING:
{
std::string strVal =
message-GetReflection()-GetString(*message, field);
printf(%s = %s\n, field-name().c_str(),
strVal.c_str());
break;
}
//case FieldDescriptor::TYPE_ENUM:
message-GetReflection()-GetEnum(*message, field); break;
case FieldDescriptor::TYPE_MESSAGE:
{
Message *messVal;
if (field-is_repeated())
{
message-GetReflection()-GetRepeatedMessage(*message, field, 1);
}
else
{
message-GetReflection()-GetMessage(*messVal, field);
}
printf(%s = %s\n, field-name().c_str(),
/*messVal.DebugString().c_str()*/Test);
break;
}
}
}
}
I have tried to look for details in specification on your pages but
didn't find anything that would answer my question.
May I once again ask for your assistance?
What you are trying to do looks similar to what the TextFormat class does.
You can refer to that class as an example to learn how to use protobuf
reflection to traverse a message tree:
https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1449
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+u...@googlegroups.com javascript:.
To post to this group, send email to prot...@googlegroups.com
javascript:.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi,
sorry for asking for advice so soon again.
I have progressed a bit futher and I am now trying to display decoded data.
For that purpose I have prepared some data and tried to decode them but I
have encountered problem with repeated structures and with nested messages.
I dont know how to get count of repeats in case of repeated fields so its
problematic to address them. In case of nested messages I have experiencing
difficulties how to get message.
Here is function I have made so far. Note that I am suspecting that it will
probably need some more tweaking so it can be called recursively (work
still in progress). For now I am satisfied just by displaying data on first
level.
void ExpandSubMessage(Message *message, Descriptor *descriptor)
{
for (int i = 0; i descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal = message-GetReflection()-GetInt32(*message,
field);
printf(%s = %d\n, field-name().c_str(), nVal);
break;
}
case FieldDescriptor::TYPE_STRING:
{
std::string strVal =
message-GetReflection()-GetString(*message, field);
printf(%s = %s\n, field-name().c_str(), strVal.c_str());
break;
}
//case FieldDescriptor::TYPE_ENUM:
message-GetReflection()-GetEnum(*message, field); break;
case FieldDescriptor::TYPE_MESSAGE:
{
Message *messVal;
if (field-is_repeated())
{
message-GetReflection()-GetRepeatedMessage(*message,
field, 1);
}
else
{
message-GetReflection()-GetMessage(*messVal, field);
}
printf(%s = %s\n, field-name().c_str(),
/*messVal.DebugString().c_str()*/Test);
break;
}
}
}
}
I have tried to look for details in specification on your pages but didn't
find anything that would answer my question.
May I once again ask for your assistance?
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi,
This is most likely last post. After converting these example data to
binary and making sure that that parsing was successful (return true), I am
finally getting decoded data. From this point on I can finish what was
requested.
Once again thank you for all your advices and patience.
On Tuesday, 5 May 2015 19:35:03 UTC+2, Feng Xiao wrote:
On Mon, May 4, 2015 at 11:36 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi,
so we are somehow back in game. And now I am getting familiar what I have
done more then half a year ago.
So for now I got some basic structure of code that will later be as a
starting point for real implementation. I am trying to do some simple
experiments to see how it works. Though I got problem that even if I get
descriptor I am not getting any data. I have used example from encoding
specification since I needed something really simple.
...
const Descriptor* descriptor =
pool.FindMessageTypeByName(tutorial.Test1);
if (descriptor == NULL)
{
return;
}
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
std::string input_data = 089601;
ParseFromString() expects protobuf binary format data (stored in a C++
string), which I believe 089601 is not.
message-ParseFromString(input_data);
You could check the return value of ParseFromString() to see whether the
parsing is a success.
// Access a specific field in the message
for (int i = 0; i descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal =
message-GetReflection()-GetInt32(*message, field);
printf(%d\n, nVal);
break;
}
...
But from reflection I am getting that value is 0. So either I have
wrongly set input data considering that I misunderstand format or there is
something else. Do you have any idea what may be cause?
Thank you in advance for your reply.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+u...@googlegroups.com javascript:.
To post to this group, send email to prot...@googlegroups.com
javascript:.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi,
so we are somehow back in game. And now I am getting familiar what I have
done more then half a year ago.
So for now I got some basic structure of code that will later be as a
starting point for real implementation. I am trying to do some simple
experiments to see how it works. Though I got problem that even if I get
descriptor I am not getting any data. I have used example from encoding
specification since I needed something really simple.
...
const Descriptor* descriptor =
pool.FindMessageTypeByName(tutorial.Test1);
if (descriptor == NULL)
{
return;
}
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
std::string input_data = 089601;
message-ParseFromString(input_data);
// Access a specific field in the message
for (int i = 0; i descriptor-field_count(); i++)
{
const FieldDescriptor* field = descriptor-field(i);
switch (field-type())
{
case FieldDescriptor::TYPE_INT32:
{
int nVal = message-GetReflection()-GetInt32(*message,
field);
printf(%d\n, nVal);
break;
}
...
But from reflection I am getting that value is 0. So either I have wrongly
set input data considering that I misunderstand format or there is
something else. Do you have any idea what may be cause?
Thank you in advance for your reply.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi, due to organization changes and budget costs this task is put on hold. Thank you for all your guidance up until now. If any further help is needed regarding this issue in the future me or someone else will continue with this thread. On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote: On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com javascript: wrote: Hi, Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our. Now I got different sort of questions. Since I can now progress further I am finding new problems. As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format? ParseFromString() accepts protobuf wire-format data (which is the output of protobuf serialization routines). I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other. You can use the DescriptorDatabase implementation we use for protocol compiler: https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76 Basically rather than adding individual files to the pool, you provide a DescriptorDatabase to the pool and the pool will use this DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of this approach is that, you need to call DescriptorPool::FindFileByName() on the .proto files before you can use DescriptorPool::FindMessageTypeByName() to search for types. Thank you in advance for your response. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Hi, due to organization changes and budget cuts this task is put on hold. Thank you for all your guidance up until now. If any further help is needed regarding this issue in the future me or someone else will continue with this thread. On Thursday, 11 September 2014 20:00:52 UTC+2, Feng Xiao wrote: On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský jan.ky...@tieto.com javascript: wrote: Hi, Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our. Now I got different sort of questions. Since I can now progress further I am finding new problems. As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format? ParseFromString() accepts protobuf wire-format data (which is the output of protobuf serialization routines). I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other. You can use the DescriptorDatabase implementation we use for protocol compiler: https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76 Basically rather than adding individual files to the pool, you provide a DescriptorDatabase to the pool and the pool will use this DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of this approach is that, you need to call DescriptorPool::FindFileByName() on the .proto files before you can use DescriptorPool::FindMessageTypeByName() to search for types. Thank you in advance for your response. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com javascript:. To post to this group, send email to prot...@googlegroups.com javascript:. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Variable proto contains some information about structure although
proto.name is blank (don't know why).
BuildFile returns NULL as well as FindMessageTypeByName. Whey I went deeper
in debugging I noticed that BuildFile is trying to look up for that file I
provide in its tables but these are empty as well.
On Tuesday, 9 September 2014 22:35:05 UTC+2, Feng Xiao wrote:
On Mon, Sep 8, 2014 at 10:32 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi,
sorry for delay. I've been experimenting with that eve since I could
return to this issue. Mostly about composition of project since whenever I
try separate your sources from projects and use them individually I§ve been
getting tons of errors (linking one included). In the end I get along with
static library linked in my project. But putting that aside.
I haven't get as far as to use all your example. I am stuck and fact that
when asking pool for specific message I am getting empty default descriptor
as result. I am attaching my code so far could you please advise where I am
making mistake?
Its just short program where I was trying to figure out how to use it.
#include io.h
#include fcntl.h
#include stdio.h
#include string
#include vector
#include descriptor.h
#include descriptor.pb.h
#include zero_copy_stream_impl.h
#include tokenizer.h
#include parser.h
#include substitute.h
#include dynamic_message.h
using namespace google::protobuf;
class ParserErrorCollector : public io::ErrorCollector
{
public:
ParserErrorCollector() {}
~ParserErrorCollector() {}
std::string text_;
// implements ErrorCollector -
void AddError(int line, int column, const std::string message)
{
strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column + 1,
message);
}
void AddWarning(int line, int column, const std::string message)
{
AddError(line, column, WARNING: + message);
}
};
bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);
void main()
{
std::string strFile =
C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto;
// Build the descriptor pool
DescriptorPool pool;
FileDescriptorProto proto;
ParseProtoFile(strFile, proto);
pool.BuildFile(proto);
Have you checked the return value of these two method calls?
// Suppose you want to parse a message type with a specific type name.
const Descriptor* descriptor = pool.FindMessageTypeByName(AddressBook);
Does this return NULL or non-NULL? If it's the later, it means you have
successfully constructed the descriptor for AddressBook.
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
}
// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
{
int file_descriptor = open(strFileName.c_str(), O_RDONLY);
ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
return true;
}
On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:
Protobuf supports creating message types dynamically at runtime and use
them for parsing/serialization/etc.
First you need to build up a DescriptorPool
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
that contains all types that you may want to use. There are two approaches
to construct this pool. One is to call DescriptorPool::BuildFile() directly
with parsed proto files. For example:
// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(string filename, FileDescriptorProto* result) {
FileInputStream stream(filename);
google::protobuf::io::Tokenizer tokenizer(stream);
google::protobuf::compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
// Build the descriptor pool
DescriptorPool pool;
for (string filename : proto_files) {
FileDescriptorProto proto;
ParseProtoFile(filename, proto);
pool.BuildFile(proto);
}
After you have the pool, you can query for a type by its name. For
example, DescriptorPool::FindMessageTypeByName().
Then to actually parse/serialize/use message types in the pool, you need
to construct message objects around them. DynamicMessage
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
is used for that:
// Suppose you want to parse a message type with a specific type name.
Descriptor* descriptor = pool.FindMessageTypeByName(
message_type_to_parse);
DynamicMessageFactory factory;
unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
message-ParseFromString(input_data);
// Access a specific field in the message
FieldDescriptor* field
Re: [protobuf] Dynamic/run-time decoding
Hi,
sorry for delay. I've been experimenting with that eve since I could return
to this issue. Mostly about composition of project since whenever I try
separate your sources from projects and use them individually I§ve been
getting tons of errors (linking one included). In the end I get along with
static library linked in my project. But putting that aside.
I haven't get as far as to use all your example. I am stuck and fact that
when asking pool for specific message I am getting empty default descriptor
as result. I am attaching my code so far could you please advise where I am
making mistake?
Its just short program where I was trying to figure out how to use it.
#include io.h
#include fcntl.h
#include stdio.h
#include string
#include vector
#include descriptor.h
#include descriptor.pb.h
#include zero_copy_stream_impl.h
#include tokenizer.h
#include parser.h
#include substitute.h
#include dynamic_message.h
using namespace google::protobuf;
class ParserErrorCollector : public io::ErrorCollector
{
public:
ParserErrorCollector() {}
~ParserErrorCollector() {}
std::string text_;
// implements ErrorCollector -
void AddError(int line, int column, const std::string message)
{
strings::SubstituteAndAppend(text_, $0:$1: $2\n, line + 1, column + 1,
message);
}
void AddWarning(int line, int column, const std::string message)
{
AddError(line, column, WARNING: + message);
}
};
bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);
void main()
{
std::string strFile =
C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto;
// Build the descriptor pool
DescriptorPool pool;
FileDescriptorProto proto;
ParseProtoFile(strFile, proto);
pool.BuildFile(proto);
// Suppose you want to parse a message type with a specific type name.
const Descriptor* descriptor = pool.FindMessageTypeByName(AddressBook);
DynamicMessageFactory factory;
Message *message = factory.GetPrototype(descriptor)-New();
}
// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)
{
int file_descriptor = open(strFileName.c_str(), O_RDONLY);
ParserErrorCollector errCollector;
io::FileInputStream stream(file_descriptor);
stream.SetCloseOnDelete(true);
io::Tokenizer tokenizer(stream, errCollector);
compiler::Parser parser;
return parser.Parse(tokenizer, result);
return true;
}
On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:
Protobuf supports creating message types dynamically at runtime and use
them for parsing/serialization/etc.
First you need to build up a DescriptorPool
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
that contains all types that you may want to use. There are two approaches
to construct this pool. One is to call DescriptorPool::BuildFile() directly
with parsed proto files. For example:
// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(string filename, FileDescriptorProto* result) {
FileInputStream stream(filename);
google::protobuf::io::Tokenizer tokenizer(stream);
google::protobuf::compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
// Build the descriptor pool
DescriptorPool pool;
for (string filename : proto_files) {
FileDescriptorProto proto;
ParseProtoFile(filename, proto);
pool.BuildFile(proto);
}
After you have the pool, you can query for a type by its name. For
example, DescriptorPool::FindMessageTypeByName().
Then to actually parse/serialize/use message types in the pool, you need
to construct message objects around them. DynamicMessage
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
is used for that:
// Suppose you want to parse a message type with a specific type name.
Descriptor* descriptor =
pool.FindMessageTypeByName(message_type_to_parse);
DynamicMessageFactory factory;
unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
message-ParseFromString(input_data);
// Access a specific field in the message
FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
switch (field-type()) {
case TYPE_INT32: message-GetReflection()-GetInt32(*message, field);
break;
...
}
On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi,
I have very specific problem. I have data and proto file available and my
application should take both and based on external configuration determine
how to interpret data (many different types/messages in proto). Yet that
can be determine only during run. My question is if there is any support
for that, I mean that I will be able to parse proto and decode data based
on content of interpret intermediate structures.
I have been trying to analyze this possibility
[protobuf] Dynamic/run-time decoding
Hi, I have very specific problem. I have data and proto file available and my application should take both and based on external configuration determine how to interpret data (many different types/messages in proto). Yet that can be determine only during run. My question is if there is any support for that, I mean that I will be able to parse proto and decode data based on content of interpret intermediate structures. I have been trying to analyze this possibility directly from codes but not with much success. I would be glad for any guidance. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Dynamic/run-time decoding
Thank you for advice. I'll try it as soon as I am done with other task
currently at hand.
On Tuesday, 12 August 2014 19:36:13 UTC+2, Feng Xiao wrote:
Protobuf supports creating message types dynamically at runtime and use
them for parsing/serialization/etc.
First you need to build up a DescriptorPool
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.h#1141
that contains all types that you may want to use. There are two approaches
to construct this pool. One is to call DescriptorPool::BuildFile() directly
with parsed proto files. For example:
// Convert .proto files into parsed FileDescriptorProto
bool ParseProtoFile(string filename, FileDescriptorProto* result) {
FileInputStream stream(filename);
google::protobuf::io::Tokenizer tokenizer(stream);
google::protobuf::compiler::Parser parser;
return parser.Parse(tokenizer, result);
}
// Build the descriptor pool
DescriptorPool pool;
for (string filename : proto_files) {
FileDescriptorProto proto;
ParseProtoFile(filename, proto);
pool.BuildFile(proto);
}
After you have the pool, you can query for a type by its name. For
example, DescriptorPool::FindMessageTypeByName().
Then to actually parse/serialize/use message types in the pool, you need
to construct message objects around them. DynamicMessage
https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/dynamic_message.h#53
is used for that:
// Suppose you want to parse a message type with a specific type name.
Descriptor* descriptor =
pool.FindMessageTypeByName(message_type_to_parse);
DynamicMessageFactory factory;
unique_ptrMessage message = factory.GetPrototype(descriptor)-New();
// Use the message object for parsing/etc.
message-ParseFromString(input_data);
// Access a specific field in the message
FieldDescriptor* field = descriptor-FindFieldByName(field_to_read);
switch (field-type()) {
case TYPE_INT32: message-GetReflection()-GetInt32(*message, field);
break;
...
}
On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský jan.ky...@tieto.com
javascript: wrote:
Hi,
I have very specific problem. I have data and proto file available and my
application should take both and based on external configuration determine
how to interpret data (many different types/messages in proto). Yet that
can be determine only during run. My question is if there is any support
for that, I mean that I will be able to parse proto and decode data based
on content of interpret intermediate structures.
I have been trying to analyze this possibility directly from codes but
not with much success. I would be glad for any guidance.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+u...@googlegroups.com javascript:.
To post to this group, send email to prot...@googlegroups.com
javascript:.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
