Hi Jason,
I was out of this discussion for quite some time, but during this time I was
able to make my large application's design work with dynamic messages using
the protobuf::Importer. However, I am now stuck with the issue of
indexing/sorting and searching messages.
I understand that the code generated messages could be sorted with
stl::sort(itor:begin, itoe::end, functor), but, that's because there is a
RepeatedPtrField iterator and RepeatedPtrField list available through the
message class.
I have to sort/search on a repeated nested message declared in my main
message depending upon the fields of the nested message, like so:
message nested_msg
{
required uint64 indx =1; //sort on this field
optional uint64 classId =2; //sort on this field
optional string network_name = 3; //search on this field
}
message coverMessage
{
required uint64 nested_msg_count =1;
repeated nested_msg msg_instance =2;
}
Could this be achieved with dynamic messages through polymorphic message
pointers? My current work on protobuf serialization will remain an
experimentation otherwise.
P.S.: I hear you people are thinking on the design of protobuf maps. Will it
be possible that protobufs parent level messages be sorted/searched at
nested message levels through maps of nested message fields? Will '[key]'
and '[value]' operators be available for the maps?
Regards,
Himanshu
On Fri, Sep 24, 2010 at 6:02 PM, Himanshu Srivastava <him.s...@gmail.com>wrote:
> >>Gotcha. If you run protoc on the file produced by your generator class,
> does it succeed?
>
> protoc.exe succeeds parsing my proto file. I have emailed the file to you
> and you can verify that.
>
> >>>>descriptor_pool is of course NULL.
>
> >>I meant that you should check the return value of BuildFile(). If it
> returns NULL, then it could not build the file. The ErrorCollector >>would
> tell you why
> I meant the same, Build_File() returns Null.
>
> However, I will get back to this issue. It will take some time to verify
> issues in my implementation.
>
> Thanks & Regards,
> Himanshu
>
> On Thu, Sep 23, 2010 at 10:37 PM, Jason Hsueh <jas...@google.com> wrote:
>
>>
>>
>> On Thu, Sep 23, 2010 at 9:52 AM, Himanshu Srivastava <him.s...@gmail.com
>> > wrote:
>>
>>> Hi Jason,
>>>
>>> In code:
>>> //********************************************//
>>> DescriptorPool descriptor_pool;
>>>
>>> for (int i = 0; i < f.file_size(); ++i) {
>>>
>>> descriptor_pool.BuildFile(f.file(i));
>>> //********************************************//
>>> descriptor_pool is of course NULL.
>>>
>>
>> I meant that you should check the return value of BuildFile(). If it
>> returns NULL, then it could not build the file. The ErrorCollector would
>> tell you why.
>>
>>
>>> And for the Importer approach my code does SEG-FAULT.
>>>
>>> Further I do need conversion from proto file directly and also a static
>>> FileDescriptor so that it can be passed around for construction of my new
>>> messages in a recursive tree algorithm in another distinct class object.
>>>
>>
>> That's up to you: you'll need to ensure that the DescriptorPool has the
>> lifetime you want.
>>
>> Previously, you mentioned that the importer approach returned NULL. That
>> would be indicative of a problem in the .proto file: the file could not be
>> built. Even if you were to successfully build the file, your code would
>> segfault, because the FileDescriptor is no longer valid, as the Importer,
>> and thus the DescriptorPool containing the FileDescriptor, have been
>> deleted.
>>
>>
>>>
>>> I do not want the descriptor set approach, since that is only available
>>> when protoc exe is present.
>>> ---------------------
>>> >>I doubt that your code will work, as Importer should be looking for a
>>> file named "dynamicProtoMessages.proto", which probably >>does not exist.
>>> ------------------------
>>> I create the proto file "dynamicProtoMessages.proto" parmanently on disk
>>> and read it from disk to Parser after it is generated from my
>>> CXMLToProtoFileGenerator class.
>>>
>>
>> Gotcha. If you run protoc on the file produced by your generator class,
>> does it succeed?
>>
>>
>>>
>>> Thanks & Regards,
>>> Himanshu
>>>
>>>
>>> On Thu, Sep 23, 2010 at 9:39 PM, Jason Hsueh <jas...@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Sep 23, 2010 at 2:40 AM, Himanshu Srivastava <
>>>> him.s...@gmail.com> wrote:
>>>>
>>>>> Kenton:
>>>>>
>>>>>
>>>>> I really sent you a only a part of the complete RecordHeader message,
>>>>> since I was focusing on "nested message not getting parsed" issue for this
>>>>> particular message. Indeed my complete proto file is a large one. I
>>>>> have attached the truncated proto file containing this complete message
>>>>> only
>>>>> for reference with this mail (plese see attachment
>>>>> 'dynamicProtoMessages.proto '.
>>>>>
>>>>> I had used Parser and Importer earlier and generated the FileDescriptor
>>>>> in a class “CMessageParser” member function as below:
>>>>>
>>>>>
>>>>>
>>>>> //***********************************************************//
>>>>>
>>>>> class PacketTracerErrorFileErrorCollector:
>>>>> publicgoogle::protobuf::compiler::MultiFileErrorCollector
>>>>>
>>>>> {
>>>>>
>>>>> ...
>>>>>
>>>>> };
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> const google::protobuf::FileDescriptor* CMessageParser
>>>>> ::generateMessageFileDescriptor()
>>>>>
>>>>> {
>>>>>
>>>>> std::pair<std::string, std::string> filePair;
>>>>>
>>>>> google::protobuf::compiler::DiskSourceTree source_tree;
>>>>>
>>>>> source_tree.MapPath(/*filePair.first*/"", "."/*filePair.second*/);
>>>>>
>>>>> PacketTracerErrorFileErrorCollector error_collector;//Defined
>>>>> above
>>>>>
>>>>> google::protobuf::compiler::Importer importer(&source_tree,
>>>>> &error_collector);
>>>>>
>>>>> std::string outFileNameString = “dynamicProtoMessages.proto”;
>>>>> const google::protobuf::FileDescriptor* file_desc =
>>>>> importer.Import(outFileNameString);
>>>>> return file_desc;
>>>>>
>>>>> }
>>>>>
>>>>> //***********************************************************//
>>>>>
>>>>>
>>>>> However, the problem with that approach was, as soon as I returned the
>>>>> pointer from the FileDescriptor to a client class object of
>>>>> CMessageParser, the FileDescriptor* becomes NULL or invalid once out
>>>>> of the class and it becomes useless.
>>>>>
>>>> //***********************************************************//
>>>>>
>>>>> CMessageParser myParser;
>>>>>
>>>>> const google::protobuf::FileDescriptor *file_desc =
>>>>> myParser.generateMessageFileDescriptor();
>>>>> //file_desc is NULL
>>>>>
>>>>> //***********************************************************//
>>>>>
>>>>>
>>>>> So, I thought, I was following some wrong approach and to retain the
>>>>> File Descriptors for a long enough period across scopes, I may need the
>>>>> DescriptorPool, which might have the permanent lifetime for the period of
>>>>> the application.
>>>>>
>>>>
>>>> Importer just wraps DescriptorPool and a database of proto files.
>>>> Neither of these guarantee static duration for the descriptors - any
>>>> descriptors that you build will only remain alive while the
>>>> DescriptorPool/Importer remains alive.
>>>>
>>>> Of course, that means your code should seg-fault, not return NULL,
>>>> assuming that you successfully built the file. You should look at the error
>>>> collector to see what went wrong. I doubt that your code will work, as
>>>> Importer should be looking for a file named "dynamicProtoMessages.proto",
>>>> which probably does not exist.
>>>>
>>>> But it sounds like you don't actually need to read .proto files, and
>>>> instead you are generating the *DescriptorProto representations directly.
>>>> If
>>>> that's the case, you don't need Importer, you should directly use a
>>>> DescriptorPool as you did in the code snippet below.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Hi Jason:
>>>>>
>>>>> I am always able to generate the FileDescriptorSet easily, although the
>>>>> FileDescriptor was never showing completely populated fields when
>>>>> debugging
>>>>> with Visual studio 2008 Express edition.
>>>>>
>>>>> I have attached the generated FileDescriptorSet 'dynProtoMessage.dsc'
>>>>> file along with the proto file containing the above messages with this
>>>>> mail.
>>>>> You can see even this dsc file (made with simple messages as in the
>>>>> attached
>>>>> proto file) does not load through DescriptorPool::BuildFile().
>>>>>
>>>>>
>>>>>
>>>>> This is the function I used to load DescriptorSet:
>>>>>
>>>>>
>>>>>
>>>>> void CMessageParser
>>>>> ::getMessagesFromDescriptorSet(constgoogle::protobuf::FileDescriptor**
>>>>> new_file_desc)
>>>>>
>>>>> {
>>>>>
>>>>> using namespace google::protobuf;
>>>>>
>>>>> using namespace google::protobuf::io;
>>>>>
>>>>> using namespace google::protobuf::compiler;
>>>>>
>>>>> ifstream desc_file("dynProtoMessage.dsc",ios::in);
>>>>>
>>>>> if (desc_file.bad()) {
>>>>>
>>>>> std::cout << "desc file not read...exiting" <<
>>>>> std::endl;
>>>>>
>>>>> return;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> FileDescriptorSet f;
>>>>>
>>>>> f.ParseFromIstream(&desc_file);
>>>>>
>>>>>
>>>>>
>>>>> // DescriptorPoolErrorCollector errorCol;
>>>>> //google::protobuf::SimpleDescriptorDatabase desc_database;
>>>>> //DescriptorPool descriptor_pool(&errorCol, &desc_database);
>>>>>
>>>>> DescriptorPool descriptor_pool;
>>>>>
>>>>> for (int i = 0; i < f.file_size(); ++i) {
>>>>>
>>>>> descriptor_pool.BuildFile(f.file(i));
>>>>>
>>>>
>>>> Did you check that these return non-null and don't have any errors?
>>>>
>>>>
>>>>> }
>>>>>
>>>>>
>>>>> const Descriptor* descriptor =
>>>>> descriptor_pool.FindMessageTypeByName("RecordHeader");
>>>>> // descriptor =>NULL here
>>>>>
>>>>
>>>> You need to use the fully-qualified name to find the message type:
>>>> descriptor_pool.FindMessageTypeByName("traceFileMessagesProto.RecordHeader");
>>>>
>>>>
>>>>> //const FileDescriptor* new_file_desc =
>>>>> descriptor_pool.FindFileByName("dynMessage.dsc");
>>>>> /*
>>>>> DynamicMessageFactory factory;
>>>>>
>>>>> Message* my_message = factory.GetPrototype(descriptor)->New();
>>>>> delete my_message;
>>>>> */
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Finally, the point raised by Kenton and you that the proto file did
>>>>> not parse correctly to yield the valid FileDescriptor might be correct,
>>>>> since, I am generating the .proto file containing the messages on the fly
>>>>> by
>>>>> converting corresponding tags from my XML file.
>>>>> When earlier I used a handmade proto file for the messages, which
>>>>> parsed correctly to provide the descriptors, I was able to stream data
>>>>> into
>>>>> protocol buffers and use the buffers correctly. I will be inspecting this
>>>>> file at my end for any typing errors.
>>>>>
>>>>> Thanks for the answer and for opening this protobuf library on public
>>>>> domain, though.
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>>
>>>>> Himanshu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 23, 2010 at 7:53 AM, Jason Hsueh <jas...@google.com>wrote:
>>>>>
>>>>>> It occurred to me that protoc shouldn't accept your file either. Did
>>>>>> it really compile successfully? What do you actually get in your
>>>>>> FileDescriptorSet?
>>>>>>
>>>>>> On Wednesday, September 22, 2010, Jason Hsueh <jas...@google.com>
>>>>>> wrote:
>>>>>> > 1) I would guess the problem is that the "action" message type
>>>>>> refers to undefined types eaction_type and ereason. There are probably
>>>>>> error
>>>>>> messages farther up indicating that action could not be built. Because
>>>>>> action isn't a valid type, RecordHeader can't refer to it, and thinks
>>>>>> that
>>>>>> action is an undefined_symbol as well. The DescriptorBuilder will read in
>>>>>> all inputs, and cross-link the types once everything has been processed.
>>>>>> It
>>>>>> doesn't matter what order they are defined in.
>>>>>> > 2) You can manually build the FileDescriptorProto representation
>>>>>> yourself, or use the classes used by the compiler: see
>>>>>> google::protobuf::compiler::Parser or
>>>>>> google::protobuf::compiler::Importer.
>>>>>> >
>>>>>> > On Wed, Sep 22, 2010 at 4:42 AM, Himanshu Srivastava <
>>>>>> him.s...@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi Kenton,
>>>>>> > I have the following message in my .proto file:
>>>>>> > //************************************************/
>>>>>> > package traceFileMessagesProto;
>>>>>> > message RecordHeader {
>>>>>> > message myqueue{
>>>>>> > optional uint32 id = 1;
>>>>>> > optional uint32 Priority = 2;
>>>>>> > }
>>>>>> >
>>>>>> > message action{
>>>>>> > optional eactiontype eactiontype_instance = 1;
>>>>>> > optional ereason ereason_instance = 2;
>>>>>> > optional myqueue myqueue_instance = 3;
>>>>>> > }
>>>>>> > optional action action_instance =1;
>>>>>> > } //end RecordHeader
>>>>>> > /***************************************************/
>>>>>> > Thus, "action" is nested message in "RecordHeader" and has "queue"
>>>>>> submessage in it.
>>>>>> > I used the protoc compiler -o <desc-file> option to generate the
>>>>>> descriptor set file and loaded it through istream in c++ application,
>>>>>> followed by streaming FileDescriptorSet object with this descriptor set
>>>>>> file:
>>>>>> >
>>>>>> >
>>>>>> > //***************************************************************//
>>>>>> > ifstream desc_file("mydescriptor.dsc",ios::in);
>>>>>> > FileDescriptorSet f;
>>>>>> > f.ParseFromIstream(&desc_file);
>>>>>> > //***************************************************************//
>>>>>> >
>>>>>> >
>>>>>> > However, while parsing through DescriptorPool::BuildFile(f.file(0));
>>>>>> > at the line corresponding to the message file line:
>>>>>> > => optional action action_instance =1;
>>>>>> > the BuildFile() function exits by giving error:
>>>>>> > undefined_symbol "action"
>>>>>> > The code returns through the following function in protoc library:
>>>>>> > //***************************************************************//
>>>>>> > void DescriptorBuilder::AddNotDefinedError(
>>>>>> > const string& element_name,
>>>>>> > const Message& descriptor,
>>>>>> > DescriptorPool::ErrorCollector::ErrorLocation location,
>>>>>> > const string& undefined_symbol) {
>>>>>> > if (possible_undeclared_dependency_ == NULL) {
>>>>>> > AddError(element_name, descriptor, location,
>>>>>> > "\"" + undefined_symbol + "\" is not defined.");
>>>>>> > //***************************************************************//
>>>>>> > The only possible explanation could be that the message "action" is
>>>>>> not parsed before its instance declaration from my .proto/descriptor_set
>>>>>> file.
>>>>>> > Probably all nested messages should be parsed on their instance
>>>>>> declarations in a recursive manner in the DescriptorPool::Buildfile()
>>>>>> function.
>>>>>> >
>>>>>> >
>>>>>> > Qustion1:
>>>>>> > How do I solve the above issue?
>>>>>> > Question 2:
>>>>>> > Can not I simply load my messages in DescriptorPool through the
>>>>>> .proto file instead of calling the service of protoc compiler to
>>>>>> generate a
>>>>>> descriptor file? I have to generate .proto file at runtime and I don't
>>>>>> want
>>>>>> to call another exe (protoc.exe) from my application for dependency
>>>>>> reasons.
>>>>>> >
>>>>>> >
>>>>>> > Thanks in advance for any solutions /directions.
>>>>>> > Regards,
>>>>>> > Himanshu
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > You received this message because you are subscribed to the Google
>>>>>> Groups "Protocol Buffers" group.
>>>>>> > To post to this group, send email to proto...@googlegroups.com.
>>>>>> > To unsubscribe from this group, send email to
>>>>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>>>>> .
>>>>>> > For more options, visit this group at
>>>>>> http://groups.google.com/group/protobuf?hl=en.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To post to this group, send email to proto...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/protobuf?hl=en.
>>>
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
