You would need to implement your own sorting routines, moving them around
with SwapElements().

Maps would make it easy for you to search based on some key field; until
then you could just build your own map of the elements on top of the
protobuf.

On Fri, Oct 8, 2010 at 7:35 AM, Himanshu Srivastava <him.s...@gmail.com>wrote:

> Hi Jason,
>
> I was out of this discussion for quite some time, but during this time I
> was able to make my large application's design work with dynamic messages
> using the protobuf::Importer. However, I am now stuck with the issue of
> indexing/sorting and searching messages.
>
> I understand that the code generated messages could be sorted with
> stl::sort(itor:begin, itoe::end, functor), but, that's because there is a
> RepeatedPtrField iterator and RepeatedPtrField list available through the
> message class.
>
> I have to sort/search on a repeated nested message declared in my main
> message depending upon the fields of the nested message, like so:
>
> message nested_msg
> {
> required uint64 indx =1; //sort on this field
> optional uint64 classId =2; //sort on this field
> optional string network_name = 3; //search on this field
>
> }
>
> message coverMessage
> {
> required uint64 nested_msg_count =1;
> repeated nested_msg  msg_instance =2;
> }
>
> Could this be achieved with dynamic messages through polymorphic message
> pointers? My current work on protobuf serialization will remain an
> experimentation otherwise.
>
> P.S.: I hear you people are thinking on the design of protobuf maps. Will
> it be possible that protobufs parent level messages be sorted/searched at
> nested message levels through maps of nested message fields? Will '[key]'
> and '[value]' operators be available for the maps?
>
>
> Regards,
> Himanshu
>
> On Fri, Sep 24, 2010 at 6:02 PM, Himanshu Srivastava 
> <him.s...@gmail.com>wrote:
>
>>  >>Gotcha. If you run protoc on the file produced by your generator
>> class, does it succeed?
>>
>> protoc.exe succeeds parsing my proto file. I have emailed the file to you
>> and you can verify that.
>>
>> >>>>descriptor_pool is of course NULL.
>>
>> >>I meant that you should check the return value of BuildFile(). If it
>> returns NULL, then it could not build the file. The ErrorCollector  >>would
>> tell you why
>> I meant the same, Build_File() returns Null.
>>
>> However, I will get back to this issue. It will take some time to verify
>> issues in my implementation.
>>
>> Thanks & Regards,
>> Himanshu
>>
>>   On Thu, Sep 23, 2010 at 10:37 PM, Jason Hsueh <jas...@google.com>wrote:
>>
>>>
>>>
>>>  On Thu, Sep 23, 2010 at 9:52 AM, Himanshu Srivastava <
>>> him.s...@gmail.com> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> In code:
>>>> //********************************************//
>>>>  DescriptorPool descriptor_pool;
>>>>
>>>>         for (int i = 0; i < f.file_size(); ++i) {
>>>>
>>>>             descriptor_pool.BuildFile(f.file(i));
>>>> //********************************************//
>>>> descriptor_pool is of course NULL.
>>>>
>>>
>>> I meant that you should check the return value of BuildFile(). If it
>>> returns NULL, then it could not build the file. The ErrorCollector would
>>> tell you why.
>>>
>>>
>>>> And for the Importer approach my code does SEG-FAULT.
>>>>
>>>> Further I do need conversion from proto file directly and also a static
>>>> FileDescriptor so that it can be passed around for construction of my new
>>>> messages in a recursive tree algorithm in  another distinct class object.
>>>>
>>>
>>> That's up to you: you'll need to ensure that the DescriptorPool has the
>>> lifetime you want.
>>>
>>> Previously, you mentioned that the importer approach returned NULL. That
>>> would be indicative of a problem in the .proto file: the file could not be
>>> built. Even if you were to successfully build the file, your code would
>>> segfault, because the FileDescriptor is no longer valid, as the Importer,
>>> and thus the DescriptorPool containing the FileDescriptor, have been
>>> deleted.
>>>
>>>
>>>>
>>>> I do not want the descriptor set approach, since that is only available
>>>> when protoc exe is present.
>>>> ---------------------
>>>>  >>I doubt that your code will work, as Importer should be looking for
>>>> a file named "dynamicProtoMessages.proto", which probably >>does not exist.
>>>> ------------------------
>>>> I create the proto file "dynamicProtoMessages.proto" parmanently on disk
>>>> and read it from disk to Parser after it is generated from my
>>>> CXMLToProtoFileGenerator class.
>>>>
>>>
>>> Gotcha. If you run protoc on the file produced by your generator class,
>>> does it succeed?
>>>
>>>
>>>>
>>>> Thanks & Regards,
>>>> Himanshu
>>>>
>>>>
>>>> On Thu, Sep 23, 2010 at 9:39 PM, Jason Hsueh <jas...@google.com> wrote:
>>>>
>>>>>
>>>>>
>>>>>  On Thu, Sep 23, 2010 at 2:40 AM, Himanshu Srivastava <
>>>>> him.s...@gmail.com> wrote:
>>>>>
>>>>>> Kenton:
>>>>>>
>>>>>>
>>>>>> I really sent you a only a part of the complete RecordHeader message,
>>>>>> since I was focusing on "nested message not getting parsed" issue for 
>>>>>> this
>>>>>> particular message. Indeed my complete proto file is a large one. I
>>>>>> have attached the truncated proto file containing this complete message 
>>>>>> only
>>>>>> for reference with this mail (plese see attachment
>>>>>> 'dynamicProtoMessages.proto '.
>>>>>>
>>>>>> I had used Parser and Importer earlier and generated the
>>>>>> FileDescriptor in a class “CMessageParser” member function as below:
>>>>>>
>>>>>>
>>>>>>
>>>>>> //***********************************************************//
>>>>>>
>>>>>> class PacketTracerErrorFileErrorCollector: 
>>>>>> publicgoogle::protobuf::compiler::MultiFileErrorCollector
>>>>>>
>>>>>> {
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> };
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> const google::protobuf::FileDescriptor* CMessageParser
>>>>>> ::generateMessageFileDescriptor()
>>>>>>
>>>>>> {
>>>>>>
>>>>>>     std::pair<std::string, std::string> filePair;
>>>>>>
>>>>>>     google::protobuf::compiler::DiskSourceTree source_tree;
>>>>>>
>>>>>>     source_tree.MapPath(/*filePair.first*/"", "."/*filePair.second*/
>>>>>> );
>>>>>>
>>>>>>     PacketTracerErrorFileErrorCollector error_collector;//Defined
>>>>>> above
>>>>>>
>>>>>>     google::protobuf::compiler::Importer importer(&source_tree,
>>>>>> &error_collector);
>>>>>>
>>>>>>     std::string outFileNameString = “dynamicProtoMessages.proto”;
>>>>>>     const google::protobuf::FileDescriptor* file_desc =
>>>>>> importer.Import(outFileNameString);
>>>>>>     return file_desc;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> //***********************************************************//
>>>>>>
>>>>>>
>>>>>> However, the problem with that approach was, as soon as I returned the
>>>>>> pointer from the FileDescriptor to a client class object of
>>>>>> CMessageParser, the FileDescriptor* becomes NULL  or invalid  once
>>>>>> out of the class and it becomes useless.
>>>>>>
>>>>>  //***********************************************************//
>>>>>>
>>>>>> CMessageParser myParser;
>>>>>>
>>>>>>  const google::protobuf::FileDescriptor *file_desc = 
>>>>>> myParser.generateMessageFileDescriptor();
>>>>>> //file_desc is NULL
>>>>>>
>>>>>> //***********************************************************//
>>>>>>
>>>>>>
>>>>>>  So, I thought, I was following some wrong approach and to retain the
>>>>>> File Descriptors for a long enough period across scopes, I may need the
>>>>>> DescriptorPool, which might have the permanent lifetime for the period of
>>>>>> the application.
>>>>>>
>>>>>
>>>>> Importer just wraps DescriptorPool and a database of proto files.
>>>>> Neither of these guarantee static duration for the descriptors - any
>>>>> descriptors that you build will only remain alive while the
>>>>> DescriptorPool/Importer remains alive.
>>>>>
>>>>> Of course, that means your code should seg-fault, not return NULL,
>>>>> assuming that you successfully built the file. You should look at the 
>>>>> error
>>>>> collector to see what went wrong. I doubt that your code will work, as
>>>>> Importer should be looking for a file named "dynamicProtoMessages.proto",
>>>>> which probably does not exist.
>>>>>
>>>>> But it sounds like you don't actually need to read .proto files, and
>>>>> instead you are generating the *DescriptorProto representations directly. 
>>>>> If
>>>>> that's the case, you don't need Importer, you should directly use a
>>>>> DescriptorPool as you did in the code snippet below.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Jason:
>>>>>>
>>>>>> I am always able to generate the FileDescriptorSet easily, although
>>>>>> the FileDescriptor was never showing completely populated fields when
>>>>>> debugging with Visual studio 2008 Express edition.
>>>>>>
>>>>>> I have attached the generated FileDescriptorSet 'dynProtoMessage.dsc'
>>>>>> file along with the proto file containing the above messages with this 
>>>>>> mail.
>>>>>> You can see even this dsc file (made with simple messages as in the 
>>>>>> attached
>>>>>> proto file) does not load through DescriptorPool::BuildFile().
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is the function I used to load DescriptorSet:
>>>>>>
>>>>>>
>>>>>>
>>>>>> void CMessageParser 
>>>>>> ::getMessagesFromDescriptorSet(constgoogle::protobuf::FileDescriptor** 
>>>>>> new_file_desc)
>>>>>>
>>>>>> {
>>>>>>
>>>>>>     using namespace google::protobuf;
>>>>>>
>>>>>>     using namespace google::protobuf::io;
>>>>>>
>>>>>>     using namespace google::protobuf::compiler;
>>>>>>
>>>>>>     ifstream desc_file("dynProtoMessage.dsc",ios::in);
>>>>>>
>>>>>>         if (desc_file.bad()) {
>>>>>>
>>>>>>                       std::cout << "desc file not read...exiting" <<
>>>>>> std::endl;
>>>>>>
>>>>>>                      return;
>>>>>>
>>>>>>         }
>>>>>>
>>>>>>
>>>>>>
>>>>>>         FileDescriptorSet f;
>>>>>>
>>>>>>         f.ParseFromIstream(&desc_file);
>>>>>>
>>>>>>
>>>>>>
>>>>>>        // DescriptorPoolErrorCollector errorCol;
>>>>>>         //google::protobuf::SimpleDescriptorDatabase desc_database;
>>>>>>        //DescriptorPool descriptor_pool(&errorCol, &desc_database);
>>>>>>
>>>>>>         DescriptorPool descriptor_pool;
>>>>>>
>>>>>>         for (int i = 0; i < f.file_size(); ++i) {
>>>>>>
>>>>>>             descriptor_pool.BuildFile(f.file(i));
>>>>>>
>>>>>
>>>>> Did you check that these return non-null and don't have any errors?
>>>>>
>>>>>
>>>>>>         }
>>>>>>
>>>>>>
>>>>>>         const Descriptor* descriptor =
>>>>>> descriptor_pool.FindMessageTypeByName("RecordHeader");
>>>>>> // descriptor =>NULL here
>>>>>>
>>>>>
>>>>> You need to use the fully-qualified name to find the message type:
>>>>> descriptor_pool.FindMessageTypeByName("traceFileMessagesProto.RecordHeader");
>>>>>
>>>>>
>>>>>>         //const FileDescriptor* new_file_desc =
>>>>>> descriptor_pool.FindFileByName("dynMessage.dsc");
>>>>>>         /*
>>>>>>         DynamicMessageFactory factory;
>>>>>>
>>>>>>         Message* my_message =
>>>>>> factory.GetPrototype(descriptor)->New();
>>>>>>         delete my_message;
>>>>>>        */
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Finally,  the point raised by Kenton and you that the proto file did
>>>>>> not parse correctly to yield the valid FileDescriptor might be correct,
>>>>>> since, I am generating the .proto file containing the messages on the 
>>>>>> fly by
>>>>>> converting corresponding tags from my XML file.
>>>>>> When earlier I used a handmade proto file for the messages, which
>>>>>> parsed correctly to provide the descriptors, I was able to stream data 
>>>>>> into
>>>>>> protocol buffers and use the buffers correctly. I will be inspecting this
>>>>>> file at my end for any typing errors.
>>>>>>
>>>>>> Thanks for the answer and for opening this protobuf library on public
>>>>>> domain, though.
>>>>>>
>>>>>>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Himanshu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 23, 2010 at 7:53 AM, Jason Hsueh <jas...@google.com>wrote:
>>>>>>
>>>>>>> It occurred to me that protoc shouldn't accept your file either. Did
>>>>>>> it really compile successfully? What do you actually get in your
>>>>>>> FileDescriptorSet?
>>>>>>>
>>>>>>> On Wednesday, September 22, 2010, Jason Hsueh <jas...@google.com>
>>>>>>> wrote:
>>>>>>> > 1) I would guess the problem is that the "action" message type
>>>>>>> refers to undefined types eaction_type and ereason. There are probably 
>>>>>>> error
>>>>>>> messages farther up indicating that action could not be built. Because
>>>>>>> action isn't a valid type, RecordHeader can't refer to it, and thinks 
>>>>>>> that
>>>>>>> action is an undefined_symbol as well. The DescriptorBuilder will read 
>>>>>>> in
>>>>>>> all inputs, and cross-link the types once everything has been 
>>>>>>> processed. It
>>>>>>> doesn't matter what order they are defined in.
>>>>>>> > 2) You can manually build the FileDescriptorProto representation
>>>>>>> yourself, or use the classes used by the compiler: see
>>>>>>> google::protobuf::compiler::Parser or 
>>>>>>> google::protobuf::compiler::Importer.
>>>>>>> >
>>>>>>> > On Wed, Sep 22, 2010 at 4:42 AM, Himanshu Srivastava <
>>>>>>> him.s...@gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hi Kenton,
>>>>>>> > I have the following message in my .proto file:
>>>>>>> > //************************************************/
>>>>>>> > package traceFileMessagesProto;
>>>>>>> > message RecordHeader {
>>>>>>> > message  myqueue{
>>>>>>> >  optional uint32 id = 1;
>>>>>>> >  optional uint32 Priority = 2;
>>>>>>> > }
>>>>>>> >
>>>>>>> > message  action{
>>>>>>> >  optional eactiontype    eactiontype_instance = 1;
>>>>>>> >  optional ereason ereason_instance = 2;
>>>>>>> >  optional myqueue myqueue_instance = 3;
>>>>>>> > }
>>>>>>> > optional action action_instance =1;
>>>>>>> > } //end RecordHeader
>>>>>>> > /***************************************************/
>>>>>>> > Thus, "action" is nested message in "RecordHeader" and has "queue"
>>>>>>> submessage in it.
>>>>>>> > I used the protoc compiler -o <desc-file> option to generate the
>>>>>>> descriptor set file and loaded it through istream in c++ application,
>>>>>>> followed by streaming FileDescriptorSet object with this descriptor set
>>>>>>> file:
>>>>>>> >
>>>>>>> >
>>>>>>> > //***************************************************************//
>>>>>>> > ifstream desc_file("mydescriptor.dsc",ios::in);
>>>>>>> > FileDescriptorSet f;
>>>>>>> > f.ParseFromIstream(&desc_file);
>>>>>>> > //***************************************************************//
>>>>>>> >
>>>>>>> >
>>>>>>> > However, while parsing through
>>>>>>> DescriptorPool::BuildFile(f.file(0));
>>>>>>> > at the line corresponding to the message file line:
>>>>>>> > => optional action action_instance =1;
>>>>>>> > the BuildFile() function exits by giving error:
>>>>>>> > undefined_symbol "action"
>>>>>>> > The code returns through the following function in protoc library:
>>>>>>> > //***************************************************************//
>>>>>>> > void DescriptorBuilder::AddNotDefinedError(
>>>>>>> >     const string& element_name,
>>>>>>> >     const Message& descriptor,
>>>>>>> >     DescriptorPool::ErrorCollector::ErrorLocation location,
>>>>>>> >     const string& undefined_symbol) {
>>>>>>> >   if (possible_undeclared_dependency_ == NULL) {
>>>>>>> >     AddError(element_name, descriptor, location,
>>>>>>> >              "\"" + undefined_symbol + "\" is not defined.");
>>>>>>> > //***************************************************************//
>>>>>>> > The only possible explanation could be that the message "action" is
>>>>>>> not parsed before its instance declaration from my .proto/descriptor_set
>>>>>>> file.
>>>>>>> > Probably all nested messages should be parsed on their instance
>>>>>>> declarations in a recursive manner in the DescriptorPool::Buildfile()
>>>>>>> function.
>>>>>>> >
>>>>>>> >
>>>>>>> > Qustion1:
>>>>>>> > How do I solve the above issue?
>>>>>>> > Question 2:
>>>>>>> > Can not I simply load my messages in DescriptorPool through the
>>>>>>> .proto file instead of calling the service of protoc compiler to 
>>>>>>> generate a
>>>>>>> descriptor file? I have to generate .proto file at runtime and I don't 
>>>>>>> want
>>>>>>> to call another exe (protoc.exe) from my application for dependency 
>>>>>>> reasons.
>>>>>>> >
>>>>>>> >
>>>>>>> > Thanks in advance for any solutions /directions.
>>>>>>> > Regards,
>>>>>>> > Himanshu
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > You received this message because you are subscribed to the Google
>>>>>>> Groups "Protocol Buffers" group.
>>>>>>> > To post to this group, send email to proto...@googlegroups.com.
>>>>>>> > To unsubscribe from this group, send email to
>>>>>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>>>>>> .
>>>>>>> > For more options, visit this group at
>>>>>>> http://groups.google.com/group/protobuf?hl=en.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>>   You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To post to this group, send email to proto...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/protobuf?hl=en.
>>>>
>>>
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to