[protobuf] Re: Regd: Resolving Wire type ambiguities
It sounds like you actually have the message class available, not just a serialized instance of the message. In that case, you can derive the original type by reading the generated code. If you only have a compiled copy of the class, you can derive the type from its descriptor -- use MessageType.getDescriptor() in Java or MessageType::descriptor() in C++ to get it. In C++ you can even call descriptor->file()->DebugString() to generate a .proto-syntax representation of the file. On Mon, Nov 16, 2009 at 1:32 PM, rahul prasad wrote: > Hi, > > Thanks for the clarification. I did try one dirty method of finding the > original types, because of my ".proto"-less situation. I relied on exception > statements thrown out when I iterated through the protobuffers by trying to > extract a known wiretype with a wrong-type getter. I know it sucks, but it > worked for me. Thanks. > > Regards, > Rahul Prasad > > > > > On Mon, Nov 16, 2009 at 1:42 PM, Jason Hsueh wrote: > >> You can decode the protocol buffer with just wire type + tag number, but >> you won't know the original types without a proto definition. Everything >> would be treated as an unknown field. You could access these by iterating >> through the UnknownFieldSet, but again, you can't recover the original >> types. >> >> On Sat, Nov 14, 2009 at 1:10 PM, rahul prasad wrote: >> >>> Hi Marc, >>> >>> Thanks for the clarification. If the actual .proto was there, i would not >>> have posted that question [?] at the first place. Anyways, to decode a >>> protocol buffer, is it not enough to have just the wire type + tag number >>> combination? (except of course, handling of the sub-messages-ness and other >>> ambiguities you mentioned below have to be done manually though) >>> >>> Regards, >>> Rahul Prasad >>> >>> >>> >>> On Sat, Nov 14, 2009 at 3:57 PM, Marc Gravell wrote: >>> If you treat it as a string (UTF8), you are likely to get garbage. If you treat it as a byte[], then you just get a BLOB - you don't lose anything, but you might not be showing some more detail that you could show. You could, however, check for likely-sub-message-ness - i.e. after getting the length, you could try decoding the next few bytes as a variant, and do the shift trick; see if it looks likely to be a sub-message etc; you could try to validate the entire "string", see if it makes sense. Note that you don't have to store any of the data - just follow the rules for each wire-format until something doesn't look right or you've checked the string. Easiest, though, is to have the .proto available ;-p Marc 2009/11/14 rahul prasad > Hi, > > As seen from the below wire types table from protobuf documentation, if > i try to extract a value from a protobuf that is of type 2, it could > either > be a string, byte array or a embedded message etc, If I cast the value as > bytes or string on the decoding side, while on the encoding side it was > actually an embedded message, what would this result in? Will I be able to > retrieve the actual value, someway or the other doing it this way? > > The available wire types are as follows: > Type Meaning Used For 0 Varint int32, int64, uint32, uint64, sint32, > sint64, bool, enum 1 64-bit fixed64, sfixed64, double > 2Length-delimitedstring, bytes, embedded messages, packed repeated > fields3Start groupgroups (deprecated)4End groupgroups > (deprecated)532-bitfixed32, sfixed32, float > Regards, > Rahul Prasad > > > > -- Regards, Marc >>> >>> >>> >>> >> > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~--- <>
[protobuf] Regd: Resolving Wire type ambiguities
Hi, Thanks for the clarification. I did try one dirty method of finding the original types, because of my ".proto"-less situation. I relied on exception statements thrown out when I iterated through the protobuffers by trying to extract a known wiretype with a wrong-type getter. I know it sucks, but it worked for me. Thanks. Regards, Rahul Prasad On Mon, Nov 16, 2009 at 1:42 PM, Jason Hsueh wrote: > You can decode the protocol buffer with just wire type + tag number, but > you won't know the original types without a proto definition. Everything > would be treated as an unknown field. You could access these by iterating > through the UnknownFieldSet, but again, you can't recover the original > types. > > On Sat, Nov 14, 2009 at 1:10 PM, rahul prasad wrote: > >> Hi Marc, >> >> Thanks for the clarification. If the actual .proto was there, i would not >> have posted that question [?] at the first place. Anyways, to decode a >> protocol buffer, is it not enough to have just the wire type + tag number >> combination? (except of course, handling of the sub-messages-ness and other >> ambiguities you mentioned below have to be done manually though) >> >> Regards, >> Rahul Prasad >> >> >> >> On Sat, Nov 14, 2009 at 3:57 PM, Marc Gravell wrote: >> >>> If you treat it as a string (UTF8), you are likely to get garbage. If you >>> treat it as a byte[], then you just get a BLOB - you don't lose anything, >>> but you might not be showing some more detail that you could show. >>> >>> You could, however, check for likely-sub-message-ness - i.e. after >>> getting the length, you could try decoding the next few bytes as a variant, >>> and do the shift trick; see if it looks likely to be a sub-message etc; you >>> could try to validate the entire "string", see if it makes sense. Note that >>> you don't have to store any of the data - just follow the rules for each >>> wire-format until something doesn't look right or you've checked the string. >>> >>> Easiest, though, is to have the .proto available ;-p >>> >>> Marc >>> >>> 2009/11/14 rahul prasad >>> Hi, As seen from the below wire types table from protobuf documentation, if i try to extract a value from a protobuf that is of type 2, it could either be a string, byte array or a embedded message etc, If I cast the value as bytes or string on the decoding side, while on the encoding side it was actually an embedded message, what would this result in? Will I be able to retrieve the actual value, someway or the other doing it this way? The available wire types are as follows: Type Meaning Used For 0 Varint int32, int64, uint32, uint64, sint32, sint64, bool, enum 1 64-bit fixed64, sfixed64, double 2Length-delimitedstring, bytes, embedded messages, packed repeated fields3Start groupgroups (deprecated)4End groupgroups (deprecated)532-bitfixed32, sfixed32, float Regards, Rahul Prasad >>> >>> >>> -- >>> Regards, >>> >>> Marc >>> >> >> >> >> >> > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~--- <>
[protobuf] Re: question about creating message classes dynamically (known at runtime)
(I think my previous reply didn't arrive)
Thanks Kenton, this worked fine for me.
Probably, with DynamicMessageFactory I was trying to solve in a single
attempt, dynamic generation of both classes known in compile-time and
unknown. I forgot then to check back again with MessageFactory. I
guess, that I have learned the lesson about MessageFactory and
DynamicMessageFactory differences :)
Thanks again for your help. I appreciate it
regards
On Nov 16, 8:48 pm, Kenton Varda wrote:
> BTW, you could catch this problem more easily if you defined a down_cast
> function like:
>
> template
> To down_cast(From from) {
> #ifdef NDEBUG
> return static_cast(from);
> #else
> To result = dynamic_cast(from);
> assert(result != NULL);
> return result;
> #endif
>
> }
>
> This way, in debug mode you use RTTI to verifythat the down_cast is valid,
> but in release builds you get static cast performance.
>
> google/protobuf/stubs/common.h contains such an implementation of down_cast,
> although you should not use the protobuf version directly since we may
> decide to change it in the future. Instead, copy it into your own code.
>
> On Mon, Nov 16, 2009 at 11:44 AM, Kenton Varda wrote:
> > On Mon, Nov 16, 2009 at 11:35 AM, JC-MAD-SP
> > wrote:
>
> >> ConcreteMessage* concreteMessage = reinterpret_cast
> >> (mesage->New());
>
> > This line is invalid. Here, *message is a DynamicMessage, and New() also
> > returns a DynamicMessage, not a ConcreteMessage. DynamicMessage is a class
> > which implements the Message interface to look like any arbitrary
> > descriptor. This allows you to represent types which are not compiled into
> > the binary at all.
>
> > If you want to construct instances of compiled-in types from their
> > descriptors, you want to use MessageFactory::generated_factory().
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
[protobuf] Re: question about creating message classes dynamically (known at runtime)
BTW, you could catch this problem more easily if you defined a down_cast
function like:
template
To down_cast(From from) {
#ifdef NDEBUG
return static_cast(from);
#else
To result = dynamic_cast(from);
assert(result != NULL);
return result;
#endif
}
This way, in debug mode you use RTTI to verifythat the down_cast is valid,
but in release builds you get static cast performance.
google/protobuf/stubs/common.h contains such an implementation of down_cast,
although you should not use the protobuf version directly since we may
decide to change it in the future. Instead, copy it into your own code.
On Mon, Nov 16, 2009 at 11:44 AM, Kenton Varda wrote:
> On Mon, Nov 16, 2009 at 11:35 AM, JC-MAD-SP wrote:
>
>> ConcreteMessage* concreteMessage = reinterpret_cast
>> (mesage->New());
>>
>
> This line is invalid. Here, *message is a DynamicMessage, and New() also
> returns a DynamicMessage, not a ConcreteMessage. DynamicMessage is a class
> which implements the Message interface to look like any arbitrary
> descriptor. This allows you to represent types which are not compiled into
> the binary at all.
>
> If you want to construct instances of compiled-in types from their
> descriptors, you want to use MessageFactory::generated_factory().
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
[protobuf] Re: question about creating message classes dynamically (known at runtime)
On Mon, Nov 16, 2009 at 11:35 AM, JC-MAD-SP wrote: > ConcreteMessage* concreteMessage = reinterpret_cast > (mesage->New()); > This line is invalid. Here, *message is a DynamicMessage, and New() also returns a DynamicMessage, not a ConcreteMessage. DynamicMessage is a class which implements the Message interface to look like any arbitrary descriptor. This allows you to represent types which are not compiled into the binary at all. If you want to construct instances of compiled-in types from their descriptors, you want to use MessageFactory::generated_factory(). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
[protobuf] question about creating message classes dynamically (known at runtime)
Hi
I'm new to ProtocolBuffer. Until now all is going ok with Protocol
Buffers for my needs.
But, now I'm struggling for a while with a Message->New issue without
a clue.
I'm trying to create "concrete" messages, which are known at runtime
using DynamicMessageFactory (C++). Following is the pseudo-code
const Descriptor* descriptor = ConcreteMessage::descriptor();
//or const Descriptor* descriptor = DescriptorPool::generated_pool()-
>FindMessageTypeByName("ConcreteMessage") ;
DynamicMessageFactory messageFactory;
Message* message = messageFactory.GetPrototype(descriptor);
ifstream input("ZZZ", ios::in | ios::binary);
ConcreteMessage* concreteMessage = reinterpret_cast
(mesage->New());
concreteMessage->ParseFromIstream(&input));
Message is created, works and encoded fine. Accessing methods through
Reflection works fine (DebugString() shows ok, which uses internally
Reflection).
However, accessing this ConcreteMessage through the "accessor" methods
generated automatically by protoc.exe compiler in
ConcreteMessage.pb.cc (i.e ConcreteMessage.has_,
ConcreteMessage.X(), ConcreteMessage.mutable_X()) doesn't
works at all.
What I have seen is that the properties defined in this automatic
generated class are not updated when ParseFromIstream is executed. I
would say that even other methods in this class are not being called
from ParseFromIstream (i.e. Clear()...).
Is this something not supported or am I missing something?
Any comment will be appreciated
regards
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
[protobuf] Re: Regd: Resolving Wire type ambiguities
You can decode the protocol buffer with just wire type + tag number, but you won't know the original types without a proto definition. Everything would be treated as an unknown field. You could access these by iterating through the UnknownFieldSet, but again, you can't recover the original types. On Sat, Nov 14, 2009 at 1:10 PM, rahul prasad wrote: > Hi Marc, > > Thanks for the clarification. If the actual .proto was there, i would not > have posted that question [?] at the first place. Anyways, to decode a > protocol buffer, is it not enough to have just the wire type + tag number > combination? (except of course, handling of the sub-messages-ness and other > ambiguities you mentioned below have to be done manually though) > > Regards, > Rahul Prasad > > > > On Sat, Nov 14, 2009 at 3:57 PM, Marc Gravell wrote: > >> If you treat it as a string (UTF8), you are likely to get garbage. If you >> treat it as a byte[], then you just get a BLOB - you don't lose anything, >> but you might not be showing some more detail that you could show. >> >> You could, however, check for likely-sub-message-ness - i.e. after getting >> the length, you could try decoding the next few bytes as a variant, and do >> the shift trick; see if it looks likely to be a sub-message etc; you could >> try to validate the entire "string", see if it makes sense. Note that you >> don't have to store any of the data - just follow the rules for each >> wire-format until something doesn't look right or you've checked the string. >> >> Easiest, though, is to have the .proto available ;-p >> >> Marc >> >> 2009/11/14 rahul prasad >> >>> Hi, >>> >>> As seen from the below wire types table from protobuf documentation, if i >>> try to extract a value from a protobuf that is of type 2, it could either be >>> a string, byte array or a embedded message etc, If I cast the value as bytes >>> or string on the decoding side, while on the encoding side it was actually >>> an embedded message, what would this result in? Will I be able to retrieve >>> the actual value, someway or the other doing it this way? >>> >>> The available wire types are as follows: >>> Type Meaning Used For 0 Varint int32, int64, uint32, uint64, sint32, >>> sint64, bool, enum 1 64-bit fixed64, sfixed64, double 2 >>> Length-delimitedstring, bytes, embedded messages, packed repeated >>> fields3Start groupgroups (deprecated)4End groupgroups >>> (deprecated)532-bitfixed32, sfixed32, float >>> Regards, >>> Rahul Prasad >>> >>> >>> >>> >> >> >> -- >> Regards, >> >> Marc >> > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~--- <>
[protobuf] Re: unexpected result from TextFormat::PrintToString in c++
The output is correct. You have not initialized the "header" field. You
need to call mutable_header() once to initialize it.
On Mon, Nov 16, 2009 at 1:45 AM, edan wrote:
>
> I have the following class:
>
> package PB_SimpleMessage;
>
> message SimpleMessage
> {
> required RootNode header = 1;
> required RootNode payload = 2;
> }
>
> message RootNode
> {
> repeated string parameters = 1;
> }
>
> In my code i am do the following:
>PB_SimpleMessage::SimpleMessage* pSimpleMsg = new
> PB_SimpleMessage::SimpleMessage;
>PB_SimpleMessage::RootNode* msgPayload = pSimpleMsg-
> >mutable_payload();
>string outString;
>google::protobuf::TextFormat::PrintToString(*pSimpleMsg ,
> &outString);
>cout << outString << endl;
>
> I believe that the EXPECTED result should be:
> header {
> }
> payload {
> }
>
> However the ACTUAL result is:
> payload {
> }
>
> Can any one tell me why?
> Is It a bug?
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
[protobuf] Re: Seek in a codedoutputstream
Oh, I see. No, CodedOutputStream doesn't support seek functionality, so I guess you'd have to go with b). Just make sure that you're using WriteLittleEndian64 and not WriteVarint64. On Sat, Nov 14, 2009 at 12:05 PM, Fishtank wrote: > > That's because I don't know the size of data1(and it is not from > protobuf parse) > I'm using protobuf for the stream libraries. > I can either a) loop through the calculation to get the total data > size, then loop again writing size and data > or b) loop once, write, get size, seek and adjust size. > > Regards > sapsi > > > On Nov 14, 2:56 pm, Jason Hsueh wrote: > > Why can't you write the actual size of data rather than a 0? Just use the > > ByteSize() method to find out how much data will be written for a > protocol > > buffer. > > > > On Sat, Nov 14, 2009 at 11:39 AM, Fishtank > wrote: > > > > > Hello, > > > I'm using the following idiom > > > ZeroCopyOutputStream* raw_output = new FileOutputStream(fd); > > > CodedOutputStream* coded_output = new CodedOutputStream(raw_output); > > > > > The format i'd like is uint64_t, followed by data1, followed by data2. > > > I write a 0 for the uint64_t, write the data1, record the bytes written > > > (B), then write data2. > > > I'd like to seek to the beginning of the file and change the uint64_t > > > val (which was 0 ) to B. > > > (data2 is updated much more frequently than data1) > > > > > It appears codedoutputstream doesn't have a seek, so I'm thinking , > > > after the above steps > > > a) deleting the CodedOutputStream and Zero... > > > b) seek on the fd, and update the value. > > > > > Does this sound right? > > > Regards > > > Sapsi > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
[protobuf] unexpected result from TextFormat::PrintToString in c++
I have the following class:
package PB_SimpleMessage;
message SimpleMessage
{
required RootNode header = 1;
required RootNode payload = 2;
}
message RootNode
{
repeated string parameters = 1;
}
In my code i am do the following:
PB_SimpleMessage::SimpleMessage* pSimpleMsg = new
PB_SimpleMessage::SimpleMessage;
PB_SimpleMessage::RootNode* msgPayload = pSimpleMsg-
>mutable_payload();
string outString;
google::protobuf::TextFormat::PrintToString(*pSimpleMsg ,
&outString);
cout << outString << endl;
I believe that the EXPECTED result should be:
header {
}
payload {
}
However the ACTUAL result is:
payload {
}
Can any one tell me why?
Is It a bug?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
