Extra data at the end would definitely be a problem. That could cause parsing to return an error, or it might happen to be in the right format so that it gets parsed as an unknown field (or even a regular known field).
If the serialization returns an error there isn't an API to get more details about the error--you just have to examine the data manually if you want to figure out what went wrong. On Fri, Apr 13, 2018 at 6:42 AM Tony Tony <[email protected]> wrote: > For both messages I think it returned 10, although the actual data for the > second one was wrong. I think that extra data might just be copying extra > data from the string printed in my log. Would extra data at the end of > serialized data affect actual deserialization? Cause the first one > deserialized just fine. > > I'll check what SerializeToArray is returning when I get home, if it > returns false is there somewhere the error would get logged to? Or is there > some sort of GetError call in protobuf you can call after certain protobuf > methods? > > Thanks again > > On Thursday, April 12, 2018 at 12:32:15 PM UTC-4, Adam Cozzette wrote: >> >> What does pak.ByteSize() return for those two messages? It would also be >> good to verify that SerializeToArray() is returning true, because if not >> then there could be an error being missed. >> >> Something seems to be going wrong with the message sizes, because the >> first data example seems to have extra data on the end while the second one >> seems to be too short (truncated). >> >> By the way, Marc, I tried out your protogen tool just now and it's pretty >> cool! That's more convenient than having to fiddle around with creating the >> binary data file and running protoc --decode_raw on it. >> >> On Wed, Apr 11, 2018 at 3:25 PM Tony Tony <[email protected]> wrote: >> >>> Well I just copied whatever text was printed after i logged what was >>> being serialized. So I probably copied extra bytes not sent from first one >>> (since first one deserializes fine). It seems like I'm using correct api >>> call to do serialization (and it works with other data combos/types) >>> >>> The second one, I agree is not right. Just gotta figure out why :| >>> >>> >>> >>> On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote: >>>> >>>> Oddly neither of those is correct! The actual hex should be: >>>> >>>> 08-03-1A-06-08-01-10-63-18-01 >>>> >>>> which is the first version minus some trailing data. The second version >>>> is not valid at all - it isn't just missing data: it is an incomplete and >>>> invalid payload. Unfortunately I'm not familiar enough with the C++ API to >>>> advise on whether you're using the wrong method vs this being a bug. >>>> >>>> >>>> On Wed, 11 Apr 2018, 01:12 Tony Tony, <[email protected]> wrote: >>>> >>>>> So I upgraded to 3.5.0, same issue. >>>>> >>>>> I guess I should give some more information: I'm basically using >>>>> protobuf to serialize/deserialize string data I'm sending through a games >>>>> socket implementation via hooks/code caves in the games network engine. I >>>>> currently can validate that the bytes after serialization are sent over >>>>> completely to the client. After looking at the decoded output that you >>>>> guys >>>>> recommended, I do see some differences. >>>>> >>>>> - When I send type=1, count=99, index=1, the server shows the below >>>>> bytes after serialization and the client shows these bytes after receiving >>>>> the data. The decoding tool shows the correct content. >>>>> >>>>> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD >>>>> >>>>> - When I send type=1, count=0, index=1, the server shows the below >>>>> bytes after serialization and the client shows these bytes after receiving >>>>> the data. The decoding tool basically shows no content, which is obvious >>>>> from the missing data. >>>>> >>>>> 08 03 1A 06 08 >>>>> >>>>> The biggest difference is that it seems the during serialization on >>>>> the server, this combo of data seems to break everything. I guess I'm not >>>>> sure why this combo of data would break during serialization. I posted >>>>> some >>>>> pseudo code of where the problem occurs for that combo of data below. It's >>>>> just strange that only that combo of data above results in this behavior. >>>>> I >>>>> am able to send large strings just fine, plenty of other data types with >>>>> no >>>>> issues (well none that I've found yet..) >>>>> >>>>> Packet pak; >>>>> pak.set_type(Packet_Type_set_info); >>>>> >>>>> set_info *setter = pak.mutable_set_info(); >>>>> setter->set_count(99); >>>>> setter->set_type(0); >>>>> setter->set_index(1); >>>>> >>>>> char* SendBuf = new char[pak.ByteSize()]; >>>>> pak.SerializeToArray(SendBuf, pak.ByteSize()); >>>>> //TODO: the bytes seem to be wrong right after this point for the >>>>> second combo of data >>>>> //we send before we delete the sendBuf >>>>> >>>>> delete[] SendBuf; >>>>> >>>>> >>>>> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote: >>>>>> >>>>>> I think Marc is right that it would be best to isolate the problem by >>>>>> manually checking that the bytes are valid. >>>>>> >>>>>> Tony, how are you framing the messages? Protocol buffers do not >>>>>> describe their own size and so you have to know what size to expect >>>>>> before >>>>>> you parse a proto. The typical approach is to frame the serialized >>>>>> messages >>>>>> by prefixing them with their size. It's easy to get this wrong, though, >>>>>> so >>>>>> that could potentially be the reason that the messages aren't showing up >>>>>> as >>>>>> you expect. >>>>>> >>>>>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Well, if it was me: the first thing I'd do is isolate whether it is >>>>>>> serialize or deserialize that is failing - by taking your currently >>>>>>> serialized data as a flat file or byte dump, and checking whether the >>>>>>> values are right or wrong. >>>>>>> >>>>>>> Depending on the volume and layout of the data, you might be able to >>>>>>> use protoc's inbuilt decode mechanisms, or (as long as it isn't huge) >>>>>>> you're welcome to try using https://protogen.marcgravell.com/decode >>>>>>> >>>>>>> That would allow you to focus your checks. >>>>>>> >>>>>>> Marc >>>>>>> >>>>>>> On 10 April 2018 at 14:20, Tony Tony <[email protected]> wrote: >>>>>>> >>>>>>>> c++/windows >>>>>>>> >>>>>>>> I'll upgrade protobuf and see if issue goes away. Just wanted to >>>>>>>> make sure I'm not missing anything. Is there any additional >>>>>>>> troubleshooting >>>>>>>> I can look into to troubleshoot further if the recent release >>>>>>>> reproduce the >>>>>>>> issue? >>>>>>>> >>>>>>>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote: >>>>>>>>> >>>>>>>>> First thought: what language / platform is this? the C++ generated >>>>>>>>> code is very different to the Java generated code, for example >>>>>>>>> >>>>>>>>> Second thought: 2.6 is pretty old; it is very possible that a bug >>>>>>>>> existed and has been fixed since then (Aug 2014) - does it still >>>>>>>>> happen >>>>>>>>> with more recent releases? >>>>>>>>> >>>>>>>>> The scenario you present seems fine and reasonable. >>>>>>>>> >>>>>>>>> Marc >>>>>>>>> >>>>>>>>> On 10 April 2018 at 03:46, Tony Tony <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have a strange case I'm dealing with. Using protobuf compiler >>>>>>>>>> 2.6. >>>>>>>>>> >>>>>>>>>> Every time I send a certain set of values over my socket, it ends >>>>>>>>>> up with missing data. Although I've compared the string received >>>>>>>>>> after >>>>>>>>>> serialization on the server and on receiving the string on the >>>>>>>>>> client and >>>>>>>>>> they are exactly the same, yet the values are all -1. >>>>>>>>>> >>>>>>>>>> I've attached a snippet of the proto file below. >>>>>>>>>> >>>>>>>>>> At this point these are below combos that always fail >>>>>>>>>> >>>>>>>>>> type=99, count= 0, index = 1 (all values are -1 after >>>>>>>>>> deserialization) >>>>>>>>>> type=99, count=99, index= 1 (works) >>>>>>>>>> type=99999, count=999999, index= 1 (works) >>>>>>>>>> >>>>>>>>>> - Basically anytime count is 0, all the values seem to be -1. >>>>>>>>>> However if I make type 0, the same behavior does not occur. Am I >>>>>>>>>> missing >>>>>>>>>> something or there is some underlying bug in my code? Again I've >>>>>>>>>> checked >>>>>>>>>> multiple times the correct raw string data is being sent (with >>>>>>>>>> correct >>>>>>>>>> size). I use the parseFromString method to parse the receiving >>>>>>>>>> string data. >>>>>>>>>> >>>>>>>>>> - I've sent plenty of other types of data just fine. >>>>>>>>>> >>>>>>>>>> - There are other messages defined in the same protobuf packet >>>>>>>>>> definition file, can that make a difference? >>>>>>>>>> >>>>>>>>>> message packet { >>>>>>>>>> required int32 type = 1 [default = -1]; >>>>>>>>>> required int32 count = 2 [default = -1]; >>>>>>>>>> required int32 index = 3 [default = -1]; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Protocol Buffers" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Marc >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Protocol Buffers" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Marc >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Protocol Buffers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
