Extra data at the end would definitely be a problem. That could cause parsing to return an error, or it might happen to be in the right format so that it gets parsed as an unknown field (or even a regular known field).
If the serialization returns an error there isn't an API to get more details about the error--you just have to examine the data manually if you want to figure out what went wrong. On Fri, Apr 13, 2018 at 6:42 AM Tony Tony <xxteknolus...@gmail.com> wrote: > For both messages I think it returned 10, although the actual data for the > second one was wrong. I think that extra data might just be copying extra > data from the string printed in my log. Would extra data at the end of > serialized data affect actual deserialization? Cause the first one > deserialized just fine. > > I'll check what SerializeToArray is returning when I get home, if it > returns false is there somewhere the error would get logged to? Or is there > some sort of GetError call in protobuf you can call after certain protobuf > methods? > > Thanks again > > On Thursday, April 12, 2018 at 12:32:15 PM UTC-4, Adam Cozzette wrote: >> >> What does pak.ByteSize() return for those two messages? It would also be >> good to verify that SerializeToArray() is returning true, because if not >> then there could be an error being missed. >> >> Something seems to be going wrong with the message sizes, because the >> first data example seems to have extra data on the end while the second one >> seems to be too short (truncated). >> >> By the way, Marc, I tried out your protogen tool just now and it's pretty >> cool! That's more convenient than having to fiddle around with creating the >> binary data file and running protoc --decode_raw on it. >> >> On Wed, Apr 11, 2018 at 3:25 PM Tony Tony <xxtekn...@gmail.com> wrote: >> >>> Well I just copied whatever text was printed after i logged what was >>> being serialized. So I probably copied extra bytes not sent from first one >>> (since first one deserializes fine). It seems like I'm using correct api >>> call to do serialization (and it works with other data combos/types) >>> >>> The second one, I agree is not right. Just gotta figure out why :| >>> >>> >>> >>> On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote: >>>> >>>> Oddly neither of those is correct! The actual hex should be: >>>> >>>> 08-03-1A-06-08-01-10-63-18-01 >>>> >>>> which is the first version minus some trailing data. The second version >>>> is not valid at all - it isn't just missing data: it is an incomplete and >>>> invalid payload. Unfortunately I'm not familiar enough with the C++ API to >>>> advise on whether you're using the wrong method vs this being a bug. >>>> >>>> >>>> On Wed, 11 Apr 2018, 01:12 Tony Tony, <xxtekn...@gmail.com> wrote: >>>> >>>>> So I upgraded to 3.5.0, same issue. >>>>> >>>>> I guess I should give some more information: I'm basically using >>>>> protobuf to serialize/deserialize string data I'm sending through a games >>>>> socket implementation via hooks/code caves in the games network engine. I >>>>> currently can validate that the bytes after serialization are sent over >>>>> completely to the client. After looking at the decoded output that you >>>>> guys >>>>> recommended, I do see some differences. >>>>> >>>>> - When I send type=1, count=99, index=1, the server shows the below >>>>> bytes after serialization and the client shows these bytes after receiving >>>>> the data. The decoding tool shows the correct content. >>>>> >>>>> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD >>>>> >>>>> - When I send type=1, count=0, index=1, the server shows the below >>>>> bytes after serialization and the client shows these bytes after receiving >>>>> the data. The decoding tool basically shows no content, which is obvious >>>>> from the missing data. >>>>> >>>>> 08 03 1A 06 08 >>>>> >>>>> The biggest difference is that it seems the during serialization on >>>>> the server, this combo of data seems to break everything. I guess I'm not >>>>> sure why this combo of data would break during serialization. I posted >>>>> some >>>>> pseudo code of where the problem occurs for that combo of data below. It's >>>>> just strange that only that combo of data above results in this behavior. >>>>> I >>>>> am able to send large strings just fine, plenty of other data types with >>>>> no >>>>> issues (well none that I've found yet..) >>>>> >>>>> Packet pak; >>>>> pak.set_type(Packet_Type_set_info); >>>>> >>>>> set_info *setter = pak.mutable_set_info(); >>>>> setter->set_count(99); >>>>> setter->set_type(0); >>>>> setter->set_index(1); >>>>> >>>>> char* SendBuf = new char[pak.ByteSize()]; >>>>> pak.SerializeToArray(SendBuf, pak.ByteSize()); >>>>> //TODO: the bytes seem to be wrong right after this point for the >>>>> second combo of data >>>>> //we send before we delete the sendBuf >>>>> >>>>> delete[] SendBuf; >>>>> >>>>> >>>>> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote: >>>>>> >>>>>> I think Marc is right that it would be best to isolate the problem by >>>>>> manually checking that the bytes are valid. >>>>>> >>>>>> Tony, how are you framing the messages? Protocol buffers do not >>>>>> describe their own size and so you have to know what size to expect >>>>>> before >>>>>> you parse a proto. The typical approach is to frame the serialized >>>>>> messages >>>>>> by prefixing them with their size. It's easy to get this wrong, though, >>>>>> so >>>>>> that could potentially be the reason that the messages aren't showing up >>>>>> as >>>>>> you expect. >>>>>> >>>>>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell <marc.g...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Well, if it was me: the first thing I'd do is isolate whether it is >>>>>>> serialize or deserialize that is failing - by taking your currently >>>>>>> serialized data as a flat file or byte dump, and checking whether the >>>>>>> values are right or wrong. >>>>>>> >>>>>>> Depending on the volume and layout of the data, you might be able to >>>>>>> use protoc's inbuilt decode mechanisms, or (as long as it isn't huge) >>>>>>> you're welcome to try using https://protogen.marcgravell.com/decode >>>>>>> >>>>>>> That would allow you to focus your checks. >>>>>>> >>>>>>> Marc >>>>>>> >>>>>>> On 10 April 2018 at 14:20, Tony Tony <xxtekn...@gmail.com> wrote: >>>>>>> >>>>>>>> c++/windows >>>>>>>> >>>>>>>> I'll upgrade protobuf and see if issue goes away. Just wanted to >>>>>>>> make sure I'm not missing anything. Is there any additional >>>>>>>> troubleshooting >>>>>>>> I can look into to troubleshoot further if the recent release >>>>>>>> reproduce the >>>>>>>> issue? >>>>>>>> >>>>>>>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote: >>>>>>>>> >>>>>>>>> First thought: what language / platform is this? the C++ generated >>>>>>>>> code is very different to the Java generated code, for example >>>>>>>>> >>>>>>>>> Second thought: 2.6 is pretty old; it is very possible that a bug >>>>>>>>> existed and has been fixed since then (Aug 2014) - does it still >>>>>>>>> happen >>>>>>>>> with more recent releases? >>>>>>>>> >>>>>>>>> The scenario you present seems fine and reasonable. >>>>>>>>> >>>>>>>>> Marc >>>>>>>>> >>>>>>>>> On 10 April 2018 at 03:46, Tony Tony <xxtekn...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have a strange case I'm dealing with. Using protobuf compiler >>>>>>>>>> 2.6. >>>>>>>>>> >>>>>>>>>> Every time I send a certain set of values over my socket, it ends >>>>>>>>>> up with missing data. Although I've compared the string received >>>>>>>>>> after >>>>>>>>>> serialization on the server and on receiving the string on the >>>>>>>>>> client and >>>>>>>>>> they are exactly the same, yet the values are all -1. >>>>>>>>>> >>>>>>>>>> I've attached a snippet of the proto file below. >>>>>>>>>> >>>>>>>>>> At this point these are below combos that always fail >>>>>>>>>> >>>>>>>>>> type=99, count= 0, index = 1 (all values are -1 after >>>>>>>>>> deserialization) >>>>>>>>>> type=99, count=99, index= 1 (works) >>>>>>>>>> type=99999, count=999999, index= 1 (works) >>>>>>>>>> >>>>>>>>>> - Basically anytime count is 0, all the values seem to be -1. >>>>>>>>>> However if I make type 0, the same behavior does not occur. Am I >>>>>>>>>> missing >>>>>>>>>> something or there is some underlying bug in my code? Again I've >>>>>>>>>> checked >>>>>>>>>> multiple times the correct raw string data is being sent (with >>>>>>>>>> correct >>>>>>>>>> size). I use the parseFromString method to parse the receiving >>>>>>>>>> string data. >>>>>>>>>> >>>>>>>>>> - I've sent plenty of other types of data just fine. >>>>>>>>>> >>>>>>>>>> - There are other messages defined in the same protobuf packet >>>>>>>>>> definition file, can that make a difference? >>>>>>>>>> >>>>>>>>>> message packet { >>>>>>>>>> required int32 type = 1 [default = -1]; >>>>>>>>>> required int32 count = 2 [default = -1]; >>>>>>>>>> required int32 index = 3 [default = -1]; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Protocol Buffers" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to protobuf+u...@googlegroups.com. >>>>>>>>>> To post to this group, send email to prot...@googlegroups.com. >>>>>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Marc >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Protocol Buffers" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to protobuf+u...@googlegroups.com. >>>>>>>> To post to this group, send email to prot...@googlegroups.com. >>>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Marc >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Protocol Buffers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to protobuf+u...@googlegroups.com. >>>>>>> To post to this group, send email to prot...@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to protobuf+u...@googlegroups.com. >>>>> To post to this group, send email to prot...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to protobuf+u...@googlegroups.com. >>> To post to this group, send email to prot...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
