Re: [protobuf] Protobuf 2.6 serialization weird behavior

['Adam Cozzette' via Protocol Buffers]

Extra data at the end would definitely be a problem. That could cause
parsing to return an error, or it might happen to be in the right format so
that it gets parsed as an unknown field (or even a regular known field).

If the serialization returns an error there isn't an API to get more
details about the error--you just have to examine the data manually if you
want to figure out what went wrong.

On Fri, Apr 13, 2018 at 6:42 AM Tony Tony  wrote:

> For both messages I think it returned 10, although the actual data for the
> second one was wrong. I think that extra data might just be copying extra
> data from the string printed in my log. Would extra data at the end of
> serialized data affect actual deserialization? Cause the first one
> deserialized just fine.
>
> I'll check what SerializeToArray is returning when I get home, if it
> returns false is there somewhere the error would get logged to? Or is there
> some sort of GetError call in protobuf you can call after certain protobuf
> methods?
>
> Thanks again
>
> On Thursday, April 12, 2018 at 12:32:15 PM UTC-4, Adam Cozzette wrote:
>>
>> What does pak.ByteSize() return for those two messages? It would also be
>> good to verify that SerializeToArray() is returning true, because if not
>> then there could be an error being missed.
>>
>> Something seems to be going wrong with the message sizes, because the
>> first data example seems to have extra data on the end while the second one
>> seems to be too short (truncated).
>>
>> By the way, Marc, I tried out your protogen tool just now and it's pretty
>> cool! That's more convenient than having to fiddle around with creating the
>> binary data file and running protoc --decode_raw on it.
>>
>> On Wed, Apr 11, 2018 at 3:25 PM Tony Tony  wrote:
>>
>>> Well I just copied whatever text was printed after i logged what was
>>> being serialized. So I probably copied extra bytes not sent from first one
>>> (since first one deserializes fine). It seems like I'm using correct api
>>> call to do serialization (and it works with other data combos/types)
>>>
>>> The second one, I agree is not right. Just gotta figure out why :|
>>>
>>>
>>>
>>> On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote:

 Oddly neither of those is correct! The actual hex should be:

 08-03-1A-06-08-01-10-63-18-01

 which is the first version minus some trailing data. The second version
 is not valid at all - it isn't just missing data: it is an incomplete and
 invalid payload. Unfortunately I'm not familiar enough with the C++ API to
 advise on whether you're using the wrong method vs this being a bug.


 On Wed, 11 Apr 2018, 01:12 Tony Tony,  wrote:

> So I upgraded to 3.5.0, same issue.
>
> I guess I should give some more information: I'm basically using
> protobuf to serialize/deserialize string data I'm sending through a games
> socket implementation via hooks/code caves in the games network engine. I
> currently can validate that the bytes after serialization are sent over
> completely to the client. After looking at the decoded output that you 
> guys
> recommended, I do see some differences.
>
> - When I send type=1, count=99, index=1, the server shows the below
> bytes after serialization and the client shows these bytes after receiving
> the data. The decoding tool shows the correct content.
>
> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>
> - When I send type=1, count=0, index=1, the server shows the below
> bytes after serialization and the client shows these bytes after receiving
> the data. The decoding tool basically shows no content, which is obvious
> from the missing data.
>
> 08 03 1A 06 08
>
> The biggest difference is that it seems the during serialization on
> the server, this combo of data seems to break everything. I guess I'm not
> sure why this combo of data would break during serialization. I posted 
> some
> pseudo code of where the problem occurs for that combo of data below. It's
> just strange that only that combo of data above results in this behavior. 
> I
> am able to send large strings just fine, plenty of other data types with 
> no
> issues (well none that I've found yet..)
>
> Packet pak;
> pak.set_type(Packet_Type_set_info);
>
> set_info *setter = pak.mutable_set_info();
> setter->set_count(99);
> setter->set_type(0);
> setter->set_index(1);
>
> char* SendBuf = new char[pak.ByteSize()];
> pak.SerializeToArray(SendBuf, pak.ByteSize());
> //TODO: the bytes seem to be wrong right after this point for the
> second combo of data
> //we send before we delete the sendBuf
>
> delete[] SendBuf;
>
>
> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>>

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Tony Tony]

For both messages I think it returned 10, although the actual data for the 
second one was wrong. I think that extra data might just be copying extra 
data from the string printed in my log. Would extra data at the end of 
serialized data affect actual deserialization? Cause the first one 
deserialized just fine.

I'll check what SerializeToArray is returning when I get home, if it 
returns false is there somewhere the error would get logged to? Or is there 
some sort of GetError call in protobuf you can call after certain protobuf 
methods?

Thanks again

On Thursday, April 12, 2018 at 12:32:15 PM UTC-4, Adam Cozzette wrote:
>
> What does pak.ByteSize() return for those two messages? It would also be 
> good to verify that SerializeToArray() is returning true, because if not 
> then there could be an error being missed.
>
> Something seems to be going wrong with the message sizes, because the 
> first data example seems to have extra data on the end while the second one 
> seems to be too short (truncated).
>
> By the way, Marc, I tried out your protogen tool just now and it's pretty 
> cool! That's more convenient than having to fiddle around with creating the 
> binary data file and running protoc --decode_raw on it.
>
> On Wed, Apr 11, 2018 at 3:25 PM Tony Tony  > wrote:
>
>> Well I just copied whatever text was printed after i logged what was 
>> being serialized. So I probably copied extra bytes not sent from first one 
>> (since first one deserializes fine). It seems like I'm using correct api 
>> call to do serialization (and it works with other data combos/types)
>>
>> The second one, I agree is not right. Just gotta figure out why :|
>>
>>
>>
>> On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote:
>>>
>>> Oddly neither of those is correct! The actual hex should be:
>>>
>>> 08-03-1A-06-08-01-10-63-18-01
>>>
>>> which is the first version minus some trailing data. The second version 
>>> is not valid at all - it isn't just missing data: it is an incomplete and 
>>> invalid payload. Unfortunately I'm not familiar enough with the C++ API to 
>>> advise on whether you're using the wrong method vs this being a bug.
>>>
>>>
>>> On Wed, 11 Apr 2018, 01:12 Tony Tony,  wrote:
>>>
 So I upgraded to 3.5.0, same issue.

 I guess I should give some more information: I'm basically using 
 protobuf to serialize/deserialize string data I'm sending through a games 
 socket implementation via hooks/code caves in the games network engine. I 
 currently can validate that the bytes after serialization are sent over 
 completely to the client. After looking at the decoded output that you 
 guys 
 recommended, I do see some differences.

 - When I send type=1, count=99, index=1, the server shows the below 
 bytes after serialization and the client shows these bytes after receiving 
 the data. The decoding tool shows the correct content.

 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD

 - When I send type=1, count=0, index=1, the server shows the below 
 bytes after serialization and the client shows these bytes after receiving 
 the data. The decoding tool basically shows no content, which is obvious 
 from the missing data.

 08 03 1A 06 08

 The biggest difference is that it seems the during serialization on the 
 server, this combo of data seems to break everything. I guess I'm not sure 
 why this combo of data would break during serialization. I posted some 
 pseudo code of where the problem occurs for that combo of data below. It's 
 just strange that only that combo of data above results in this behavior. 
 I 
 am able to send large strings just fine, plenty of other data types with 
 no 
 issues (well none that I've found yet..)

 Packet pak;
 pak.set_type(Packet_Type_set_info);

 set_info *setter = pak.mutable_set_info();
 setter->set_count(99);
 setter->set_type(0);
 setter->set_index(1);

 char* SendBuf = new char[pak.ByteSize()];
 pak.SerializeToArray(SendBuf, pak.ByteSize());
 //TODO: the bytes seem to be wrong right after this point for the 
 second combo of data
 //we send before we delete the sendBuf

 delete[] SendBuf;


 On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>
> I think Marc is right that it would be best to isolate the problem by 
> manually checking that the bytes are valid.
>
> Tony, how are you framing the messages? Protocol buffers do not 
> describe their own size and so you have to know what size to expect 
> before 
> you parse a proto. The typical approach is to frame the serialized 
> messages 
> by prefixing them with their size. It's easy to get this wrong, though, 
> so 
> that could potentially be the reason that the messages aren't showing up 
> as 

Re: [protobuf] Protobuf 2.6 serialization weird behavior

['Adam Cozzette' via Protocol Buffers]

What does pak.ByteSize() return for those two messages? It would also be
good to verify that SerializeToArray() is returning true, because if not
then there could be an error being missed.

Something seems to be going wrong with the message sizes, because the first
data example seems to have extra data on the end while the second one seems
to be too short (truncated).

By the way, Marc, I tried out your protogen tool just now and it's pretty
cool! That's more convenient than having to fiddle around with creating the
binary data file and running protoc --decode_raw on it.

On Wed, Apr 11, 2018 at 3:25 PM Tony Tony  wrote:

> Well I just copied whatever text was printed after i logged what was being
> serialized. So I probably copied extra bytes not sent from first one (since
> first one deserializes fine). It seems like I'm using correct api call to
> do serialization (and it works with other data combos/types)
>
> The second one, I agree is not right. Just gotta figure out why :|
>
>
>
> On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote:
>>
>> Oddly neither of those is correct! The actual hex should be:
>>
>> 08-03-1A-06-08-01-10-63-18-01
>>
>> which is the first version minus some trailing data. The second version
>> is not valid at all - it isn't just missing data: it is an incomplete and
>> invalid payload. Unfortunately I'm not familiar enough with the C++ API to
>> advise on whether you're using the wrong method vs this being a bug.
>>
>>
>> On Wed, 11 Apr 2018, 01:12 Tony Tony,  wrote:
>>
>>> So I upgraded to 3.5.0, same issue.
>>>
>>> I guess I should give some more information: I'm basically using
>>> protobuf to serialize/deserialize string data I'm sending through a games
>>> socket implementation via hooks/code caves in the games network engine. I
>>> currently can validate that the bytes after serialization are sent over
>>> completely to the client. After looking at the decoded output that you guys
>>> recommended, I do see some differences.
>>>
>>> - When I send type=1, count=99, index=1, the server shows the below
>>> bytes after serialization and the client shows these bytes after receiving
>>> the data. The decoding tool shows the correct content.
>>>
>>> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>>>
>>> - When I send type=1, count=0, index=1, the server shows the below bytes
>>> after serialization and the client shows these bytes after receiving the
>>> data. The decoding tool basically shows no content, which is obvious from
>>> the missing data.
>>>
>>> 08 03 1A 06 08
>>>
>>> The biggest difference is that it seems the during serialization on the
>>> server, this combo of data seems to break everything. I guess I'm not sure
>>> why this combo of data would break during serialization. I posted some
>>> pseudo code of where the problem occurs for that combo of data below. It's
>>> just strange that only that combo of data above results in this behavior. I
>>> am able to send large strings just fine, plenty of other data types with no
>>> issues (well none that I've found yet..)
>>>
>>> Packet pak;
>>> pak.set_type(Packet_Type_set_info);
>>>
>>> set_info *setter = pak.mutable_set_info();
>>> setter->set_count(99);
>>> setter->set_type(0);
>>> setter->set_index(1);
>>>
>>> char* SendBuf = new char[pak.ByteSize()];
>>> pak.SerializeToArray(SendBuf, pak.ByteSize());
>>> //TODO: the bytes seem to be wrong right after this point for the second
>>> combo of data
>>> //we send before we delete the sendBuf
>>>
>>> delete[] SendBuf;
>>>
>>>
>>> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:

 I think Marc is right that it would be best to isolate the problem by
 manually checking that the bytes are valid.

 Tony, how are you framing the messages? Protocol buffers do not
 describe their own size and so you have to know what size to expect before
 you parse a proto. The typical approach is to frame the serialized messages
 by prefixing them with their size. It's easy to get this wrong, though, so
 that could potentially be the reason that the messages aren't showing up as
 you expect.

 On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell 
 wrote:

> Well, if it was me: the first thing I'd do is isolate whether it is
> serialize or deserialize that is failing - by taking your currently
> serialized data as a flat file or byte dump, and checking whether the
> values are right or wrong.
>
> Depending on the volume and layout of the data, you might be able to
> use protoc's inbuilt decode mechanisms, or (as long as it isn't huge)
> you're welcome to try using https://protogen.marcgravell.com/decode
>
> That would allow you to focus your checks.
>
> Marc
>
> On 10 April 2018 at 14:20, Tony Tony  wrote:
>
>> c++/windows
>>
>> I'll upgrade protobuf and see if 

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Tony Tony]

Well I just copied whatever text was printed after i logged what was being 
serialized. So I probably copied extra bytes not sent from first one (since 
first one deserializes fine). It seems like I'm using correct api call to 
do serialization (and it works with other data combos/types)

The second one, I agree is not right. Just gotta figure out why :|



On Tuesday, April 10, 2018 at 11:36:29 PM UTC-4, Marc Gravell wrote:
>
> Oddly neither of those is correct! The actual hex should be:
>
> 08-03-1A-06-08-01-10-63-18-01
>
> which is the first version minus some trailing data. The second version is 
> not valid at all - it isn't just missing data: it is an incomplete and 
> invalid payload. Unfortunately I'm not familiar enough with the C++ API to 
> advise on whether you're using the wrong method vs this being a bug.
>
>
> On Wed, 11 Apr 2018, 01:12 Tony Tony,  
> wrote:
>
>> So I upgraded to 3.5.0, same issue.
>>
>> I guess I should give some more information: I'm basically using protobuf 
>> to serialize/deserialize string data I'm sending through a games socket 
>> implementation via hooks/code caves in the games network engine. I 
>> currently can validate that the bytes after serialization are sent over 
>> completely to the client. After looking at the decoded output that you guys 
>> recommended, I do see some differences.
>>
>> - When I send type=1, count=99, index=1, the server shows the below bytes 
>> after serialization and the client shows these bytes after receiving the 
>> data. The decoding tool shows the correct content.
>>
>> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>>
>> - When I send type=1, count=0, index=1, the server shows the below bytes 
>> after serialization and the client shows these bytes after receiving the 
>> data. The decoding tool basically shows no content, which is obvious from 
>> the missing data.
>>
>> 08 03 1A 06 08
>>
>> The biggest difference is that it seems the during serialization on the 
>> server, this combo of data seems to break everything. I guess I'm not sure 
>> why this combo of data would break during serialization. I posted some 
>> pseudo code of where the problem occurs for that combo of data below. It's 
>> just strange that only that combo of data above results in this behavior. I 
>> am able to send large strings just fine, plenty of other data types with no 
>> issues (well none that I've found yet..)
>>
>> Packet pak;
>> pak.set_type(Packet_Type_set_info);
>>
>> set_info *setter = pak.mutable_set_info();
>> setter->set_count(99);
>> setter->set_type(0);
>> setter->set_index(1);
>>
>> char* SendBuf = new char[pak.ByteSize()];
>> pak.SerializeToArray(SendBuf, pak.ByteSize());
>> //TODO: the bytes seem to be wrong right after this point for the second 
>> combo of data
>> //we send before we delete the sendBuf
>>
>> delete[] SendBuf;
>>
>>
>> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>>>
>>> I think Marc is right that it would be best to isolate the problem by 
>>> manually checking that the bytes are valid.
>>>
>>> Tony, how are you framing the messages? Protocol buffers do not describe 
>>> their own size and so you have to know what size to expect before you parse 
>>> a proto. The typical approach is to frame the serialized messages by 
>>> prefixing them with their size. It's easy to get this wrong, though, so 
>>> that could potentially be the reason that the messages aren't showing up as 
>>> you expect.
>>>
>>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell  
>>> wrote:
>>>
 Well, if it was me: the first thing I'd do is isolate whether it is 
 serialize or deserialize that is failing - by taking your currently 
 serialized data as a flat file or byte dump, and checking whether the 
 values are right or wrong.

 Depending on the volume and layout of the data, you might be able to 
 use protoc's inbuilt decode mechanisms, or (as long as it isn't huge) 
 you're welcome to try using https://protogen.marcgravell.com/decode

 That would allow you to focus your checks.

 Marc

 On 10 April 2018 at 14:20, Tony Tony  wrote:

> c++/windows
>
> I'll upgrade protobuf and see if issue goes away. Just wanted to make 
> sure I'm not missing anything. Is there any additional troubleshooting I 
> can look into to troubleshoot further if the recent release reproduce the 
> issue?
>
> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>
>> First thought: what language / platform is this? the C++ generated 
>> code is very different to the Java generated code, for example
>>
>> Second thought: 2.6 is pretty old; it is very possible that a bug 
>> existed and has been fixed since then (Aug 2014) - does it still happen 
>> with more recent releases?
>>
>> The scenario you present seems fine and reasonable.
>>
>> Marc

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Oddly neither of those is correct! The actual hex should be:

08-03-1A-06-08-01-10-63-18-01

which is the first version minus some trailing data. The second version is
not valid at all - it isn't just missing data: it is an incomplete and
invalid payload. Unfortunately I'm not familiar enough with the C++ API to
advise on whether you're using the wrong method vs this being a bug.


On Wed, 11 Apr 2018, 01:12 Tony Tony,  wrote:

> So I upgraded to 3.5.0, same issue.
>
> I guess I should give some more information: I'm basically using protobuf
> to serialize/deserialize string data I'm sending through a games socket
> implementation via hooks/code caves in the games network engine. I
> currently can validate that the bytes after serialization are sent over
> completely to the client. After looking at the decoded output that you guys
> recommended, I do see some differences.
>
> - When I send type=1, count=99, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool shows the correct content.
>
> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>
> - When I send type=1, count=0, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool basically shows no content, which is obvious from
> the missing data.
>
> 08 03 1A 06 08
>
> The biggest difference is that it seems the during serialization on the
> server, this combo of data seems to break everything. I guess I'm not sure
> why this combo of data would break during serialization. I posted some
> pseudo code of where the problem occurs for that combo of data below. It's
> just strange that only that combo of data above results in this behavior. I
> am able to send large strings just fine, plenty of other data types with no
> issues (well none that I've found yet..)
>
> Packet pak;
> pak.set_type(Packet_Type_set_info);
>
> set_info *setter = pak.mutable_set_info();
> setter->set_count(99);
> setter->set_type(0);
> setter->set_index(1);
>
> char* SendBuf = new char[pak.ByteSize()];
> pak.SerializeToArray(SendBuf, pak.ByteSize());
> //TODO: the bytes seem to be wrong right after this point for the second
> combo of data
> //we send before we delete the sendBuf
>
> delete[] SendBuf;
>
>
> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>>
>> I think Marc is right that it would be best to isolate the problem by
>> manually checking that the bytes are valid.
>>
>> Tony, how are you framing the messages? Protocol buffers do not describe
>> their own size and so you have to know what size to expect before you parse
>> a proto. The typical approach is to frame the serialized messages by
>> prefixing them with their size. It's easy to get this wrong, though, so
>> that could potentially be the reason that the messages aren't showing up as
>> you expect.
>>
>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell  wrote:
>>
>>> Well, if it was me: the first thing I'd do is isolate whether it is
>>> serialize or deserialize that is failing - by taking your currently
>>> serialized data as a flat file or byte dump, and checking whether the
>>> values are right or wrong.
>>>
>>> Depending on the volume and layout of the data, you might be able to use
>>> protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
>>> welcome to try using https://protogen.marcgravell.com/decode
>>>
>>> That would allow you to focus your checks.
>>>
>>> Marc
>>>
>>> On 10 April 2018 at 14:20, Tony Tony  wrote:
>>>
 c++/windows

 I'll upgrade protobuf and see if issue goes away. Just wanted to make
 sure I'm not missing anything. Is there any additional troubleshooting I
 can look into to troubleshoot further if the recent release reproduce the
 issue?

 On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>
> First thought: what language / platform is this? the C++ generated
> code is very different to the Java generated code, for example
>
> Second thought: 2.6 is pretty old; it is very possible that a bug
> existed and has been fixed since then (Aug 2014) - does it still happen
> with more recent releases?
>
> The scenario you present seems fine and reasonable.
>
> Marc
>
> On 10 April 2018 at 03:46, Tony Tony  wrote:
>
>> Hi,
>>
>> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>>
>> Every time I send a certain set of values over my socket, it ends up
>> with missing data. Although I've compared the string received after
>> serialization on the server and on receiving the string on the client and
>> they are exactly the same, yet the values are all -1.
>>
>> I've attached a snippet of the proto file below.
>>
>> At this point these are below 

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Tony Tony]

So I upgraded to 3.5.0, same issue.

I guess I should give some more information: I'm basically using protobuf 
to serialize/deserialize string data I'm sending through a games socket 
implementation via hooks/code caves in the games network engine. I 
currently can validate that the bytes after serialization are sent over 
completely to the client. After looking at the decoded output that you guys 
recommended, I do see some differences.

- When I send type=1, count=99, index=1, the server shows the below bytes 
after serialization and the client shows these bytes after receiving the 
data. The decoding tool shows the correct content.

08 03 1A 06 08 01 10 63 18 01 FD FD FD FD

- When I send type=1, count=0, index=1, the server shows the below bytes 
after serialization and the client shows these bytes after receiving the 
data. The decoding tool basically shows no content, which is obvious from 
the missing data.

08 03 1A 06 08

The biggest difference is that it seems the during serialization on the 
server, this combo of data seems to break everything. I guess I'm not sure 
why this combo of data would break during serialization. I posted some 
pseudo code of where the problem occurs for that combo of data below. It's 
just strange that only that combo of data above results in this behavior. I 
am able to send large strings just fine, plenty of other data types with no 
issues (well none that I've found yet..)

Packet pak;
pak.set_type(Packet_Type_set_info);

set_info *setter = pak.mutable_set_info();
setter->set_count(99);
setter->set_type(0);
setter->set_index(1);

char* SendBuf = new char[pak.ByteSize()];
pak.SerializeToArray(SendBuf, pak.ByteSize());
//TODO: the bytes seem to be wrong right after this point for the second 
combo of data
//we send before we delete the sendBuf

delete[] SendBuf;


On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>
> I think Marc is right that it would be best to isolate the problem by 
> manually checking that the bytes are valid.
>
> Tony, how are you framing the messages? Protocol buffers do not describe 
> their own size and so you have to know what size to expect before you parse 
> a proto. The typical approach is to frame the serialized messages by 
> prefixing them with their size. It's easy to get this wrong, though, so 
> that could potentially be the reason that the messages aren't showing up as 
> you expect.
>
> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell  > wrote:
>
>> Well, if it was me: the first thing I'd do is isolate whether it is 
>> serialize or deserialize that is failing - by taking your currently 
>> serialized data as a flat file or byte dump, and checking whether the 
>> values are right or wrong.
>>
>> Depending on the volume and layout of the data, you might be able to use 
>> protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're 
>> welcome to try using https://protogen.marcgravell.com/decode
>>
>> That would allow you to focus your checks.
>>
>> Marc
>>
>> On 10 April 2018 at 14:20, Tony Tony  
>> wrote:
>>
>>> c++/windows
>>>
>>> I'll upgrade protobuf and see if issue goes away. Just wanted to make 
>>> sure I'm not missing anything. Is there any additional troubleshooting I 
>>> can look into to troubleshoot further if the recent release reproduce the 
>>> issue?
>>>
>>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:

 First thought: what language / platform is this? the C++ generated code 
 is very different to the Java generated code, for example

 Second thought: 2.6 is pretty old; it is very possible that a bug 
 existed and has been fixed since then (Aug 2014) - does it still happen 
 with more recent releases?

 The scenario you present seems fine and reasonable.

 Marc

 On 10 April 2018 at 03:46, Tony Tony  wrote:

> Hi,
>
> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>
> Every time I send a certain set of values over my socket, it ends up 
> with missing data. Although I've compared the string received after 
> serialization on the server and on receiving the string on the client and 
> they are exactly the same, yet the values are all -1.
>
> I've attached a snippet of the proto file below. 
>
> At this point these are below combos that always fail
>
> type=99, count= 0, index = 1 (all values are -1 after deserialization)
> type=99, count=99, index= 1 (works)
> type=9, count=99, index= 1 (works)
>
> - Basically anytime count is 0, all the values seem to be -1. However 
> if I make type 0, the same behavior does not occur. Am I missing 
> something 
> or there is some underlying bug in my code? Again I've checked multiple 
> times the correct raw string data is being sent (with correct size). I 
> use 
> the 

Re: [protobuf] Protobuf 2.6 serialization weird behavior

['Adam Cozzette' via Protocol Buffers]

I think Marc is right that it would be best to isolate the problem by
manually checking that the bytes are valid.

Tony, how are you framing the messages? Protocol buffers do not describe
their own size and so you have to know what size to expect before you parse
a proto. The typical approach is to frame the serialized messages by
prefixing them with their size. It's easy to get this wrong, though, so
that could potentially be the reason that the messages aren't showing up as
you expect.

On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell  wrote:

> Well, if it was me: the first thing I'd do is isolate whether it is
> serialize or deserialize that is failing - by taking your currently
> serialized data as a flat file or byte dump, and checking whether the
> values are right or wrong.
>
> Depending on the volume and layout of the data, you might be able to use
> protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
> welcome to try using https://protogen.marcgravell.com/decode
>
> That would allow you to focus your checks.
>
> Marc
>
> On 10 April 2018 at 14:20, Tony Tony  wrote:
>
>> c++/windows
>>
>> I'll upgrade protobuf and see if issue goes away. Just wanted to make
>> sure I'm not missing anything. Is there any additional troubleshooting I
>> can look into to troubleshoot further if the recent release reproduce the
>> issue?
>>
>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>>
>>> First thought: what language / platform is this? the C++ generated code
>>> is very different to the Java generated code, for example
>>>
>>> Second thought: 2.6 is pretty old; it is very possible that a bug
>>> existed and has been fixed since then (Aug 2014) - does it still happen
>>> with more recent releases?
>>>
>>> The scenario you present seems fine and reasonable.
>>>
>>> Marc
>>>
>>> On 10 April 2018 at 03:46, Tony Tony  wrote:
>>>
 Hi,

 I have a strange case I'm dealing with. Using protobuf compiler 2.6.

 Every time I send a certain set of values over my socket, it ends up
 with missing data. Although I've compared the string received after
 serialization on the server and on receiving the string on the client and
 they are exactly the same, yet the values are all -1.

 I've attached a snippet of the proto file below.

 At this point these are below combos that always fail

 type=99, count= 0, index = 1 (all values are -1 after deserialization)
 type=99, count=99, index= 1 (works)
 type=9, count=99, index= 1 (works)

 - Basically anytime count is 0, all the values seem to be -1. However
 if I make type 0, the same behavior does not occur. Am I missing something
 or there is some underlying bug in my code? Again I've checked multiple
 times the correct raw string data is being sent (with correct size). I use
 the parseFromString method to parse the receiving string data.

 - I've sent plenty of other types of data just fine.

 - There are other messages defined in the same protobuf packet
 definition file, can that make a difference?

 message packet {
 required int32 type = 1 [default = -1];
 required int32 count = 2 [default = -1];
 required int32 index = 3 [default = -1];
 }

 Thanks

 --
 You received this message because you are subscribed to the Google
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at https://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Marc
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Regards,
>
> Marc
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Well, if it was me: the first thing I'd do is isolate whether it is
serialize or deserialize that is failing - by taking your currently
serialized data as a flat file or byte dump, and checking whether the
values are right or wrong.

Depending on the volume and layout of the data, you might be able to use
protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
welcome to try using https://protogen.marcgravell.com/decode

That would allow you to focus your checks.

Marc

On 10 April 2018 at 14:20, Tony Tony  wrote:

> c++/windows
>
> I'll upgrade protobuf and see if issue goes away. Just wanted to make sure
> I'm not missing anything. Is there any additional troubleshooting I can
> look into to troubleshoot further if the recent release reproduce the issue?
>
> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>
>> First thought: what language / platform is this? the C++ generated code
>> is very different to the Java generated code, for example
>>
>> Second thought: 2.6 is pretty old; it is very possible that a bug existed
>> and has been fixed since then (Aug 2014) - does it still happen with more
>> recent releases?
>>
>> The scenario you present seems fine and reasonable.
>>
>> Marc
>>
>> On 10 April 2018 at 03:46, Tony Tony  wrote:
>>
>>> Hi,
>>>
>>> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>>>
>>> Every time I send a certain set of values over my socket, it ends up
>>> with missing data. Although I've compared the string received after
>>> serialization on the server and on receiving the string on the client and
>>> they are exactly the same, yet the values are all -1.
>>>
>>> I've attached a snippet of the proto file below.
>>>
>>> At this point these are below combos that always fail
>>>
>>> type=99, count= 0, index = 1 (all values are -1 after deserialization)
>>> type=99, count=99, index= 1 (works)
>>> type=9, count=99, index= 1 (works)
>>>
>>> - Basically anytime count is 0, all the values seem to be -1. However if
>>> I make type 0, the same behavior does not occur. Am I missing something or
>>> there is some underlying bug in my code? Again I've checked multiple times
>>> the correct raw string data is being sent (with correct size). I use the
>>> parseFromString method to parse the receiving string data.
>>>
>>> - I've sent plenty of other types of data just fine.
>>>
>>> - There are other messages defined in the same protobuf packet
>>> definition file, can that make a difference?
>>>
>>> message packet {
>>> required int32 type = 1 [default = -1];
>>> required int32 count = 2 [default = -1];
>>> required int32 index = 3 [default = -1];
>>> }
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Marc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Tony Tony]

c++/windows

I'll upgrade protobuf and see if issue goes away. Just wanted to make sure 
I'm not missing anything. Is there any additional troubleshooting I can 
look into to troubleshoot further if the recent release reproduce the issue?

On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>
> First thought: what language / platform is this? the C++ generated code is 
> very different to the Java generated code, for example
>
> Second thought: 2.6 is pretty old; it is very possible that a bug existed 
> and has been fixed since then (Aug 2014) - does it still happen with more 
> recent releases?
>
> The scenario you present seems fine and reasonable.
>
> Marc
>
> On 10 April 2018 at 03:46, Tony Tony  
> wrote:
>
>> Hi,
>>
>> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>>
>> Every time I send a certain set of values over my socket, it ends up with 
>> missing data. Although I've compared the string received after 
>> serialization on the server and on receiving the string on the client and 
>> they are exactly the same, yet the values are all -1.
>>
>> I've attached a snippet of the proto file below. 
>>
>> At this point these are below combos that always fail
>>
>> type=99, count= 0, index = 1 (all values are -1 after deserialization)
>> type=99, count=99, index= 1 (works)
>> type=9, count=99, index= 1 (works)
>>
>> - Basically anytime count is 0, all the values seem to be -1. However if 
>> I make type 0, the same behavior does not occur. Am I missing something or 
>> there is some underlying bug in my code? Again I've checked multiple times 
>> the correct raw string data is being sent (with correct size). I use the 
>> parseFromString method to parse the receiving string data.
>>
>> - I've sent plenty of other types of data just fine.
>>
>> - There are other messages defined in the same protobuf packet definition 
>> file, can that make a difference?
>>
>> message packet {
>> required int32 type = 1 [default = -1];
>> required int32 count = 2 [default = -1];
>> required int32 index = 3 [default = -1];
>> }
>>
>> Thanks
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Regards, 
>
> Marc
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

First thought: what language / platform is this? the C++ generated code is
very different to the Java generated code, for example

Second thought: 2.6 is pretty old; it is very possible that a bug existed
and has been fixed since then (Aug 2014) - does it still happen with more
recent releases?

The scenario you present seems fine and reasonable.

Marc

On 10 April 2018 at 03:46, Tony Tony  wrote:

> Hi,
>
> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>
> Every time I send a certain set of values over my socket, it ends up with
> missing data. Although I've compared the string received after
> serialization on the server and on receiving the string on the client and
> they are exactly the same, yet the values are all -1.
>
> I've attached a snippet of the proto file below.
>
> At this point these are below combos that always fail
>
> type=99, count= 0, index = 1 (all values are -1 after deserialization)
> type=99, count=99, index= 1 (works)
> type=9, count=99, index= 1 (works)
>
> - Basically anytime count is 0, all the values seem to be -1. However if I
> make type 0, the same behavior does not occur. Am I missing something or
> there is some underlying bug in my code? Again I've checked multiple times
> the correct raw string data is being sent (with correct size). I use the
> parseFromString method to parse the receiving string data.
>
> - I've sent plenty of other types of data just fine.
>
> - There are other messages defined in the same protobuf packet definition
> file, can that make a difference?
>
> message packet {
> required int32 type = 1 [default = -1];
> required int32 count = 2 [default = -1];
> required int32 index = 3 [default = -1];
> }
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Protobuf 2.6 serialization weird behavior

[Tony Tony]

Hi,

I have a strange case I'm dealing with. Using protobuf compiler 2.6.

Every time I send a certain set of values over my socket, it ends up with 
missing data. Although I've compared the string received after 
serialization on the server and on receiving the string on the client and 
they are exactly the same, yet the values are all -1.

I've attached a snippet of the proto file below. 

At this point these are below combos that always fail

type=99, count= 0, index = 1 (all values are -1 after deserialization)
type=99, count=99, index= 1 (works)
type=9, count=99, index= 1 (works)

- Basically anytime count is 0, all the values seem to be -1. However if I 
make type 0, the same behavior does not occur. Am I missing something or 
there is some underlying bug in my code? Again I've checked multiple times 
the correct raw string data is being sent (with correct size). I use the 
parseFromString method to parse the receiving string data.

- I've sent plenty of other types of data just fine.

- There are other messages defined in the same protobuf packet definition 
file, can that make a difference?

message packet {
required int32 type = 1 [default = -1];
required int32 count = 2 [default = -1];
required int32 index = 3 [default = -1];
}

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.