Re: [protobuf] Re: Re-Parsing Received CodedMessages Over Network (Proto3)

[Chazix]

Thank you for the suggestion to nix the +1 Adam, I should have heeded this 
advice sooner - I just encountered the problem with a larger than 127 bytes 
message not being re-parsed correctly once the server received the message 
from my client.

On Monday, August 1, 2016 at 7:20:49 AM UTC-7, Adam Cozzette wrote:
>
> Great, I'm glad that worked. In your C# client I think what's going on is 
> that you're calculating the size of the message but excluding the length 
> prefix that comes before it, so the +1 is correcting for that. In general 
> adding 1 will not work reliably, though, since the length prefix is 
> varint-encoded and therefore has a variable-length encoding that will use 
> more than one byte for messages 128 bytes and longer. I'm not familiar with 
> C# at all but is there perhaps a way to just query the MemoryStream to ask 
> how many bytes it has written? I see that there's a Position 
> 
>  
> property that might do the trick.
>
> On Sat, Jul 30, 2016 at 5:33 PM, Chazix > 
> wrote:
>
>> Thanks for the insight, that definitely does work for collecting my 
>> message, and it seems to be more so a standard compared to what I was 
>> doing, at least for my server side.
>>
>> On my C# client side, I still have to add + 1 to the 
>> IMessage.CalculateSize() function when sending over my UdpClient object 
>> unfortunately.
>>
>> public void DataToSend(IMessage message)
>>> {
>>>   CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream, 
>>> true);
>>>   output.WriteMessage(message);
>>>   output.Flush();
>>>   (m_ostream.BaseStream as MemoryStream).SetLength(0); // reset stream 
>>> for next packet(s)
>>>   m_client.GetServerConnection().GetClient().Send(m_packet, 
>>> message.CalculateSize() + 1);
>>> }
>>>
>>
>> On Friday, July 29, 2016 at 4:18:29 PM UTC-7, Adam Cozzette wrote:
>>>
>>> I believe the problem you're running into is on the server side; it 
>>> looks like you're parsing the size correctly but just skipping over it. 
>>> Since protocol buffers are not self-delimiting, you must actually rely on 
>>> the size to know how many bytes to read. I think what's probably happening 
>>> is that the CodedInputStream is reading past the end of your message and 
>>> trying to interpret whatever the bytes are that happen to come after it.
>>> The way to solve this is to use CodedInputStream's PushLimit and 
>>> PopLimit methods. In your example something like this should work:
>>>
>>> google::protobuf::io::CodedInputStream::Limit limit = 
>>> inputStream.PushLimit(msgSize);
>>> if (!msg.ParseFromCodedStream(&inputStream) || 
>>> !inputStream.ConsumedEntireMessage()) {
>>>   // handle error ...
>>> }
>>> inputStream.PopLimit(limit);
>>>
>>> Let me know if this fixes it.
>>>
>>> On Sat, Jul 23, 2016 at 3:51 PM, Chazix  wrote:
>>>
 I'm continuing to fiddle with this problem. For some reason it seems 
 that I'm missing a byte count when sending from my client.

 m_client.GetServerConnection.GetClient().Send(m_packet, 
 message.CalculateSize() + 1) // I needed to add + 1

 I thought it was strange that my message was only being received as: 
 #\u0012!\n\u001Ftype.googleapis.com/server.Pin

 I'm not sure why I have to add + 1 tho and CalculateSize isn't giving 
 the correct size naturally.


 On Saturday, July 23, 2016 at 2:56:08 PM UTC-7, Chazix wrote:
>
> I'm utilizing Protobuf3 with c# for my client & c++ for my server, 
> where the .proto's are generated from the corresponding protoc3 compiler. 
> I'm new to utilizing Google Protocol Buffers, where I'm currently trying 
> to 
> figure out how to re-parse the received bytes on my server that were sent 
> from my client to turn it back into it's originating 
> google::protobuf::Message object on my c++ server.
>
> My client is sending a CodedOutputStream in c# :
>
> public PacketHandler()
>> {
>>   m_client  = GetComponent();
>>   m_packet  = new 
>> byte[m_client.GetServerConnection().GetMaxPacketSize()];
>>   m_ostream = new BinaryWriter(new MemoryStream(m_packet));
>> }
>>
>> public void DataToSend(IMessage message)
>> {
>>   // message looks like :
>>   // Base.BaseMessage msg = new Base.BaseMessage();
>>   // msg.Msg = Google.Protobuf.WellKnownTypes.Any.Pack(new 
>> Server.Ping());
>>   CodedOutputStream output = new 
>> CodedOutputStream(m_ostream.BaseStream, true);
>>   output.WriteMessage(message);
>>   output.Flush();
>>   m_client.GetServerConnection().GetClient().Send(m_packet, 
>> message.CalculateSize());
>> }
>>
>
> This seems to be working, the message that is sent right now is a 
> simple Ping message that looks like this:
>
> // Ping.proto
>> syntax = "

Re: [protobuf] Re: Re-Parsing Received CodedMessages Over Network (Proto3)

['Adam Cozzette' via Protocol Buffers]

Great, I'm glad that worked. In your C# client I think what's going on is
that you're calculating the size of the message but excluding the length
prefix that comes before it, so the +1 is correcting for that. In general
adding 1 will not work reliably, though, since the length prefix is
varint-encoded and therefore has a variable-length encoding that will use
more than one byte for messages 128 bytes and longer. I'm not familiar with
C# at all but is there perhaps a way to just query the MemoryStream to ask
how many bytes it has written? I see that there's a Position

property that might do the trick.

On Sat, Jul 30, 2016 at 5:33 PM, Chazix  wrote:

> Thanks for the insight, that definitely does work for collecting my
> message, and it seems to be more so a standard compared to what I was
> doing, at least for my server side.
>
> On my C# client side, I still have to add + 1 to the
> IMessage.CalculateSize() function when sending over my UdpClient object
> unfortunately.
>
> public void DataToSend(IMessage message)
>> {
>>   CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream,
>> true);
>>   output.WriteMessage(message);
>>   output.Flush();
>>   (m_ostream.BaseStream as MemoryStream).SetLength(0); // reset stream
>> for next packet(s)
>>   m_client.GetServerConnection().GetClient().Send(m_packet,
>> message.CalculateSize() + 1);
>> }
>>
>
> On Friday, July 29, 2016 at 4:18:29 PM UTC-7, Adam Cozzette wrote:
>>
>> I believe the problem you're running into is on the server side; it looks
>> like you're parsing the size correctly but just skipping over it. Since
>> protocol buffers are not self-delimiting, you must actually rely on the
>> size to know how many bytes to read. I think what's probably happening is
>> that the CodedInputStream is reading past the end of your message and
>> trying to interpret whatever the bytes are that happen to come after it.
>> The way to solve this is to use CodedInputStream's PushLimit and PopLimit
>> methods. In your example something like this should work:
>>
>> google::protobuf::io::CodedInputStream::Limit limit =
>> inputStream.PushLimit(msgSize);
>> if (!msg.ParseFromCodedStream(&inputStream) ||
>> !inputStream.ConsumedEntireMessage()) {
>>   // handle error ...
>> }
>> inputStream.PopLimit(limit);
>>
>> Let me know if this fixes it.
>>
>> On Sat, Jul 23, 2016 at 3:51 PM, Chazix  wrote:
>>
>>> I'm continuing to fiddle with this problem. For some reason it seems
>>> that I'm missing a byte count when sending from my client.
>>>
>>> m_client.GetServerConnection.GetClient().Send(m_packet,
>>> message.CalculateSize() + 1) // I needed to add + 1
>>>
>>> I thought it was strange that my message was only being received as:
>>> #\u0012!\n\u001Ftype.googleapis.com/server.Pin
>>>
>>> I'm not sure why I have to add + 1 tho and CalculateSize isn't giving
>>> the correct size naturally.
>>>
>>>
>>> On Saturday, July 23, 2016 at 2:56:08 PM UTC-7, Chazix wrote:

 I'm utilizing Protobuf3 with c# for my client & c++ for my server,
 where the .proto's are generated from the corresponding protoc3 compiler.
 I'm new to utilizing Google Protocol Buffers, where I'm currently trying to
 figure out how to re-parse the received bytes on my server that were sent
 from my client to turn it back into it's originating
 google::protobuf::Message object on my c++ server.

 My client is sending a CodedOutputStream in c# :

 public PacketHandler()
> {
>   m_client  = GetComponent();
>   m_packet  = new
> byte[m_client.GetServerConnection().GetMaxPacketSize()];
>   m_ostream = new BinaryWriter(new MemoryStream(m_packet));
> }
>
> public void DataToSend(IMessage message)
> {
>   // message looks like :
>   // Base.BaseMessage msg = new Base.BaseMessage();
>   // msg.Msg = Google.Protobuf.WellKnownTypes.Any.Pack(new
> Server.Ping());
>   CodedOutputStream output = new
> CodedOutputStream(m_ostream.BaseStream, true);
>   output.WriteMessage(message);
>   output.Flush();
>   m_client.GetServerConnection().GetClient().Send(m_packet,
> message.CalculateSize());
> }
>

 This seems to be working, the message that is sent right now is a
 simple Ping message that looks like this:

 // Ping.proto
> syntax = "proto3";
> package server;
>
> import "BaseMessage.proto";
>
> message Ping {
>
> }
>
> message Pong {
>
> }
>

 My BaseMessage looks like this :

 // BaseMessage.proto
> syntax = "proto3";
>
> package base;
>
> message BaseMessage {
>   uint32 size = 1;
>   google.protobuf.Any msg = 2;
> }
>

 The received message that I am g

Re: [protobuf] Re: Re-Parsing Received CodedMessages Over Network (Proto3)

[Chazix]

Thanks for the insight, that definitely does work for collecting my 
message, and it seems to be more so a standard compared to what I was 
doing, at least for my server side.

On my C# client side, I still have to add + 1 to the 
IMessage.CalculateSize() function when sending over my UdpClient object 
unfortunately.

public void DataToSend(IMessage message)
> {
>   CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream, 
> true);
>   output.WriteMessage(message);
>   output.Flush();
>   (m_ostream.BaseStream as MemoryStream).SetLength(0); // reset stream for 
> next packet(s)
>   m_client.GetServerConnection().GetClient().Send(m_packet, 
> message.CalculateSize() + 1);
> }
>

On Friday, July 29, 2016 at 4:18:29 PM UTC-7, Adam Cozzette wrote:
>
> I believe the problem you're running into is on the server side; it looks 
> like you're parsing the size correctly but just skipping over it. Since 
> protocol buffers are not self-delimiting, you must actually rely on the 
> size to know how many bytes to read. I think what's probably happening is 
> that the CodedInputStream is reading past the end of your message and 
> trying to interpret whatever the bytes are that happen to come after it.
> The way to solve this is to use CodedInputStream's PushLimit and PopLimit 
> methods. In your example something like this should work:
>
> google::protobuf::io::CodedInputStream::Limit limit = 
> inputStream.PushLimit(msgSize);
> if (!msg.ParseFromCodedStream(&inputStream) || 
> !inputStream.ConsumedEntireMessage()) {
>   // handle error ...
> }
> inputStream.PopLimit(limit);
>
> Let me know if this fixes it.
>
> On Sat, Jul 23, 2016 at 3:51 PM, Chazix > 
> wrote:
>
>> I'm continuing to fiddle with this problem. For some reason it seems that 
>> I'm missing a byte count when sending from my client.
>>
>> m_client.GetServerConnection.GetClient().Send(m_packet, 
>> message.CalculateSize() + 1) // I needed to add + 1
>>
>> I thought it was strange that my message was only being received as: 
>> #\u0012!\n\u001Ftype.googleapis.com/server.Pin
>>
>> I'm not sure why I have to add + 1 tho and CalculateSize isn't giving the 
>> correct size naturally.
>>
>>
>> On Saturday, July 23, 2016 at 2:56:08 PM UTC-7, Chazix wrote:
>>>
>>> I'm utilizing Protobuf3 with c# for my client & c++ for my server, where 
>>> the .proto's are generated from the corresponding protoc3 compiler. I'm new 
>>> to utilizing Google Protocol Buffers, where I'm currently trying to figure 
>>> out how to re-parse the received bytes on my server that were sent from my 
>>> client to turn it back into it's originating google::protobuf::Message 
>>> object on my c++ server.
>>>
>>> My client is sending a CodedOutputStream in c# :
>>>
>>> public PacketHandler()
 {
   m_client  = GetComponent();
   m_packet  = new 
 byte[m_client.GetServerConnection().GetMaxPacketSize()];
   m_ostream = new BinaryWriter(new MemoryStream(m_packet));
 }

 public void DataToSend(IMessage message)
 {
   // message looks like :
   // Base.BaseMessage msg = new Base.BaseMessage();
   // msg.Msg = Google.Protobuf.WellKnownTypes.Any.Pack(new 
 Server.Ping());
   CodedOutputStream output = new 
 CodedOutputStream(m_ostream.BaseStream, true);
   output.WriteMessage(message);
   output.Flush();
   m_client.GetServerConnection().GetClient().Send(m_packet, 
 message.CalculateSize());
 }

>>>
>>> This seems to be working, the message that is sent right now is a simple 
>>> Ping message that looks like this:
>>>
>>> // Ping.proto
 syntax = "proto3";
 package server;

 import "BaseMessage.proto";
 
 message Ping {
   
 }
 
 message Pong {
   
 }

>>>
>>> My BaseMessage looks like this :
>>>
>>> // BaseMessage.proto
 syntax = "proto3";
 
 package base;
 
 message BaseMessage {
   uint32 size = 1;
   google.protobuf.Any msg = 2;
 }

>>>
>>> The received message that I am getting on my c++ server side looks like 
>>> this : #\u0012!\n\u001Ftype.googleapis.com/server.Pin
>>>
>>> When receiving the message I am attempting to re-parse using the 
>>> CodedInputStream object by attempting to parse the received bytes.
>>>
>>> PacketHandler::PacketHandler(QByteArray& packet, const 
 Manager::ClientPtr client) :
 m_packet(packet),
 m_client(client)
 {
   unsigned char buffer[512] = { 0 };
   unsigned char data[packet.size()] = { 0 };
   memcpy(data, packet.data(), packet.size());
 
   google::protobuf::uint32 msgSize;
   google::protobuf::io::CodedInputStream inputStream(data, 
 packet.size());
   inputStream.ReadVarint32(&msgSize); // read the prefixed message 
 length

>

Re: [protobuf] Re: Re-Parsing Received CodedMessages Over Network (Proto3)

['Adam Cozzette' via Protocol Buffers]

I believe the problem you're running into is on the server side; it looks
like you're parsing the size correctly but just skipping over it. Since
protocol buffers are not self-delimiting, you must actually rely on the
size to know how many bytes to read. I think what's probably happening is
that the CodedInputStream is reading past the end of your message and
trying to interpret whatever the bytes are that happen to come after it.
The way to solve this is to use CodedInputStream's PushLimit and PopLimit
methods. In your example something like this should work:

google::protobuf::io::CodedInputStream::Limit limit =
inputStream.PushLimit(msgSize);
if (!msg.ParseFromCodedStream(&inputStream) ||
!inputStream.ConsumedEntireMessage()) {
  // handle error ...
}
inputStream.PopLimit(limit);

Let me know if this fixes it.

On Sat, Jul 23, 2016 at 3:51 PM, Chazix  wrote:

> I'm continuing to fiddle with this problem. For some reason it seems that
> I'm missing a byte count when sending from my client.
>
> m_client.GetServerConnection.GetClient().Send(m_packet,
> message.CalculateSize() + 1) // I needed to add + 1
>
> I thought it was strange that my message was only being received as:
> #\u0012!\n\u001Ftype.googleapis.com/server.Pin
>
> I'm not sure why I have to add + 1 tho and CalculateSize isn't giving the
> correct size naturally.
>
>
> On Saturday, July 23, 2016 at 2:56:08 PM UTC-7, Chazix wrote:
>>
>> I'm utilizing Protobuf3 with c# for my client & c++ for my server, where
>> the .proto's are generated from the corresponding protoc3 compiler. I'm new
>> to utilizing Google Protocol Buffers, where I'm currently trying to figure
>> out how to re-parse the received bytes on my server that were sent from my
>> client to turn it back into it's originating google::protobuf::Message
>> object on my c++ server.
>>
>> My client is sending a CodedOutputStream in c# :
>>
>> public PacketHandler()
>>> {
>>>   m_client  = GetComponent();
>>>   m_packet  = new
>>> byte[m_client.GetServerConnection().GetMaxPacketSize()];
>>>   m_ostream = new BinaryWriter(new MemoryStream(m_packet));
>>> }
>>>
>>> public void DataToSend(IMessage message)
>>> {
>>>   // message looks like :
>>>   // Base.BaseMessage msg = new Base.BaseMessage();
>>>   // msg.Msg = Google.Protobuf.WellKnownTypes.Any.Pack(new
>>> Server.Ping());
>>>   CodedOutputStream output = new
>>> CodedOutputStream(m_ostream.BaseStream, true);
>>>   output.WriteMessage(message);
>>>   output.Flush();
>>>   m_client.GetServerConnection().GetClient().Send(m_packet,
>>> message.CalculateSize());
>>> }
>>>
>>
>> This seems to be working, the message that is sent right now is a simple
>> Ping message that looks like this:
>>
>> // Ping.proto
>>> syntax = "proto3";
>>> package server;
>>>
>>> import "BaseMessage.proto";
>>>
>>> message Ping {
>>>
>>> }
>>>
>>> message Pong {
>>>
>>> }
>>>
>>
>> My BaseMessage looks like this :
>>
>> // BaseMessage.proto
>>> syntax = "proto3";
>>>
>>> package base;
>>>
>>> message BaseMessage {
>>>   uint32 size = 1;
>>>   google.protobuf.Any msg = 2;
>>> }
>>>
>>
>> The received message that I am getting on my c++ server side looks like
>> this : #\u0012!\n\u001Ftype.googleapis.com/server.Pin
>>
>> When receiving the message I am attempting to re-parse using the
>> CodedInputStream object by attempting to parse the received bytes.
>>
>> PacketHandler::PacketHandler(QByteArray& packet, const
>>> Manager::ClientPtr client) :
>>> m_packet(packet),
>>> m_client(client)
>>> {
>>>   unsigned char buffer[512] = { 0 };
>>>   unsigned char data[packet.size()] = { 0 };
>>>   memcpy(data, packet.data(), packet.size());
>>>
>>>   google::protobuf::uint32 msgSize;
>>>   google::protobuf::io::CodedInputStream inputStream(data,
>>> packet.size());
>>>   inputStream.ReadVarint32(&msgSize); // read the prefixed message
>>> length
>>>
>>>   base::BaseMessage msg;
>>>   msg.ParseFromCodedStream(&inputStream);
>>>
>>>   if (msg.msg().Is())
>>>   {
>>> // This never occurs
>>> server::Ping pingMsg;
>>> msg.msg().UnpackTo(&pingMsg);
>>> qDebug() << "Is a Ping";
>>>   }
>>>   qDebug() << msg.size(); // this isn't the correct value either
>>> }
>>>
>>
>> This is where I am a bit unsure of the process that is needing to be done
>> to re-parse the message into the particular message. I'm reading the
>> initial prefixed length of message to remove that from the parse, where the
>> inputStream has the remaining bytes for the message identification.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com