Thank you for the suggestion to nix the +1 Adam, I should have heeded this advice sooner - I just encountered the problem with a larger than 127 bytes message not being re-parsed correctly once the server received the message from my client.
On Monday, August 1, 2016 at 7:20:49 AM UTC-7, Adam Cozzette wrote: > > Great, I'm glad that worked. In your C# client I think what's going on is > that you're calculating the size of the message but excluding the length > prefix that comes before it, so the +1 is correcting for that. In general > adding 1 will not work reliably, though, since the length prefix is > varint-encoded and therefore has a variable-length encoding that will use > more than one byte for messages 128 bytes and longer. I'm not familiar with > C# at all but is there perhaps a way to just query the MemoryStream to ask > how many bytes it has written? I see that there's a Position > <https://msdn.microsoft.com/en-us/library/system.io.memorystream.position(v=vs.110).aspx> > > property that might do the trick. > > On Sat, Jul 30, 2016 at 5:33 PM, Chazix <[email protected] <javascript:>> > wrote: > >> Thanks for the insight, that definitely does work for collecting my >> message, and it seems to be more so a standard compared to what I was >> doing, at least for my server side. >> >> On my C# client side, I still have to add + 1 to the >> IMessage.CalculateSize() function when sending over my UdpClient object >> unfortunately. >> >> public void DataToSend(IMessage message) >>> { >>> CodedOutputStream output = new CodedOutputStream(m_ostream.BaseStream, >>> true); >>> output.WriteMessage(message); >>> output.Flush(); >>> (m_ostream.BaseStream as MemoryStream).SetLength(0); // reset stream >>> for next packet(s) >>> m_client.GetServerConnection().GetClient().Send(m_packet, >>> message.CalculateSize() + 1); >>> } >>> >> >> On Friday, July 29, 2016 at 4:18:29 PM UTC-7, Adam Cozzette wrote: >>> >>> I believe the problem you're running into is on the server side; it >>> looks like you're parsing the size correctly but just skipping over it. >>> Since protocol buffers are not self-delimiting, you must actually rely on >>> the size to know how many bytes to read. I think what's probably happening >>> is that the CodedInputStream is reading past the end of your message and >>> trying to interpret whatever the bytes are that happen to come after it. >>> The way to solve this is to use CodedInputStream's PushLimit and >>> PopLimit methods. In your example something like this should work: >>> >>> google::protobuf::io::CodedInputStream::Limit limit = >>> inputStream.PushLimit(msgSize); >>> if (!msg.ParseFromCodedStream(&inputStream) || >>> !inputStream.ConsumedEntireMessage()) { >>> // handle error ... >>> } >>> inputStream.PopLimit(limit); >>> >>> Let me know if this fixes it. >>> >>> On Sat, Jul 23, 2016 at 3:51 PM, Chazix <[email protected]> wrote: >>> >>>> I'm continuing to fiddle with this problem. For some reason it seems >>>> that I'm missing a byte count when sending from my client. >>>> >>>> m_client.GetServerConnection.GetClient().Send(m_packet, >>>> message.CalculateSize() + 1) // I needed to add + 1 >>>> >>>> I thought it was strange that my message was only being received as: >>>> #\u0012!\n\u001Ftype.googleapis.com/server.Pin >>>> >>>> I'm not sure why I have to add + 1 tho and CalculateSize isn't giving >>>> the correct size naturally. >>>> >>>> >>>> On Saturday, July 23, 2016 at 2:56:08 PM UTC-7, Chazix wrote: >>>>> >>>>> I'm utilizing Protobuf3 with c# for my client & c++ for my server, >>>>> where the .proto's are generated from the corresponding protoc3 compiler. >>>>> I'm new to utilizing Google Protocol Buffers, where I'm currently trying >>>>> to >>>>> figure out how to re-parse the received bytes on my server that were sent >>>>> from my client to turn it back into it's originating >>>>> google::protobuf::Message object on my c++ server. >>>>> >>>>> My client is sending a CodedOutputStream in c# : >>>>> >>>>> public PacketHandler() >>>>>> { >>>>>> m_client = GetComponent<ClientObject>(); >>>>>> m_packet = new >>>>>> byte[m_client.GetServerConnection().GetMaxPacketSize()]; >>>>>> m_ostream = new BinaryWriter(new MemoryStream(m_packet)); >>>>>> } >>>>>> >>>>>> public void DataToSend(IMessage message) >>>>>> { >>>>>> // message looks like : >>>>>> // Base.BaseMessage msg = new Base.BaseMessage(); >>>>>> // msg.Msg = Google.Protobuf.WellKnownTypes.Any.Pack(new >>>>>> Server.Ping()); >>>>>> CodedOutputStream output = new >>>>>> CodedOutputStream(m_ostream.BaseStream, true); >>>>>> output.WriteMessage(message); >>>>>> output.Flush(); >>>>>> m_client.GetServerConnection().GetClient().Send(m_packet, >>>>>> message.CalculateSize()); >>>>>> } >>>>>> >>>>> >>>>> This seems to be working, the message that is sent right now is a >>>>> simple Ping message that looks like this: >>>>> >>>>> // Ping.proto >>>>>> syntax = "proto3"; >>>>>> package server; >>>>>> >>>>>> import "BaseMessage.proto"; >>>>>> >>>>>> message Ping { >>>>>> >>>>>> } >>>>>> >>>>>> message Pong { >>>>>> >>>>>> } >>>>>> >>>>> >>>>> My BaseMessage looks like this : >>>>> >>>>> // BaseMessage.proto >>>>>> syntax = "proto3"; >>>>>> >>>>>> package base; >>>>>> >>>>>> message BaseMessage { >>>>>> uint32 size = 1; >>>>>> google.protobuf.Any msg = 2; >>>>>> } >>>>>> >>>>> >>>>> The received message that I am getting on my c++ server side looks >>>>> like this : #\u0012!\n\u001Ftype.googleapis.com/server.Pin >>>>> >>>>> When receiving the message I am attempting to re-parse using the >>>>> CodedInputStream object by attempting to parse the received bytes. >>>>> >>>>> PacketHandler::PacketHandler(QByteArray& packet, const >>>>>> Manager::ClientPtr client) : >>>>>> m_packet(packet), >>>>>> m_client(client) >>>>>> { >>>>>> unsigned char buffer[512] = { 0 }; >>>>>> unsigned char data[packet.size()] = { 0 }; >>>>>> memcpy(data, packet.data(), packet.size()); >>>>>> >>>>>> google::protobuf::uint32 msgSize; >>>>>> google::protobuf::io::CodedInputStream inputStream(data, >>>>>> packet.size()); >>>>>> inputStream.ReadVarint32(&msgSize); // read the prefixed >>>>>> message length >>>>>> >>>>>> base::BaseMessage msg; >>>>>> msg.ParseFromCodedStream(&inputStream); >>>>>> >>>>>> if (msg.msg().Is<server::Ping>()) >>>>>> { >>>>>> // This never occurs >>>>>> server::Ping pingMsg; >>>>>> msg.msg().UnpackTo(&pingMsg); >>>>>> qDebug() << "Is a Ping"; >>>>>> } >>>>>> qDebug() << msg.size(); // this isn't the correct value either >>>>>> } >>>>>> >>>>> >>>>> This is where I am a bit unsure of the process that is needing to be >>>>> done to re-parse the message into the particular message. I'm reading the >>>>> initial prefixed length of message to remove that from the parse, where >>>>> the >>>>> inputStream has the remaining bytes for the message identification. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/protobuf. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
