Could you post some code showing the problem? I still suspect that it's probably an issue involving how the messages are framed, or maybe some kind of buffering issue. You could also try serializing the message to a string (e.g. with message.SerializeToString(&s)) and see if you can successfully parse the message again. That could help narrow down whether it's a problem with protobuf serialization or something else.
On Thu, Feb 8, 2018 at 9:29 PM, hce h <[email protected]> wrote: > Hi Adam, I built from the git source, but that did not work. After several > hours debugging, I found the problem but not quite sure the which library > causes the problem: > > I sent two protobuf messages consecutively in server side within the same > thread, it uses boost::asio::streambuf and std::ostream to call > message.SerializeToOstream(&os), then to call boost::asio::async_write to > send the message. For each sendMessage, a new shared pointer streambuf is > used, it should not cause buffer overlap. It works file when the message > smaller then 17 KB, when the message is larger, it corrupted the serialized > message, I suspect it is the protobuf serialization issue rather than boost > asio asyc write issue, but I don't know it enough, you might give me some > insight... > > > > On Wednesday, February 7, 2018 at 12:10:30 PM UTC+11, hce h wrote: >> >> Thanks Adam, good to know I don't need to change my protocol to set >> syntax = "proto2", I am building the code right now, will let you know if >> the new library fix the issue or not. >> >> >> >> On Wednesday, February 7, 2018 at 11:51:02 AM UTC+11, Adam Cozzette wrote: >>> >>> You can still upgrade to our most recent code on GitHub. As long as you >>> don't set syntax = "proto3" in your .proto files, your code should work >>> fine without needing any modifications. >>> >>> On Tue, Feb 6, 2018 at 4:25 PM, hce h <[email protected]> wrote: >>> >>>> I am going to try to build protobuf from git source for armfs to verify >>>> if the old version protobuf caused the issue or not, the trouble is my >>>> protobuf code as all defined for protobuf v2, I don't want to run into the >>>> trouble with protobuf v3 source code, anyway I can defined to build v2 only >>>> during my build process? >>>> >>>> Thank you >>>> >>>> >>>> On Wednesday, February 7, 2018 at 10:44:14 AM UTC+11, hce h wrote: >>>>> >>>>> Thanks Adam, I had further investigation, the original post of error >>>>> was not correct, it was not the problems of length, the file length was >>>>> correct, but the file was corrupted, there were some junk characters like >>>>> "^@^@^@^L^B^@^B^P^D^H<80><9b>^F^P]^X<80><9b>^F" inserted to the >>>>> middle of protobuf bytes / string and chopped off the connect string after >>>>> serialisation. I don't think there are any issues of coding and >>>>> serialisation / deserialisation in general, I have used the same code for >>>>> years without any issues until when the files increased to 17 KB, it >>>>> corrupted. >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wednesday, February 7, 2018 at 4:13:22 AM UTC+11, Adam Cozzette >>>>> wrote: >>>>>> >>>>>> Could you post some example code showing the problem? I would guess >>>>>> that it's an issue with how your code is framing the message. Serialized >>>>>> protos do not indicate their own length, so when you parse a message you >>>>>> have to know how many bytes to expect. If you try to parse the wrong >>>>>> number >>>>>> of bytes, you can inadvertently parse the wrong thing or parsing might >>>>>> fail >>>>>> entirely. >>>>>> >>>>>> On Mon, Feb 5, 2018 at 5:04 PM, hce h <[email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am running protobuf v2 C++ library on Debian 8 ARM processor, I >>>>>>> can only send a text file in bytes field about 17797 bytes, if the file >>>>>>> size is more than 17797 bytes, the file will be corrupted and chopped >>>>>>> off >>>>>>> to 17797 bytes. Are there any workarounds to fix this issue? The >>>>>>> detailed >>>>>>> the package information: >>>>>>> >>>>>>> Package: libprotobuf9 >>>>>>> Source: protobuf >>>>>>> Version: 2.6.1-1 >>>>>>> Installed-Size: 866 >>>>>>> Maintainer: Robert Edmonds <[email protected]> >>>>>>> Architecture: armhf >>>>>>> Depends: libc6 (>= 2.4), libgcc1 (>= 1:4.4.0), libstdc++6 (>= 4.9), >>>>>>> zlib1g (>= 1:1.1.4) >>>>>>> Size: 312100 >>>>>>> SHA256: 7e6a4ffa5c4ccea20a339425c5fe51c0954b6f222960ac87e26d9d7ae5b3 >>>>>>> e79b >>>>>>> SHA1: 807d3ed197797546f87d050cf02ea3d3c96fc4ee >>>>>>> MD5sum: 58bfed0950ad07ef1b06d82a42f24eae >>>>>>> Description: protocol buffers C++ library >>>>>>> Description-md5: c69f5af146ab9bc64e44b912e5af457e >>>>>>> Multi-Arch: same >>>>>>> Homepage: https://code.google.com/p/protobuf/ >>>>>>> Tag: implemented-in::c++, role::shared-lib >>>>>>> Section: libs >>>>>>> Priority: optional >>>>>>> Filename: pool/main/p/protobuf/libprotobuf9_2.6.1-1_armhf.deb >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Protocol Buffers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/protobuf. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
