Hi Adam, I've finally fixed the problem, it is actually nothing to do with the protobuf, it was the issue in boost async_write.
Sorry for raising a false alarm here. Thank you. Kind regards, - j On Tue, Feb 13, 2018 at 7:29 AM, Adam Cozzette <[email protected]> wrote: > Could you post some code showing the problem? I still suspect that it's > probably an issue involving how the messages are framed, or maybe some kind > of buffering issue. You could also try serializing the message to a string > (e.g. with message.SerializeToString(&s)) and see if you can successfully > parse the message again. That could help narrow down whether it's a problem > with protobuf serialization or something else. > > On Thu, Feb 8, 2018 at 9:29 PM, hce h <[email protected]> wrote: > >> Hi Adam, I built from the git source, but that did not work. After >> several hours debugging, I found the problem but not quite sure the which >> library causes the problem: >> >> I sent two protobuf messages consecutively in server side within the same >> thread, it uses boost::asio::streambuf and std::ostream to call >> message.SerializeToOstream(&os), then to call boost::asio::async_write >> to send the message. For each sendMessage, a new shared pointer streambuf >> is used, it should not cause buffer overlap. It works file when the message >> smaller then 17 KB, when the message is larger, it corrupted the serialized >> message, I suspect it is the protobuf serialization issue rather than boost >> asio asyc write issue, but I don't know it enough, you might give me some >> insight... >> >> >> >> On Wednesday, February 7, 2018 at 12:10:30 PM UTC+11, hce h wrote: >>> >>> Thanks Adam, good to know I don't need to change my protocol to set >>> syntax = "proto2", I am building the code right now, will let you know if >>> the new library fix the issue or not. >>> >>> >>> >>> On Wednesday, February 7, 2018 at 11:51:02 AM UTC+11, Adam Cozzette >>> wrote: >>>> >>>> You can still upgrade to our most recent code on GitHub. As long as you >>>> don't set syntax = "proto3" in your .proto files, your code should work >>>> fine without needing any modifications. >>>> >>>> On Tue, Feb 6, 2018 at 4:25 PM, hce h <[email protected]> wrote: >>>> >>>>> I am going to try to build protobuf from git source for armfs to >>>>> verify if the old version protobuf caused the issue or not, the trouble is >>>>> my protobuf code as all defined for protobuf v2, I don't want to run into >>>>> the trouble with protobuf v3 source code, anyway I can defined to build v2 >>>>> only during my build process? >>>>> >>>>> Thank you >>>>> >>>>> >>>>> On Wednesday, February 7, 2018 at 10:44:14 AM UTC+11, hce h wrote: >>>>>> >>>>>> Thanks Adam, I had further investigation, the original post of error >>>>>> was not correct, it was not the problems of length, the file length was >>>>>> correct, but the file was corrupted, there were some junk characters like >>>>>> "^@^@^@^L^B^@^B^P^D^H<80><9b>^F^P]^X<80><9b>^F" inserted to the >>>>>> middle of protobuf bytes / string and chopped off the connect string >>>>>> after >>>>>> serialisation. I don't think there are any issues of coding and >>>>>> serialisation / deserialisation in general, I have used the same code for >>>>>> years without any issues until when the files increased to 17 KB, it >>>>>> corrupted. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wednesday, February 7, 2018 at 4:13:22 AM UTC+11, Adam Cozzette >>>>>> wrote: >>>>>>> >>>>>>> Could you post some example code showing the problem? I would guess >>>>>>> that it's an issue with how your code is framing the message. Serialized >>>>>>> protos do not indicate their own length, so when you parse a message you >>>>>>> have to know how many bytes to expect. If you try to parse the wrong >>>>>>> number >>>>>>> of bytes, you can inadvertently parse the wrong thing or parsing might >>>>>>> fail >>>>>>> entirely. >>>>>>> >>>>>>> On Mon, Feb 5, 2018 at 5:04 PM, hce h <[email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am running protobuf v2 C++ library on Debian 8 ARM processor, I >>>>>>>> can only send a text file in bytes field about 17797 bytes, if the file >>>>>>>> size is more than 17797 bytes, the file will be corrupted and chopped >>>>>>>> off >>>>>>>> to 17797 bytes. Are there any workarounds to fix this issue? The >>>>>>>> detailed >>>>>>>> the package information: >>>>>>>> >>>>>>>> Package: libprotobuf9 >>>>>>>> Source: protobuf >>>>>>>> Version: 2.6.1-1 >>>>>>>> Installed-Size: 866 >>>>>>>> Maintainer: Robert Edmonds <[email protected]> >>>>>>>> Architecture: armhf >>>>>>>> Depends: libc6 (>= 2.4), libgcc1 (>= 1:4.4.0), libstdc++6 (>= 4.9), >>>>>>>> zlib1g (>= 1:1.1.4) >>>>>>>> Size: 312100 >>>>>>>> SHA256: 7e6a4ffa5c4ccea20a339425c5fe51 >>>>>>>> c0954b6f222960ac87e26d9d7ae5b3e79b >>>>>>>> SHA1: 807d3ed197797546f87d050cf02ea3d3c96fc4ee >>>>>>>> MD5sum: 58bfed0950ad07ef1b06d82a42f24eae >>>>>>>> Description: protocol buffers C++ library >>>>>>>> Description-md5: c69f5af146ab9bc64e44b912e5af457e >>>>>>>> Multi-Arch: same >>>>>>>> Homepage: https://code.google.com/p/protobuf/ >>>>>>>> Tag: implemented-in::c++, role::shared-lib >>>>>>>> Section: libs >>>>>>>> Priority: optional >>>>>>>> Filename: pool/main/p/protobuf/libprotobuf9_2.6.1-1_armhf.deb >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Protocol Buffers" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Protocol Buffers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/protobuf. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
