[protobuf] Re: Status of protobufs
Hello, Thanks for the status update. I guess I will be reusing the message delimiter I had developed a couple of years ago =]. Jean-Sebastien On Aug 26, 12:49 pm, Evan Jones ev...@mit.edu wrote: On Aug 26, 2010, at 12:07 , Jean-Sebastien Stoezel wrote: More specifically how they are parsed from real time datastreams? You should manually insert a leading length of next message field into the data stream. The Java implementation even has a shortcut methods for this (see below). In C++ you have to implement it yourself, but it is only a few lines of code. See: http://code.google.com/apis/protocolbuffers/docs/techniques.html#stre... http://code.google.com/apis/protocolbuffers/docs/reference/java/com/g...)http://code.google.com/apis/protocolbuffers/docs/reference/java/com/g...) Evan -- Evan Joneshttp://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Issue 216 in protobuf: FileOutputStream::Flush broken?
Status: New Owner: Labels: Type-Defect Priority-Medium New issue 216 by rosskinder: FileOutputStream::Flush broken? http://code.google.com/p/protobuf/issues/detail?id=216 Either I am misunderstanding the point of FileOuputStream::Flush(), or it doesn't work. I suspect the latter. In the attached minimal example when complied with Visual Studio 2008, the call to Flush() causes 8192 bytes to be written. The extra bytes are all 0xCD, the Debug CRT's default for newly allocated bytes. I expect only the bytes I've actually written to the stream to be written. I have not tried other platforms. Attachments: file_output_stream_bug.cc 1.4 KB -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 216 in protobuf: FileOutputStream::Flush broken?
Comment #1 on issue 216 by ev...@mit.edu: FileOutputStream::Flush broken? http://code.google.com/p/protobuf/issues/detail?id=216 The problem is that the CodedOutputStream assumes that it owns the underlying FileOutputStream. Try the following: coded_stream-WriteRaw(header, 8); delete coded_stream; // Flushes the CodedOutputStream bool rv = raw_stream-Flush(); This should do what you expect. This weirdness is due to the fact that the FileOutputStream is a zero copy stream, so CodedOutputStream gets a big buffer from FileOutputStream, then uses it. In the destructor, it tells the FileOutputStream I actually only wrote X bytes, and then the FileOutputStream does the right thing. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Memory leak detected in protobuf
Hello, I am observing a memory leak after including the protobufs in my project. I would like to investigate whether this leak is due to this library or the use I make of it. I am using the protobufs (latest version) with VC++, running 3 threads. 2 server threads are pending, waiting for connections, another thread acts as client, connecting to the servers and sending messages to them. I tested this code without the protobufs and I did not notice an increase in memory usage over time. After adding the protobufs, I see a constant increase in memory usage. I instrumented the code with VC++ crtdbg library, the tool used to detect memory leaks. The tool is reporting memory leaks which I believe point to protobuf. I read the crtdbg documentation and if _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the file where the leak happened. However no file name is provided by the tool, it could be because I statically link to the protobufs. Anyhow, from the server thread's context, protobufs are stored in a vector. A packet delimiter parses data from a socket. Each packet is parsed for a single protobuf message. Pseudo code: m_IncomingMessages is defined as std::vectorLFLComProtocol::LFLComProtocolMsg m_IncomingMessages, where LFLComProtocol::LFLComProtocolMsg is a protobuf message. // adding a protobuf to the protobuf queue, after a packet of data was received // Add an element to the vector m_IncomingMessages.resize(m_IncomingMessages.size() + 1); // Parse the data from the new message m_IncomingMessages[m_IncomingMessages.size() - 1].ParseFromArray(p_Data, dataSize); // consuming the protobufs, FIFO style m_IncomingMessages.erase(m_IncomingMessages.begin()); I have checked that the messages are properly queued and dequeued, the size of the vector does not increase and stay constant (usually 0 since the message is consumed right away) At this point I am thinking that this way of erasing may be causing the leak. From the STL documentation I gather that the vector::erase method is supposed to call the destructor for each element... Is this good enough to free all the memory allocated for the message? Any other idea why this memory leak is happening? Jean-Sebastien -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Memory leak detected in protobuf
On Thu, Sep 2, 2010 at 08:03, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, I am observing a memory leak after including the protobufs in my project. I would like to investigate whether this leak is due to this library or the use I make of it. I am using the protobufs (latest version) with VC++, running 3 threads. 2 server threads are pending, waiting for connections, another thread acts as client, connecting to the servers and sending messages to them. I tested this code without the protobufs and I did not notice an increase in memory usage over time. After adding the protobufs, I see a constant increase in memory usage. I instrumented the code with VC++ crtdbg library, the tool used to detect memory leaks. The tool is reporting memory leaks which I believe point to protobuf. I read the crtdbg documentation and if _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the file where the leak happened. However no file name is provided by the tool, it could be because I statically link to the protobufs. Anyhow, from the server thread's context, protobufs are stored in a vector. A packet delimiter parses data from a socket. Each packet is parsed for a single protobuf message. Pseudo code: m_IncomingMessages is defined as std::vectorLFLComProtocol::LFLComProtocolMsg m_IncomingMessages, where LFLComProtocol::LFLComProtocolMsg is a protobuf message. // adding a protobuf to the protobuf queue, after a packet of data was received // Add an element to the vector m_IncomingMessages.resize(m_IncomingMessages.size() + 1); // Parse the data from the new message m_IncomingMessages[m_IncomingMessages.size() - 1].ParseFromArray(p_Data, dataSize); // consuming the protobufs, FIFO style m_IncomingMessages.erase(m_IncomingMessages.begin()); Can you create a compilable self contained C++ file including proto-file (stripped down to the essentials) with the essence of your code which would make it simpler (for you and people on the list) to verify if there is a problem ? Also, I would probably not use protocol buffers as values within vectors (esp. if they're big - this advice is true for any bigger object) because you implicitly might call copy constructors if you resize() the vector and it has to realloc. Rather allocate protos on the heap (new/delete). This allows for further improvements later, like having a pool of pre-allocated protobuffers. -h I have checked that the messages are properly queued and dequeued, the size of the vector does not increase and stay constant (usually 0 since the message is consumed right away) At this point I am thinking that this way of erasing may be causing the leak. From the STL documentation I gather that the vector::erase method is supposed to call the destructor for each element... Is this good enough to free all the memory allocated for the message? Any other idea why this memory leak is happening? Jean-Sebastien -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 216 in protobuf: FileOutputStream::Flush broken?
Comment #2 on issue 216 by rosskinder: FileOutputStream::Flush broken? http://code.google.com/p/protobuf/issues/detail?id=216 Works for me. Thanks. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Status of protobufs
See http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming Your solution of trying to parse after each byte received is not just slow, it's completely incorrect. It's entirely possible, and quite likely, that if you break the encoded version of a message in half, each half will parse successfully on its own, but the contents of the message will be incorrect. On Thu, Sep 2, 2010 at 7:29 AM, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, Thanks for the status update. I guess I will be reusing the message delimiter I had developed a couple of years ago =]. Jean-Sebastien On Aug 26, 12:49 pm, Evan Jones ev...@mit.edu wrote: On Aug 26, 2010, at 12:07 , Jean-Sebastien Stoezel wrote: More specifically how they are parsed from real time datastreams? You should manually insert a leading length of next message field into the data stream. The Java implementation even has a shortcut methods for this (see below). In C++ you have to implement it yourself, but it is only a few lines of code. See: http://code.google.com/apis/protocolbuffers/docs/techniques.html#stre... http://code.google.com/apis/protocolbuffers/docs/reference/java/com/g...)http://code.google.com/apis/protocolbuffers/docs/reference/java/com/g.. .) Evan -- Evan Joneshttp://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Status of protobufs
Hence I'm now using delimiters. Thanks for the headsup. On Sep 2, 2010 11:53 AM, Daniel Wright dwri...@google.com wrote: See http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming Your solution of trying to parse after each byte received is not just slow, it's completely incorrect. It's entirely possible, and quite likely, that if you break the encoded version of a message in half, each half will parse successfully on its own, but the contents of the message will be incorrect. On Thu, Sep 2, 2010 at 7:29 AM, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, ... -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Status of protobufs
On Thu, Sep 2, 2010 at 09:59, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hence I'm now using delimiters. But make sure to use length delimiting: length + message. Don't feel tempted to just delimit the message with 'special characters' because of course they might be just part of a message. Thanks for the headsup. On Sep 2, 2010 11:53 AM, Daniel Wright dwri...@google.com wrote: See http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming Your solution of trying to parse after each byte received is not just slow, it's completely incorrect. It's entirely possible, and quite likely, that if you break the encoded version of a message in half, each half will parse successfully on its own, but the contents of the message will be incorrect. On Thu, Sep 2, 2010 at 7:29 AM, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, ... -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
