Re: [protobuf] Memory leak detected in protobuf
Calling the destructor is the only thing you need to do to completely free a protobuf object. Please make sure you are calling ShutdownProtobufLibrary() before your program exits, otherwise some leak detectors will report false positives. On Thu, Sep 2, 2010 at 8:03 AM, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, I am observing a memory leak after including the protobufs in my project. I would like to investigate whether this leak is due to this library or the use I make of it. I am using the protobufs (latest version) with VC++, running 3 threads. 2 server threads are pending, waiting for connections, another thread acts as client, connecting to the servers and sending messages to them. I tested this code without the protobufs and I did not notice an increase in memory usage over time. After adding the protobufs, I see a constant increase in memory usage. I instrumented the code with VC++ crtdbg library, the tool used to detect memory leaks. The tool is reporting memory leaks which I believe point to protobuf. I read the crtdbg documentation and if _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the file where the leak happened. However no file name is provided by the tool, it could be because I statically link to the protobufs. Anyhow, from the server thread's context, protobufs are stored in a vector. A packet delimiter parses data from a socket. Each packet is parsed for a single protobuf message. Pseudo code: m_IncomingMessages is defined as std::vectorLFLComProtocol::LFLComProtocolMsg m_IncomingMessages, where LFLComProtocol::LFLComProtocolMsg is a protobuf message. // adding a protobuf to the protobuf queue, after a packet of data was received // Add an element to the vector m_IncomingMessages.resize(m_IncomingMessages.size() + 1); // Parse the data from the new message m_IncomingMessages[m_IncomingMessages.size() - 1].ParseFromArray(p_Data, dataSize); // consuming the protobufs, FIFO style m_IncomingMessages.erase(m_IncomingMessages.begin()); I have checked that the messages are properly queued and dequeued, the size of the vector does not increase and stay constant (usually 0 since the message is consumed right away) At this point I am thinking that this way of erasing may be causing the leak. From the STL documentation I gather that the vector::erase method is supposed to call the destructor for each element... Is this good enough to free all the memory allocated for the message? Any other idea why this memory leak is happening? Jean-Sebastien -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Memory leak detected in protobuf
Hello, I am observing a memory leak after including the protobufs in my project. I would like to investigate whether this leak is due to this library or the use I make of it. I am using the protobufs (latest version) with VC++, running 3 threads. 2 server threads are pending, waiting for connections, another thread acts as client, connecting to the servers and sending messages to them. I tested this code without the protobufs and I did not notice an increase in memory usage over time. After adding the protobufs, I see a constant increase in memory usage. I instrumented the code with VC++ crtdbg library, the tool used to detect memory leaks. The tool is reporting memory leaks which I believe point to protobuf. I read the crtdbg documentation and if _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the file where the leak happened. However no file name is provided by the tool, it could be because I statically link to the protobufs. Anyhow, from the server thread's context, protobufs are stored in a vector. A packet delimiter parses data from a socket. Each packet is parsed for a single protobuf message. Pseudo code: m_IncomingMessages is defined as std::vectorLFLComProtocol::LFLComProtocolMsg m_IncomingMessages, where LFLComProtocol::LFLComProtocolMsg is a protobuf message. // adding a protobuf to the protobuf queue, after a packet of data was received // Add an element to the vector m_IncomingMessages.resize(m_IncomingMessages.size() + 1); // Parse the data from the new message m_IncomingMessages[m_IncomingMessages.size() - 1].ParseFromArray(p_Data, dataSize); // consuming the protobufs, FIFO style m_IncomingMessages.erase(m_IncomingMessages.begin()); I have checked that the messages are properly queued and dequeued, the size of the vector does not increase and stay constant (usually 0 since the message is consumed right away) At this point I am thinking that this way of erasing may be causing the leak. From the STL documentation I gather that the vector::erase method is supposed to call the destructor for each element... Is this good enough to free all the memory allocated for the message? Any other idea why this memory leak is happening? Jean-Sebastien -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Memory leak detected in protobuf
On Thu, Sep 2, 2010 at 08:03, Jean-Sebastien Stoezel js.stoe...@gmail.com wrote: Hello, I am observing a memory leak after including the protobufs in my project. I would like to investigate whether this leak is due to this library or the use I make of it. I am using the protobufs (latest version) with VC++, running 3 threads. 2 server threads are pending, waiting for connections, another thread acts as client, connecting to the servers and sending messages to them. I tested this code without the protobufs and I did not notice an increase in memory usage over time. After adding the protobufs, I see a constant increase in memory usage. I instrumented the code with VC++ crtdbg library, the tool used to detect memory leaks. The tool is reporting memory leaks which I believe point to protobuf. I read the crtdbg documentation and if _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the file where the leak happened. However no file name is provided by the tool, it could be because I statically link to the protobufs. Anyhow, from the server thread's context, protobufs are stored in a vector. A packet delimiter parses data from a socket. Each packet is parsed for a single protobuf message. Pseudo code: m_IncomingMessages is defined as std::vectorLFLComProtocol::LFLComProtocolMsg m_IncomingMessages, where LFLComProtocol::LFLComProtocolMsg is a protobuf message. // adding a protobuf to the protobuf queue, after a packet of data was received // Add an element to the vector m_IncomingMessages.resize(m_IncomingMessages.size() + 1); // Parse the data from the new message m_IncomingMessages[m_IncomingMessages.size() - 1].ParseFromArray(p_Data, dataSize); // consuming the protobufs, FIFO style m_IncomingMessages.erase(m_IncomingMessages.begin()); Can you create a compilable self contained C++ file including proto-file (stripped down to the essentials) with the essence of your code which would make it simpler (for you and people on the list) to verify if there is a problem ? Also, I would probably not use protocol buffers as values within vectors (esp. if they're big - this advice is true for any bigger object) because you implicitly might call copy constructors if you resize() the vector and it has to realloc. Rather allocate protos on the heap (new/delete). This allows for further improvements later, like having a pool of pre-allocated protobuffers. -h I have checked that the messages are properly queued and dequeued, the size of the vector does not increase and stay constant (usually 0 since the message is consumed right away) At this point I am thinking that this way of erasing may be causing the leak. From the STL documentation I gather that the vector::erase method is supposed to call the destructor for each element... Is this good enough to free all the memory allocated for the message? Any other idea why this memory leak is happening? Jean-Sebastien -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
