I am running against multiple data partitions. The current dataset I'm using is around 19 million rows, around 1 GB, and spread cross 6 partitions. Unfortunately it contains sensitive data which I can't share at this time.
Because the data is so large, I can easily spot the memory usage increase around 100 MB with each query run. When I ran the ibis.exe through the memory profiler it shows that m_begin buffer was being leaked. Stepping through the code, starting on line 1844 in the method enlarge, ibis::fileManager::storage cp(nelm); memcpy(cp.m_begin, m_begin, oldsize); swap(cp); cp's n_ref is zero and the current object's n_ref is 1. The swap method ends up swapping the buffers and the reference. When cp exits scope and is destroyed, it's clear() method will not free m_begin because nref() > 0 I'm not sure I see a call to nosharing in my stack crawl, which looks like ibis.exe!ibis::fileManager::storage::clear() Line 1863 C++ ibis.exe!ibis::fileManager::storage::~storage() Line 270 + 0x45 bytes C++ ibis.exe!ibis::fileManager::storage::enlarge(unsigned int nelm=24687688) Line 1847 + 0xb bytes C++ ibis.exe!ibis::array_t<int>::reserve(unsigned int n=4307553) Line 1214 C++ ibis.exe!ibis::util::addIncoreData<int> ibis.exe!ibis::table::select ibis.exe!ibis::table::select ibis.exe!ibis::mensa::select(const char * sel=0x016c0930, const char * cond=0x0018f2f0) ibis.exe!tableSelect ibis.exe!parseString ibis.exe!main ibis.exe!__tmainCRTStartup() Line 266 + 0x19 bytes C ibis.exe!mainCRTStartup() Line 182 C When I get a chance I will try to reproduce it with the test data. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of K. John Wu Sent: Thursday, February 11, 2010 11:09 PM To: FastBit Users Subject: Re: [FastBit-users] Memory leak in fileManager::storage::enlarge ? Hi, Zeid, Thanks for checking this problem out. I am probably missing something obvious. Do you have any idea how I might be able to reproduce the problem? I thought the variable nref should be one because the called to enlarge is always proceeded by a call to nosharing. Please help me figure this one out. John PS: I presume the problem occurs when querying multiple data partitions. If that is the case, you might want to know that 'make check' now has some test cases involving multiple data partitions. It would be perfect if you can reproduce the problem with data in tests/tmp/t2 (after you run make check). Otherwise, you might have to give me some sample data to reproduce the problem. On 2/11/2010 8:30 AM, Derhally, Zeid wrote: > Hi John, > > I tried this with 1.1.6 and the memory leak is still present. > > Looking at the code I still see that the enlarge method is still calling > the swap method which ends up replacing the reference count of the > temporary storage object with the existing one which results in the > buffer not getting freed up. > > Zeid > > > -----Original Message----- > From: K. John Wu [mailto:[email protected]] > Sent: Wednesday, February 10, 2010 10:41 PM > To: FastBit Users > Cc: Derhally, Zeid > Subject: Re: [FastBit-users] Memory leak in > fileManager::storage::enlarge ? > > Hi, Zeid, > > Thanks for the report. This particular issue should have been fixed > in ibis1.1.6. If you get a chance, please give it a try. > > There is a note about the change from Jan 29 in ChangeLog, which reads > > * src/fileManager.cpp: storage::enlarge used copy-and-swap idiom > incorrectly and caused the client to assume the operation was a > failure > > Please feel free to let us know if you continue to see problems with > this function. > > John > > > On 2/10/2010 2:47 PM, Derhally, Zeid wrote: >> Hi, >> >> I believe I've run into a memory leak with ibis 1.1.5 >> >> In the function, ibis::fileManager::storage::enlarge, there is a call > to >> swap() around line 1909, looks like >> >> try { >> >> ibis::fileManager::storage cp(nelm); >> >> memcpy(cp.m_begin, m_begin, oldsize); >> >> cp.m_end; >> >> swap(cp); >> >> } >> >> catch (...) { >> >> LOGGER(ibis::gVerbose>= 0) >> >> << "Warning -- "<< evt<< " failed to allocate new storage," >> >> "current storage unchanged"; >> >> } >> >> The swap function also swaps the reference count value which causes > the >> m_begin buffer to not be freed up because the temporary object on the >> stack inherited the reference count of the current object. >> >> I've made a change in my sources where I did not swap the reference >> count when enlarging the storage and that took care of the leak. >> >> Thanks. >> >> Zeid >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
