Hi, Jan, Thanks for the confirmation. By the way, the latest SVN version number is 309.
Looking forward to study your test case. John On 7/21/2010 6:36 AM, Krivanek, Jan wrote: > Hi John, > > I wanted to react once more to the original problem - memory leaks after > unsuccessful memory allocation. We tested my scenario once more on the > newest revision (307 I guess), and this problem seems to be already > resolved - nice job here! > > As to the second problem - problem with memory allocation - I will > shortly send offline an example repro for this. > > Thanks > Jan > > -----Original Message----- > From: K. John Wu [mailto:[email protected]] > Sent: Thursday, July 08, 2010 8:50 PM > To: Krivanek, Jan > Cc: FastBit Users > Subject: Re: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi, Jan, > > Do you mind to describe your use case a bit more? Better yet, produce > a small test program that I can use for tracking down this memory leak > problem? Thanks. > > John > > > On 7/8/2010 9:23 AM, Krivanek, Jan wrote: >> I just realized that ibis::bord::bufferList is the vector of pointers, >> so this is probably not a case. But anyway did anybody came across >> similar problem or are there any hints how to better use the memory? >> >> Thanks >> Jan >> >> -----Original Message----- >> From: Krivanek, Jan >> Sent: Thursday, July 08, 2010 6:21 PM >> To: Krivanek, Jan; 'K. John Wu'; 'FastBit Users' >> Subject: RE: [FastBit-users] Memory leak when selecting data and >> exception is thrown >> >> Hi John, >> >> I finally found some time to play with this new functionality. I like >> the idea pretty much. I used the revision 301 and found a strange >> behavior. When there is not sufficient amount of memory and >> ibis::fileManager::storage::storage (called from inside >> ibis::table::select where the ibis::util::guard was instantiated) > throws >> an exception the ibis::bord::freeBuffers method is called but will not >> free-up anything because its parameters - the buf buffer, and also the >> list of types, are already an empty vectors. I didn't have time to > play >> with it more, I can try to dig-in more and provide more information >> after my vacations, but there is one thing that is currently more >> concerning me: >> >> The out of memory exceptions are thrown much earlier then I would >> expected (e.g. when trying to allocate ~200Mb of memory when I have > more >> than 800Mb available). The reason seems to be that the buffer inside >> ibis::table::select (line 3731, ibis::bord::bufferList buff;) is an >> vector an so the code is trying to stuff all data into one continuous >> block of memory. Were there some plans or testing around there. > Wouldn't >> it help if the ibis::bord::bufferList would be a list instead of a >> vector? I am pretty sure that somebody already investigated this, so I >> just wanted to ask if there are some plans for changes here or if > there >> are some hints how to easily reduce the need for huge continuous > blocks >> of memory. >> >> Thanks much for your contribution >> Jan >> >> -----Original Message----- >> From: Krivanek, Jan >> Sent: Thursday, June 24, 2010 5:33 PM >> To: 'K. John Wu'; FastBit Users >> Subject: RE: [FastBit-users] Memory leak when selecting data and >> exception is thrown >> >> Hi John, >> >> Thank you for your update. I will try to check the impact of this >> changes as soon as I will find some spare time for it - hopefully > during >> next week. >> >> Thanks >> Jan >> >> -----Original Message----- >> From: K. John Wu [mailto:[email protected]] >> Sent: Wednesday, June 23, 2010 7:09 AM >> To: FastBit Users >> Cc: Krivanek, Jan >> Subject: Re: [FastBit-users] Memory leak when selecting data and >> exception is thrown >> >> Hi, Jan, >> >> We have wrapped the raw pointers in ibis::table::select with auto_ptr >> and arrays of raw pointers in a set of scope guards (ibis::util::guard >> in src/util.h). Please let us know if they addresses the problem >> you've observed. Thanks for the report. >> >> John >> >> >> On 6/14/2010 10:53 AM, Krivanek, Jan wrote: >>> When I called FastBit queries form code that was running in infinite >>> loop (and executing requested queries), I eventually end up >> (bad_alloc) >>> exceptions in FileManager and huge amount of consumed memory. I tried >> to >>> track the problem down and found that exceptions from fileManager are >>> not caught on some places in code and so memory remains consumed. I >>> tried to fix the problem in >>> >>> ibis::table::select(const std::vector<const ibis::part*>& mylist, >> const >>> char *sel, const ibis::qExpr *cond) >>> >>> function that was being called by wrapping calls to >>> ibis::util::addIncoreData and final groupby() in try-catch and >> releasing >>> local buffers when exception throw. Fixing of this partially resolved >>> the issue, but I am still resolving this as some significant amount > of >>> memory still remains unreleased and program still eventually fails. >>> >>> Thank you >>> >>> Jan Krivanek >>> >>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
