Hi, Jan, Do you mind to describe your use case a bit more? Better yet, produce a small test program that I can use for tracking down this memory leak problem? Thanks.
John On 7/8/2010 9:23 AM, Krivanek, Jan wrote: > I just realized that ibis::bord::bufferList is the vector of pointers, > so this is probably not a case. But anyway did anybody came across > similar problem or are there any hints how to better use the memory? > > Thanks > Jan > > -----Original Message----- > From: Krivanek, Jan > Sent: Thursday, July 08, 2010 6:21 PM > To: Krivanek, Jan; 'K. John Wu'; 'FastBit Users' > Subject: RE: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi John, > > I finally found some time to play with this new functionality. I like > the idea pretty much. I used the revision 301 and found a strange > behavior. When there is not sufficient amount of memory and > ibis::fileManager::storage::storage (called from inside > ibis::table::select where the ibis::util::guard was instantiated) throws > an exception the ibis::bord::freeBuffers method is called but will not > free-up anything because its parameters - the buf buffer, and also the > list of types, are already an empty vectors. I didn't have time to play > with it more, I can try to dig-in more and provide more information > after my vacations, but there is one thing that is currently more > concerning me: > > The out of memory exceptions are thrown much earlier then I would > expected (e.g. when trying to allocate ~200Mb of memory when I have more > than 800Mb available). The reason seems to be that the buffer inside > ibis::table::select (line 3731, ibis::bord::bufferList buff;) is an > vector an so the code is trying to stuff all data into one continuous > block of memory. Were there some plans or testing around there. Wouldn't > it help if the ibis::bord::bufferList would be a list instead of a > vector? I am pretty sure that somebody already investigated this, so I > just wanted to ask if there are some plans for changes here or if there > are some hints how to easily reduce the need for huge continuous blocks > of memory. > > Thanks much for your contribution > Jan > > -----Original Message----- > From: Krivanek, Jan > Sent: Thursday, June 24, 2010 5:33 PM > To: 'K. John Wu'; FastBit Users > Subject: RE: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi John, > > Thank you for your update. I will try to check the impact of this > changes as soon as I will find some spare time for it - hopefully during > next week. > > Thanks > Jan > > -----Original Message----- > From: K. John Wu [mailto:[email protected]] > Sent: Wednesday, June 23, 2010 7:09 AM > To: FastBit Users > Cc: Krivanek, Jan > Subject: Re: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi, Jan, > > We have wrapped the raw pointers in ibis::table::select with auto_ptr > and arrays of raw pointers in a set of scope guards (ibis::util::guard > in src/util.h). Please let us know if they addresses the problem > you've observed. Thanks for the report. > > John > > > On 6/14/2010 10:53 AM, Krivanek, Jan wrote: >> When I called FastBit queries form code that was running in infinite >> loop (and executing requested queries), I eventually end up > (bad_alloc) >> exceptions in FileManager and huge amount of consumed memory. I tried > to >> track the problem down and found that exceptions from fileManager are >> not caught on some places in code and so memory remains consumed. I >> tried to fix the problem in >> >> ibis::table::select(const std::vector<const ibis::part*>& mylist, > const >> char *sel, const ibis::qExpr *cond) >> >> function that was being called by wrapping calls to >> ibis::util::addIncoreData and final groupby() in try-catch and > releasing >> local buffers when exception throw. Fixing of this partially resolved >> the issue, but I am still resolving this as some significant amount of >> memory still remains unreleased and program still eventually fails. >> >> Thank you >> >> Jan Krivanek >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
