Hi John, The use case is when there is a really huge amount of data that I need to select and aggregate and select top N aggregated entities. In this scenario I am hitting bad alloc exception when reading data from partitions that occupy approximately 2GB of disk space (with indexes and actual column data), even though I am trying to read projection of data that contains approximately half of all columns and I have still quite lot of available memory.
I am actually on vacation from today so I will try to prepare some repro once I will be back, since this is something that we really need to solve or workaround somehow (since we plan to hove much more data, so even after selection there still will be approximately same amount of rows as in my example where I hit the problems with memory). Thank you Jan -----Original Message----- From: K. John Wu [mailto:[email protected]] Sent: Thursday, July 08, 2010 8:50 PM To: Krivanek, Jan Cc: FastBit Users Subject: Re: [FastBit-users] Memory leak when selecting data and exception is thrown Hi, Jan, Do you mind to describe your use case a bit more? Better yet, produce a small test program that I can use for tracking down this memory leak problem? Thanks. John On 7/8/2010 9:23 AM, Krivanek, Jan wrote: > I just realized that ibis::bord::bufferList is the vector of pointers, > so this is probably not a case. But anyway did anybody came across > similar problem or are there any hints how to better use the memory? > > Thanks > Jan > > -----Original Message----- > From: Krivanek, Jan > Sent: Thursday, July 08, 2010 6:21 PM > To: Krivanek, Jan; 'K. John Wu'; 'FastBit Users' > Subject: RE: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi John, > > I finally found some time to play with this new functionality. I like > the idea pretty much. I used the revision 301 and found a strange > behavior. When there is not sufficient amount of memory and > ibis::fileManager::storage::storage (called from inside > ibis::table::select where the ibis::util::guard was instantiated) throws > an exception the ibis::bord::freeBuffers method is called but will not > free-up anything because its parameters - the buf buffer, and also the > list of types, are already an empty vectors. I didn't have time to play > with it more, I can try to dig-in more and provide more information > after my vacations, but there is one thing that is currently more > concerning me: > > The out of memory exceptions are thrown much earlier then I would > expected (e.g. when trying to allocate ~200Mb of memory when I have more > than 800Mb available). The reason seems to be that the buffer inside > ibis::table::select (line 3731, ibis::bord::bufferList buff;) is an > vector an so the code is trying to stuff all data into one continuous > block of memory. Were there some plans or testing around there. Wouldn't > it help if the ibis::bord::bufferList would be a list instead of a > vector? I am pretty sure that somebody already investigated this, so I > just wanted to ask if there are some plans for changes here or if there > are some hints how to easily reduce the need for huge continuous blocks > of memory. > > Thanks much for your contribution > Jan > > -----Original Message----- > From: Krivanek, Jan > Sent: Thursday, June 24, 2010 5:33 PM > To: 'K. John Wu'; FastBit Users > Subject: RE: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi John, > > Thank you for your update. I will try to check the impact of this > changes as soon as I will find some spare time for it - hopefully during > next week. > > Thanks > Jan > > -----Original Message----- > From: K. John Wu [mailto:[email protected]] > Sent: Wednesday, June 23, 2010 7:09 AM > To: FastBit Users > Cc: Krivanek, Jan > Subject: Re: [FastBit-users] Memory leak when selecting data and > exception is thrown > > Hi, Jan, > > We have wrapped the raw pointers in ibis::table::select with auto_ptr > and arrays of raw pointers in a set of scope guards (ibis::util::guard > in src/util.h). Please let us know if they addresses the problem > you've observed. Thanks for the report. > > John > > > On 6/14/2010 10:53 AM, Krivanek, Jan wrote: >> When I called FastBit queries form code that was running in infinite >> loop (and executing requested queries), I eventually end up > (bad_alloc) >> exceptions in FileManager and huge amount of consumed memory. I tried > to >> track the problem down and found that exceptions from fileManager are >> not caught on some places in code and so memory remains consumed. I >> tried to fix the problem in >> >> ibis::table::select(const std::vector<const ibis::part*>& mylist, > const >> char *sel, const ibis::qExpr *cond) >> >> function that was being called by wrapping calls to >> ibis::util::addIncoreData and final groupby() in try-catch and > releasing >> local buffers when exception throw. Fixing of this partially resolved >> the issue, but I am still resolving this as some significant amount of >> memory still remains unreleased and program still eventually fails. >> >> Thank you >> >> Jan Krivanek >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
