Hi John, I finally found some time to play with this new functionality. I like the idea pretty much. I used the revision 301 and found a strange behavior. When there is not sufficient amount of memory and ibis::fileManager::storage::storage (called from inside ibis::table::select where the ibis::util::guard was instantiated) throws an exception the ibis::bord::freeBuffers method is called but will not free-up anything because its parameters - the buf buffer, and also the list of types, are already an empty vectors. I didn't have time to play with it more, I can try to dig-in more and provide more information after my vacations, but there is one thing that is currently more concerning me:
The out of memory exceptions are thrown much earlier then I would expected (e.g. when trying to allocate ~200Mb of memory when I have more than 800Mb available). The reason seems to be that the buffer inside ibis::table::select (line 3731, ibis::bord::bufferList buff;) is an vector an so the code is trying to stuff all data into one continuous block of memory. Were there some plans or testing around there. Wouldn't it help if the ibis::bord::bufferList would be a list instead of a vector? I am pretty sure that somebody already investigated this, so I just wanted to ask if there are some plans for changes here or if there are some hints how to easily reduce the need for huge continuous blocks of memory. Thanks much for your contribution Jan -----Original Message----- From: Krivanek, Jan Sent: Thursday, June 24, 2010 5:33 PM To: 'K. John Wu'; FastBit Users Subject: RE: [FastBit-users] Memory leak when selecting data and exception is thrown Hi John, Thank you for your update. I will try to check the impact of this changes as soon as I will find some spare time for it - hopefully during next week. Thanks Jan -----Original Message----- From: K. John Wu [mailto:[email protected]] Sent: Wednesday, June 23, 2010 7:09 AM To: FastBit Users Cc: Krivanek, Jan Subject: Re: [FastBit-users] Memory leak when selecting data and exception is thrown Hi, Jan, We have wrapped the raw pointers in ibis::table::select with auto_ptr and arrays of raw pointers in a set of scope guards (ibis::util::guard in src/util.h). Please let us know if they addresses the problem you've observed. Thanks for the report. John On 6/14/2010 10:53 AM, Krivanek, Jan wrote: > When I called FastBit queries form code that was running in infinite > loop (and executing requested queries), I eventually end up (bad_alloc) > exceptions in FileManager and huge amount of consumed memory. I tried to > track the problem down and found that exceptions from fileManager are > not caught on some places in code and so memory remains consumed. I > tried to fix the problem in > > ibis::table::select(const std::vector<const ibis::part*>& mylist, const > char *sel, const ibis::qExpr *cond) > > function that was being called by wrapping calls to > ibis::util::addIncoreData and final groupby() in try-catch and releasing > local buffers when exception throw. Fixing of this partially resolved > the issue, but I am still resolving this as some significant amount of > memory still remains unreleased and program still eventually fails. > > Thank you > > Jan Krivanek > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
