Hi, Petr, Not sure if you are still interested in this, but here is what I have been puzzling over.
There are many cases where an array_t<unsigned> is deleted first, but then copied into another array_t<unsigned>. One possible scenario for this is that a pointer to a bitvector is returned from a function that generated the bitvector as a local object. From the log file, I am not able to tell if this true or not. I have tried to run valgrind a few times, but not able to spot this problem so far. If you have a small test case for me, I am willing to look into it a little further. John On 4/7/14, 11:28 PM, Petr Velan wrote: > Hi John, > > did you by any chance have a bit of time to look into this matter? > > Thanks, > Petr > > > On Fri, Mar 21, 2014 at 1:25 PM, Petr Velan <[email protected] > <mailto:[email protected]>> wrote: > > Hi John, > > I did the testing only with disabled mmap, no other modifications. > I'm now sending you an email with link to download the complete > log. I'm trying it on an older version from SVN revision 657. > Sadly, the newest is not compatible with the tested system. > > The log file is quite big 19GB, I compressed it to 2.5GB and the > link should be in another email shortly. I'm using > https://filesender.cesnet.cz/ service for the transfer. > > Let me know if there is anything more that I can do. > > Best, > Petr > > > > > On Thu, Mar 20, 2014 at 5:21 PM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Petr, > > FastBit is only tracking the memory usage of a few object > types such > as ibis::array_t and ibis::buffer. Since you are using a > compression > layer, I presume that you are also allocating considerable > amounts of > memory using malloc or new directly. This would mean FastBit > is only > tracking a part of the memory uses. If you turn up the > verboseness of > the FastBit code, say, ibis::util::setVerboseLevel(10), and > ship me > the log messages, I can take a look and see if I can spot anything > more useful to suggest. > > John > > > On 3/20/14, 6:35 AM, Petr Velan wrote: > > Hi John, > > > > I'm currently working on FastBit support for compressed > files. To > > simplify things a bit for a start, I have disabled the MMAP > support > > and forced FastBit to use open, read, write and seek > functions. Then I > > hooked into these functions and provided a compression > layer. The > > problem is that just disabling the MMAP causes FastBit to > use so much > > memory that OOM killer terminates the process for some > aggregation > > queries. > > > > Is there anything I can do to tell FastBit to limit the > amount of > > memory it can use? I thought that only half of the available > memory > > will be used by default. I've tested this issue on latest > SVN revision. > > > > Thank you in advance for any help, > > Petr Velan > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > <mailto:[email protected]> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > _______________________________________________ > FastBit-users mailing list > [email protected] <mailto:[email protected]> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
