Hi John, I did the testing only with disabled mmap, no other modifications. I'm now sending you an email with link to download the complete log. I'm trying it on an older version from SVN revision 657. Sadly, the newest is not compatible with the tested system.
The log file is quite big 19GB, I compressed it to 2.5GB and the link should be in another email shortly. I'm using https://filesender.cesnet.cz/service for the transfer. Let me know if there is anything more that I can do. Best, Petr On Thu, Mar 20, 2014 at 5:21 PM, K. John Wu <[email protected]> wrote: > Hi, Petr, > > FastBit is only tracking the memory usage of a few object types such > as ibis::array_t and ibis::buffer. Since you are using a compression > layer, I presume that you are also allocating considerable amounts of > memory using malloc or new directly. This would mean FastBit is only > tracking a part of the memory uses. If you turn up the verboseness of > the FastBit code, say, ibis::util::setVerboseLevel(10), and ship me > the log messages, I can take a look and see if I can spot anything > more useful to suggest. > > John > > > On 3/20/14, 6:35 AM, Petr Velan wrote: > > Hi John, > > > > I'm currently working on FastBit support for compressed files. To > > simplify things a bit for a start, I have disabled the MMAP support > > and forced FastBit to use open, read, write and seek functions. Then I > > hooked into these functions and provided a compression layer. The > > problem is that just disabling the MMAP causes FastBit to use so much > > memory that OOM killer terminates the process for some aggregation > > queries. > > > > Is there anything I can do to tell FastBit to limit the amount of > > memory it can use? I thought that only half of the available memory > > will be used by default. I've tested this issue on latest SVN revision. > > > > Thank you in advance for any help, > > Petr Velan > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
