Hi John, I managed to get recent enough compiler to build latest SVN revision. I rebased my changes on that and tried a query on a quite large data sample. I did not notice any problems apart from following warning:
Warning -- fileManager::storage::ctor unable to find 58,720,256 bytes of space in memory What does happen when this warning is encountered? Is the query not completed and I§m given only partial answer, or does it find another way to finish the query? Thanks for your help, Petr On Tue, Jun 17, 2014 at 8:16 PM, K. John Wu <[email protected]> wrote: > Hi, Petr, > > Not sure if you are still interested in this, but here is what I have > been puzzling over. > > There are many cases where an array_t<unsigned> is deleted first, but > then copied into another array_t<unsigned>. One possible scenario for > this is that a pointer to a bitvector is returned from a function that > generated the bitvector as a local object. From the log file, I am > not able to tell if this true or not. > > I have tried to run valgrind a few times, but not able to spot this > problem so far. If you have a small test case for me, I am willing to > look into it a little further. > > John > > > > On 4/7/14, 11:28 PM, Petr Velan wrote: > > Hi John, > > > > did you by any chance have a bit of time to look into this matter? > > > > Thanks, > > Petr > > > > > > On Fri, Mar 21, 2014 at 1:25 PM, Petr Velan <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi John, > > > > I did the testing only with disabled mmap, no other modifications. > > I'm now sending you an email with link to download the complete > > log. I'm trying it on an older version from SVN revision 657. > > Sadly, the newest is not compatible with the tested system. > > > > The log file is quite big 19GB, I compressed it to 2.5GB and the > > link should be in another email shortly. I'm using > > https://filesender.cesnet.cz/ service for the transfer. > > > > Let me know if there is anything more that I can do. > > > > Best, > > Petr > > > > > > > > > > On Thu, Mar 20, 2014 at 5:21 PM, K. John Wu <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, Petr, > > > > FastBit is only tracking the memory usage of a few object > > types such > > as ibis::array_t and ibis::buffer. Since you are using a > > compression > > layer, I presume that you are also allocating considerable > > amounts of > > memory using malloc or new directly. This would mean FastBit > > is only > > tracking a part of the memory uses. If you turn up the > > verboseness of > > the FastBit code, say, ibis::util::setVerboseLevel(10), and > > ship me > > the log messages, I can take a look and see if I can spot > anything > > more useful to suggest. > > > > John > > > > > > On 3/20/14, 6:35 AM, Petr Velan wrote: > > > Hi John, > > > > > > I'm currently working on FastBit support for compressed > > files. To > > > simplify things a bit for a start, I have disabled the MMAP > > support > > > and forced FastBit to use open, read, write and seek > > functions. Then I > > > hooked into these functions and provided a compression > > layer. The > > > problem is that just disabling the MMAP causes FastBit to > > use so much > > > memory that OOM killer terminates the process for some > > aggregation > > > queries. > > > > > > Is there anything I can do to tell FastBit to limit the > > amount of > > > memory it can use? I thought that only half of the available > > memory > > > will be used by default. I've tested this issue on latest > > SVN revision. > > > > > > Thank you in advance for any help, > > > Petr Velan > > > > > > > > > _______________________________________________ > > > FastBit-users mailing list > > > [email protected] > > <mailto:[email protected]> > > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] <mailto: > [email protected]> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
