Hi, Petr, When fileManager fails to allocate memory, the operations depending the memory would fail. There isn't a systematic recovery mechanism in place to deal with this in FastBit. You might consider the following options:
- increase the memory allowed by FastBit by setting a high cache limit on fileManager. You can do this by creating a configuration file to be read by FastBit. The parameter fileManager.maxBytes is used to specify the memory cache size. Alternatively you can call ibis::fileManager::adjustCacheSize. - breaking you data into smaller partitions. This has the potential of complicating your data processing, but will reduce the memory requirement in most cases. John On 7/27/14 11:56 PM, Petr Velan wrote: > Hi John, > > I managed to get recent enough compiler to build latest SVN revision. I > rebased my changes on that and tried a query on a quite large data sample. > I did not notice any problems apart from following warning: > > Warning -- fileManager::storage::ctor unable to find 58,720,256 bytes > of space in memory > > What does happen when this warning is encountered? Is the query not > completed and I§m given only partial answer, or does it find another way to > finish the query? > > Thanks for your help, > Petr > > > On Tue, Jun 17, 2014 at 8:16 PM, K. John Wu <[email protected]> wrote: > >> Hi, Petr, >> >> Not sure if you are still interested in this, but here is what I have >> been puzzling over. >> >> There are many cases where an array_t<unsigned> is deleted first, but >> then copied into another array_t<unsigned>. One possible scenario for >> this is that a pointer to a bitvector is returned from a function that >> generated the bitvector as a local object. From the log file, I am >> not able to tell if this true or not. >> >> I have tried to run valgrind a few times, but not able to spot this >> problem so far. If you have a small test case for me, I am willing to >> look into it a little further. >> >> John >> >> >> >> On 4/7/14, 11:28 PM, Petr Velan wrote: >>> Hi John, >>> >>> did you by any chance have a bit of time to look into this matter? >>> >>> Thanks, >>> Petr >>> >>> >>> On Fri, Mar 21, 2014 at 1:25 PM, Petr Velan <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi John, >>> >>> I did the testing only with disabled mmap, no other modifications. >>> I'm now sending you an email with link to download the complete >>> log. I'm trying it on an older version from SVN revision 657. >>> Sadly, the newest is not compatible with the tested system. >>> >>> The log file is quite big 19GB, I compressed it to 2.5GB and the >>> link should be in another email shortly. I'm using >>> https://filesender.cesnet.cz/ service for the transfer. >>> >>> Let me know if there is anything more that I can do. >>> >>> Best, >>> Petr >>> >>> >>> >>> >>> On Thu, Mar 20, 2014 at 5:21 PM, K. John Wu <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi, Petr, >>> >>> FastBit is only tracking the memory usage of a few object >>> types such >>> as ibis::array_t and ibis::buffer. Since you are using a >>> compression >>> layer, I presume that you are also allocating considerable >>> amounts of >>> memory using malloc or new directly. This would mean FastBit >>> is only >>> tracking a part of the memory uses. If you turn up the >>> verboseness of >>> the FastBit code, say, ibis::util::setVerboseLevel(10), and >>> ship me >>> the log messages, I can take a look and see if I can spot >> anything >>> more useful to suggest. >>> >>> John >>> >>> >>> On 3/20/14, 6:35 AM, Petr Velan wrote: >>> > Hi John, >>> > >>> > I'm currently working on FastBit support for compressed >>> files. To >>> > simplify things a bit for a start, I have disabled the MMAP >>> support >>> > and forced FastBit to use open, read, write and seek >>> functions. Then I >>> > hooked into these functions and provided a compression >>> layer. The >>> > problem is that just disabling the MMAP causes FastBit to >>> use so much >>> > memory that OOM killer terminates the process for some >>> aggregation >>> > queries. >>> > >>> > Is there anything I can do to tell FastBit to limit the >>> amount of >>> > memory it can use? I thought that only half of the available >>> memory >>> > will be used by default. I've tested this issue on latest >>> SVN revision. >>> > >>> > Thank you in advance for any help, >>> > Petr Velan >>> > >>> > >>> > _______________________________________________ >>> > FastBit-users mailing list >>> > [email protected] >>> <mailto:[email protected]> >>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> > >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] <mailto: >> [email protected]> >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >>> >>> >>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
