Hi, Darryl, Ahh, the usage note from that program happens to be missing the -c option. You should be able to use '-c ibis.rc'. One way to know whether you have set maxBytes to the value you want is to specify '-v 2' on the same command line and then examine the output message for line with
fileManager initialization complete -- maxBytes=dddd Hope this helps. John On 7/28/14 8:53 PM, Darryl Reeves wrote: > Hi John, > > I am using a program that is included with fastquery in the "examples" > directory called "buildIndex." It doesn't appear to have a -c option, > however: > > > > fastquery-0.8.2.8/examples/buildIndex -f data-file-name [-i > index-file-name] [-g log-file] [-n variable-name] [-p variable-path] [-b > '<binning nbins=1000 />' (default unbinned)] [-r (force-rebuild-index)] [-v > verboseness] [-m fileFormat [HDF5(default), H5PART, NETCDF, PNETCDF]] [-l > mpi_subarray_size(default=100000)] > It builds index for a set of variables whose dataset location has > the prefix > variable-path and postfix variable-name. > > Use option "-i" to specify the output file for storing indexes. > Otherwise, the indexes are written back to data file > "data-file-name". > > Use option "-r" to enforce rebuild and replace the existing index. > > Under parallel mode, use "-l" to set the subarray size for spitting > dataset. > > Use option "-b" to specify the binning option to build the index. > The available binning option is defined and provided by the FastBit. > More information can be found at > http://crd.lbl.gov/~kewu/fastbit/doc/indexSpec.html. > Binning option is suggested to be used with large dataset to reduce > the size of built index. > Precision option is suggested to be used when the query involves > floating point numbers. > > > On Mon, Jul 28, 2014 at 11:48 PM, K. John Wu <[email protected]> wrote: > >> Hi, Darryl, >> >> Thanks for your patience. >> >> What program are using using? One of our programs or your own >> program? With most of the example programs we provide, there is an >> option -c, which is for you to specify the configuration file. >> >> John >> >> >> >> >> On 7/28/14 3:50 PM, Darryl Reeves wrote: >>> My apologies. I should have continued reading the documentation. I found >> a >>> sample configuration file here: >>> >>> http://crd-legacy.lbl.gov/~kewu/fastbit/doc/dataLoading.html#samplerc >>> >>> Unfortunately, even when I set the value to 1000000Mb >>> (fileManager.maxBytes=1000000Mb), I still get the same error originally >>> reported. >>> >>> >>> On Mon, Jul 28, 2014 at 1:29 PM, Darryl Reeves <[email protected]> >> wrote: >>> >>>> Hi John, >>>> >>>> After reading your response on a different thread ("Out of memory >> without >>>> MMAP"), I'm thinking that increasing the fileManager cache might help me >>>> with this problem as well. From the online documentation, I was able to >>>> determine that I can set the cache size in a file named "ibis.rc" that >> is >>>> located in the same directory where I am running "buildIndex." What >> should >>>> the format of the configuration parameter be? >>>> >>>> I tried: >>>> >>>> fileManager.maxBytes = 5GB >>>> >>>> But this doesn't seem to make a difference. Can you provide an example >>>> configuration file? >>>> >>>> Thanks, >>>> Darryl >>>> >>>> >>>> On Fri, Jul 25, 2014 at 7:02 PM, Darryl Reeves <[email protected]> >> wrote: >>>> >>>>> Hi John, >>>>> >>>>> I just shared the file with you through my Google Drive account. Let me >>>>> know if you have any problems accessing the file. >>>>> >>>>> The command that I have tried to run is: >>>>> >>>>> fastquery-0.8.2.8/examples/buildIndex -f sample.2.h5 -n k >>>>> >>>>> Thanks, >>>>> Darryl >>>>> >>>>> >>>>> On Thu, Jul 24, 2014 at 10:59 PM, John Wu <[email protected]> wrote: >>>>> >>>>>> Hi, Darryl, >>>>>> >>>>>> Are you able to share the test data with us? It would be useful for >> us >>>>>> to reproduce the problem. >>>>>> >>>>>> If yes, please provide instructions on reproducing the problem. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> K Wu >>>>>> On Jul 24, 2014 1:25 AM, "Darryl Reeves" <[email protected]> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am attempting to index a file consisting of unsigned integer data >>>>>>> using the example program "buildIndex" included with fastquery. This >>>>>>> program works fine for the test data included with the source code. >>>>>>> >>>>>>> However, when I try to index my data, I receive the following memory >>>>>>> allocation error: >>>>>>> >>>>>>> Warning -- fileManager::storage::ctor unable to find 6,572,816,304 >>>>>>> bytes of space in memory >>>>>>> terminate called after throwing an instance of 'ibis::bad_alloc' >>>>>>> what(): storage::ctor(memory) failed >>>>>>> >>>>>>> This error is revealed very early in the execution of the program. By >>>>>>> monitoring the memory being used, I can tell that only 58756 Kb have >> been >>>>>>> allocated by the program. The server where this is running has no >> shortage >>>>>>> of RAM available. I am willing to share the dataset that I am using >> but it >>>>>>> is 3.2 Gb, so it cannot be transferred as part of this message. >>>>>>> >>>>>>> Any help that can you can provide to figure out this issue is >>>>>>> appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -- >>>>>>> Darryl Reeves >>>>>>> Ph.D. Candidate >>>>>>> Mason Lab >>>>>>> Weill Cornell Medical College of Cornell University >>>>>>> Tri-Institutional Program in Computational Biology and Medicine >>>>>>> >>>>>>> _______________________________________________ >>>>>>> FastBit-users mailing list >>>>>>> [email protected] >>>>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> FastBit-users mailing list >>>>>> [email protected] >>>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Darryl Reeves >>>>> Ph.D. Candidate >>>>> Mason Lab >>>>> Weill Cornell Medical College of Cornell University >>>>> Tri-Institutional Program in Computational Biology and Medicine >>>>> >>>> >>>> >>>> >>>> -- >>>> Darryl Reeves >>>> Ph.D. Candidate >>>> Mason Lab >>>> Weill Cornell Medical College of Cornell University >>>> Tri-Institutional Program in Computational Biology and Medicine >>>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
