Hi John,
I am using a program that is included with fastquery in the "examples"
directory called "buildIndex." It doesn't appear to have a -c option,
however:
fastquery-0.8.2.8/examples/buildIndex -f data-file-name [-i
index-file-name] [-g log-file] [-n variable-name] [-p variable-path] [-b
'<binning nbins=1000 />' (default unbinned)] [-r (force-rebuild-index)] [-v
verboseness] [-m fileFormat [HDF5(default), H5PART, NETCDF, PNETCDF]] [-l
mpi_subarray_size(default=100000)]
It builds index for a set of variables whose dataset location has
the prefix
variable-path and postfix variable-name.
Use option "-i" to specify the output file for storing indexes.
Otherwise, the indexes are written back to data file
"data-file-name".
Use option "-r" to enforce rebuild and replace the existing index.
Under parallel mode, use "-l" to set the subarray size for spitting
dataset.
Use option "-b" to specify the binning option to build the index.
The available binning option is defined and provided by the FastBit.
More information can be found at
http://crd.lbl.gov/~kewu/fastbit/doc/indexSpec.html.
Binning option is suggested to be used with large dataset to reduce
the size of built index.
Precision option is suggested to be used when the query involves
floating point numbers.
On Mon, Jul 28, 2014 at 11:48 PM, K. John Wu <[email protected]> wrote:
> Hi, Darryl,
>
> Thanks for your patience.
>
> What program are using using? One of our programs or your own
> program? With most of the example programs we provide, there is an
> option -c, which is for you to specify the configuration file.
>
> John
>
>
>
>
> On 7/28/14 3:50 PM, Darryl Reeves wrote:
> > My apologies. I should have continued reading the documentation. I found
> a
> > sample configuration file here:
> >
> > http://crd-legacy.lbl.gov/~kewu/fastbit/doc/dataLoading.html#samplerc
> >
> > Unfortunately, even when I set the value to 1000000Mb
> > (fileManager.maxBytes=1000000Mb), I still get the same error originally
> > reported.
> >
> >
> > On Mon, Jul 28, 2014 at 1:29 PM, Darryl Reeves <[email protected]>
> wrote:
> >
> >> Hi John,
> >>
> >> After reading your response on a different thread ("Out of memory
> without
> >> MMAP"), I'm thinking that increasing the fileManager cache might help me
> >> with this problem as well. From the online documentation, I was able to
> >> determine that I can set the cache size in a file named "ibis.rc" that
> is
> >> located in the same directory where I am running "buildIndex." What
> should
> >> the format of the configuration parameter be?
> >>
> >> I tried:
> >>
> >> fileManager.maxBytes = 5GB
> >>
> >> But this doesn't seem to make a difference. Can you provide an example
> >> configuration file?
> >>
> >> Thanks,
> >> Darryl
> >>
> >>
> >> On Fri, Jul 25, 2014 at 7:02 PM, Darryl Reeves <[email protected]>
> wrote:
> >>
> >>> Hi John,
> >>>
> >>> I just shared the file with you through my Google Drive account. Let me
> >>> know if you have any problems accessing the file.
> >>>
> >>> The command that I have tried to run is:
> >>>
> >>> fastquery-0.8.2.8/examples/buildIndex -f sample.2.h5 -n k
> >>>
> >>> Thanks,
> >>> Darryl
> >>>
> >>>
> >>> On Thu, Jul 24, 2014 at 10:59 PM, John Wu <[email protected]> wrote:
> >>>
> >>>> Hi, Darryl,
> >>>>
> >>>> Are you able to share the test data with us? It would be useful for
> us
> >>>> to reproduce the problem.
> >>>>
> >>>> If yes, please provide instructions on reproducing the problem.
> >>>>
> >>>> Thanks.
> >>>>
> >>>> K Wu
> >>>> On Jul 24, 2014 1:25 AM, "Darryl Reeves" <[email protected]> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I am attempting to index a file consisting of unsigned integer data
> >>>>> using the example program "buildIndex" included with fastquery. This
> >>>>> program works fine for the test data included with the source code.
> >>>>>
> >>>>> However, when I try to index my data, I receive the following memory
> >>>>> allocation error:
> >>>>>
> >>>>> Warning -- fileManager::storage::ctor unable to find 6,572,816,304
> >>>>> bytes of space in memory
> >>>>> terminate called after throwing an instance of 'ibis::bad_alloc'
> >>>>> what(): storage::ctor(memory) failed
> >>>>>
> >>>>> This error is revealed very early in the execution of the program. By
> >>>>> monitoring the memory being used, I can tell that only 58756 Kb have
> been
> >>>>> allocated by the program. The server where this is running has no
> shortage
> >>>>> of RAM available. I am willing to share the dataset that I am using
> but it
> >>>>> is 3.2 Gb, so it cannot be transferred as part of this message.
> >>>>>
> >>>>> Any help that can you can provide to figure out this issue is
> >>>>> appreciated.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> --
> >>>>> Darryl Reeves
> >>>>> Ph.D. Candidate
> >>>>> Mason Lab
> >>>>> Weill Cornell Medical College of Cornell University
> >>>>> Tri-Institutional Program in Computational Biology and Medicine
> >>>>>
> >>>>> _______________________________________________
> >>>>> FastBit-users mailing list
> >>>>> [email protected]
> >>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> FastBit-users mailing list
> >>>> [email protected]
> >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Darryl Reeves
> >>> Ph.D. Candidate
> >>> Mason Lab
> >>> Weill Cornell Medical College of Cornell University
> >>> Tri-Institutional Program in Computational Biology and Medicine
> >>>
> >>
> >>
> >>
> >> --
> >> Darryl Reeves
> >> Ph.D. Candidate
> >> Mason Lab
> >> Weill Cornell Medical College of Cornell University
> >> Tri-Institutional Program in Computational Biology and Medicine
> >>
> >
> >
> >
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected]
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>
--
Darryl Reeves
Ph.D. Candidate
Mason Lab
Weill Cornell Medical College of Cornell University
Tri-Institutional Program in Computational Biology and Medicine
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
