Hi, Anderson, I have tried similar queries on two small (much too small, I guess) data partitions and things seem to work fine. My guess is that FastBit is making unwise use of memory somewhere and causing it to require about 1.5GB memory (in 4 pieces: 240MB for file col0 from partition 1, and 240MB for file col0 from partition 2, 480MB for storing all the values together in one array, and 480MB for an unintended second copy). If you are on a machine with 2GB of memory and the default memory cache size for FastBit is 1GB (half of 2GB), then the program will encounter a problem while attempting to allocate the second copy. We will try to verify whether this speculation is true and how to fix it.
In the mean time, you should be able to process queries that do not select all the records -- hopefully, most of the queries will select a lot less than 60 million records and therefore can be processed without any problem.. John On 11/13/2010 11:23 AM, Anderson C. Carniel wrote: > Hi John, > > I own two data partitions with about 30 milion lines each. Each > partition has two columns: > > col0 of type double > col1 of type key > > When I do the query: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select sum(col0), col1 where col1 != ''" > > I giver an error that is shown at the end. But when I run the > following query: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select col1 where col1 != ''" > > Its run normally: > this result: > > col1 (with counts) > "EUROPE ", 6046909 > "ASIA ", 5970191 > "AFRICA ", 5965676 > "MIDDLE EAST ", 5995910 > "AMERICA ", 6014422 > col1 (with counts) > "EUROPE ", 6029149 > "ASIA ", 6090108 > "AMERICA ", 5902212 > "AFRICA ", 5989584 > "MIDDLE EAST ", 598205 > > this is the error: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select sum(col0), col1 where col1 != ''" > > /opt/fastbit-ibis1.2.1/examples/.libs/lt-ibis: batch mode, log level 1 > > Completed construction of an part named index0 > 29993108 rows and 2 columns > Completed construction of an part named index1 > 29993106 rows and 2 columns > liga -- constructed table T-index0 (a simple list of partitions: > index0 and index1) from a list of 2 data partitions, with 2 columns > and 59986214 rows > tableSelect -- the number of hits is 59986214 > countQuery::evaluate -- Select count(*) From index0 Where ( ! > 0x9aa5f90) --> 29993108 > countQuery::evaluate -- duration: 0 sec(CPU), 0.0982819 sec(elapsed) > countQuery::evaluate -- Select count(*) From index1 Where ( ! > 0x9aa5f90) --> 29993106 > countQuery::evaluate -- duration: 0 sec(CPU), 0.000159979 sec(elapsed) > Warning -- table::select absorbed an exception, will return a nil pointer > > --- Fri Nov 12 15:13:29 2010 > The number of memory mapped files is 2. (max = 768) > file name: /tests/index0/col0 > storage @ 0x9aa75d0, 0xa9a90000, 1st 32 bits = 0, 1st 64 bits = > 4132445800000000 > file descriptor 3 > fmap size 239944864 > base address 0xa9a90000 > mapped y opened at Fri Nov 12 15:07:06 2010 last used at Fri Nov 12 > 15:07:06 2010 > # of bytes 239944864 # of past acc 1 # of active acc 0 > file name: /tests/index1/col0 > storage @ 0x42df8a58, 0x9b5bb000, 1st 32 bits = c0000000, 1st 64 bits > = 41507d12c0000000 > file descriptor 4 > fmap size 239944848 > base address 0x9b5bb000 > mapped y opened at Fri Nov 12 15:10:27 2010 last used at Fri Nov 12 > 15:10:27 2010 > # of bytes 239944848 # of past acc 1 # of active acc 0 > Size of all mapped files is 479,889,712 > > The number of files read into memory is 0 > > The total size of all files read into memory is 0 > Size of all named storages is 479,889,712 > Size of all unnamed storages is 480 > The total size of all named and unnamed storages is 479,890,192 > The prescribed maximum size is 1,568,624,640 > Number of pages accessed (recorded so far) is 117168 (page size = 4096) > > > tableSelect:: select(sum(col0), col1 , col1 != '') failed on table > T-index0 > /opt/fastbit-ibis1.2.1/examples/.libs/lt-ibis -- total CPU time > 81.6956 s, total elapsed time 383.965 s > > Thanks > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
