Hi, Anderson, Apparently, FastBit is using more memory than necessary in the test case. Unfortunately, this issue will require more extensive change to FastBit than we currently have time for. We will be put the item on our longer term to-do list. If you have the time to work on it, we would very much appreciate the contribution.
John PS: Just in case anyone might be interested in doing something here, we basically need to divide the aggregation operators into separable functions and non-separable functions. For separable functions, a lot less memory is needed. FastBit currently treats all aggregation operator as non-separable and therefore is using more memory than necessary. On 11/13/2010 11:23 AM, Anderson C. Carniel wrote: > Hi John, > > I own two data partitions with about 30 milion lines each. Each > partition has two columns: > > col0 of type double > col1 of type key > > When I do the query: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select sum(col0), col1 where col1 != ''" > > I giver an error that is shown at the end. But when I run the > following query: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select col1 where col1 != ''" > > Its run normally: > this result: > > col1 (with counts) > "EUROPE ", 6046909 > "ASIA ", 5970191 > "AFRICA ", 5965676 > "MIDDLE EAST ", 5995910 > "AMERICA ", 6014422 > col1 (with counts) > "EUROPE ", 6029149 > "ASIA ", 6090108 > "AMERICA ", 5902212 > "AFRICA ", 5989584 > "MIDDLE EAST ", 598205 > > this is the error: > > [r...@localhost ~]# ./ibis -d /tests/index0 -d /tests/index1 -v -o > /tests/result.csv -q "select sum(col0), col1 where col1 != ''" > > /opt/fastbit-ibis1.2.1/examples/.libs/lt-ibis: batch mode, log level 1 > > Completed construction of an part named index0 > 29993108 rows and 2 columns > Completed construction of an part named index1 > 29993106 rows and 2 columns > liga -- constructed table T-index0 (a simple list of partitions: > index0 and index1) from a list of 2 data partitions, with 2 columns > and 59986214 rows > tableSelect -- the number of hits is 59986214 > countQuery::evaluate -- Select count(*) From index0 Where ( ! > 0x9aa5f90) --> 29993108 > countQuery::evaluate -- duration: 0 sec(CPU), 0.0982819 sec(elapsed) > countQuery::evaluate -- Select count(*) From index1 Where ( ! > 0x9aa5f90) --> 29993106 > countQuery::evaluate -- duration: 0 sec(CPU), 0.000159979 sec(elapsed) > Warning -- table::select absorbed an exception, will return a nil pointer > > --- Fri Nov 12 15:13:29 2010 > The number of memory mapped files is 2. (max = 768) > file name: /tests/index0/col0 > storage @ 0x9aa75d0, 0xa9a90000, 1st 32 bits = 0, 1st 64 bits = > 4132445800000000 > file descriptor 3 > fmap size 239944864 > base address 0xa9a90000 > mapped y opened at Fri Nov 12 15:07:06 2010 last used at Fri Nov 12 > 15:07:06 2010 > # of bytes 239944864 # of past acc 1 # of active acc 0 > file name: /tests/index1/col0 > storage @ 0x42df8a58, 0x9b5bb000, 1st 32 bits = c0000000, 1st 64 bits > = 41507d12c0000000 > file descriptor 4 > fmap size 239944848 > base address 0x9b5bb000 > mapped y opened at Fri Nov 12 15:10:27 2010 last used at Fri Nov 12 > 15:10:27 2010 > # of bytes 239944848 # of past acc 1 # of active acc 0 > Size of all mapped files is 479,889,712 > > The number of files read into memory is 0 > > The total size of all files read into memory is 0 > Size of all named storages is 479,889,712 > Size of all unnamed storages is 480 > The total size of all named and unnamed storages is 479,890,192 > The prescribed maximum size is 1,568,624,640 > Number of pages accessed (recorded so far) is 117168 (page size = 4096) > > > tableSelect:: select(sum(col0), col1 , col1 != '') failed on table > T-index0 > /opt/fastbit-ibis1.2.1/examples/.libs/lt-ibis -- total CPU time > 81.6956 s, total elapsed time 383.965 s > > Thanks > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
