Hi, Amihay, Looks like you have run out of memory while building the index for shipping date and the cleanup function failed to do its job, which led to the query evaluation procedure to fail.
A way to work around this would be to build the index in shipping date before doing the query. You can try something like ibis -b "shipdate:<binning precision=3/>" John On 6/10/13 6:09 AM, amihay gonen wrote: > Hi , I'm trying to use fastbit ibis 64bit. > > I've allocate 5G to fastbit (use memory mapped file) > should be enough , but it still failed , > > What am i doing wrong ? > > see bellow for the output > > > c:\GitHub\fastbit\win\Release64\ibis > Options summary: batch mode, log level 3, skipping estimation > > resource::read -- parsing configuration file > "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt" > resource::read -- parsing configuration file > "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt" > fileManager initialization complete -- maxBytes=5242880000, > maxOpenFiles=60 > util::gatherParts -- examining C:\db\fastbit\xaa > Constructed a part named xaa > activeDir = "C:\db\fastbit\xaa" > 5000000 rows and 15 columns > util::gatherParts -- examining C:\db\fastbit\xab > Constructed a part named xab > activeDir = "C:\db\fastbit\xab" > 5000000 rows and 15 columns > Partitions[2]: > xaa > xab > Query[1]: > SELECT returnflag,linestatus,sum(quantity) as > sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg > antity) as avg_qty, avg(price) as avg_price, avg(discount) as > avg_disc, count(*) as count_order WHERE shipdate <= 904608000 ; > > liga -- constructed table T-xaa (a simple list of partitions: xaa and > xab) from a list of 2 data partitions, with 15 columns and 10000000 rows > tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) > as sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*( > ax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > avg(discount) as avg_disc, count(*) as count_order FROM T-xaa WHERE > shipdate > 904608000 ;" > selectClause::fillNames -- select clause internal details: > low-level expressions (names_[8], aggr_[8], atms_[8]): > 0: returnflag, returnflag > 1: linestatus, linestatus > 2: __2, SUM(quantity) > 3: __3, SUM((price * (1 - discount))) > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) > 5: __5, COUNT(*) > 6: __6, SUM(price) > 7: __7, SUM(discount) > high-level expressions (xnames_[9], xtms_[9]): > 0: returnflag, returnflag > 1: linestatus, linestatus > 2: sum_qty, SUM(quantity) > 3: sum_disc_price, SUM((price * (1 - discount))) > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax))) > 5: avg_qty, (SUM(quantity) / COUNT(*)) > 6: avg_price, (SUM(price) / COUNT(*)) > 7: avg_disc, (SUM(discount) / COUNT(*)) > 8: count_order, COUNT(*) > countQuery::setWhereClause accepted new query conditions "shipdate <= > 9.04608e+008" > Constructed in-memory data partition hW04N1 -- filter::sift2S(SELECT > returnflag,linestatu ... FROM 2 data partitions WHERE shipd ...) -- > with 8 col > s > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions > WHERE shipd ...) -- processing data partition xaa > countQuery assigned data partition xaa > countQuery::evaluate -- start timer ... > bin[xaa.shipdate]::ctor -- initialization completed with 2527 bins for > 5000000 rows from a storage object @ 0000000001F2C5F0 offset 8 > index::create -- the bin index for column xaa.shipdate was read from > C:\db\fastbit\xaa in 0.002 sec(CPU), 0.00239134 sec(elapsed) > countQuery::evaluate -- Select count(*) From xaa Where shipdate <= > 9.04608e+008 --> 4927856 > countQuery::evaluate -- duration: 0.009 sec(CPU), 0.00776298 sec(elapsed) > part[xaa]::calculate - starting to evaluate "(price * (1 - discount))" > with mask (4927856 out of 5000000) > part[xaa]::calculate -- evaluating (price * (1 - discount)) on 4927856 > records (total: 5000000) took 0.146679 sec elapsed time and produced > 4927856 > lues > bord[hW04N1]::append -- adding 4927856 elements to column __3 from > (price * (1 - discount)) > part[xaa]::calculate - starting to evaluate "((price * (1 - discount)) > * (1 + tax))" with mask (4927856 out of 5000000) > part[xaa]::calculate -- evaluating ((price * (1 - discount)) * (1 + > tax)) on 4927856 records (total: 5000000) took 0.222345 sec elapsed > time and pr > ced 4927856 values > bord[hW04N1]::append -- adding 4927856 elements to column __4 from > ((price * (1 - discount)) * (1 + tax)) > text[xaa.returnflag]::selectStrings -- duration: 0.597 sec(CPU), > 0.597355 sec(elapsed) > bundles::ctor created a colValues for "returnflag" as cols[1] with > size 4927856 > bundles::ctor created a colValues for "linestatus" as cols[2] with > size 4927856 > bundles::ctor created a colValues for "quantity" as cols[3] with size > 4927856 > bundles::ctor created a colValues for "(price * (1 - discount))" as > cols[4] with size 4927856 > bundles::ctor created a colValues for "((price * (1 - discount)) * (1 > + tax))" as cols[5] with size 4927856 > bundles::ctor created a colValues for "price" as cols[6] with size 4927856 > bundles::ctor created a colValues for "discount" as cols[7] with size > 4927856 > Constructed in-memory data partition ELH5k -- Select returnflag, > linestatus, SUM(quantity), SUM((price * (1 - discount))), SUM(((price > * (1 - disco > )) * (1 + tax))), COUNT(*), SUM(price), SUM(discount) From hW04N1 -- > with 4 rows and 8 columns > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions > WHERE shipd ...) -- processing data partition xab > countQuery changing data partition from xaa to xab > countQuery::evaluate -- start timer ... > column[xab.shipdate](INT)::index::mapValues -- generated histogram > (2526 distinct values) > fileManager::storage::ctor failed malloc(112), will attempt to free > some memory > fileManager::storage::ctor failed malloc(28672), will attempt to free > some memory > Warning -- column[shipdate]::bin::ctor encountered an exception, > cleaning up ... > Warning -- index::create(shipdate) received an std::exception -- bad > allocation > Warning -- index[xab.shipdate]::create failed to create an index of > type default > column[xab.shipdate]::evaluateRange(shipdate <= 9.04608e+008) received > a std::exception -- bad allocation > part::doScan -- evaluating shipdate <= 9.04608e+008 on 5000000 int > values (total: 5000000) took 0.0192468 sec elapsed time and produced > 4928205 hit > countQuery::evaluate -- Select count(*) From xab Where shipdate <= > 9.04608e+008 --> 4928205 > countQuery::evaluate -- duration: 0.523 sec(CPU), 0.523351 sec(elapsed) > part[xab]::calculate - starting to evaluate "(price * (1 - discount))" > with mask (4928205 out of 5000000) > clearing data partition ELH5k > clearing data partition hW04N1 > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions > WHERE shipd ...) -- duration: 3.897 sec(CPU), 3.89773 sec(elapsed) > Warning -- table::select absorbed a std::exception (bad allocation), > will return a nil pointer > tableSelect:: select(returnflag,linestatus,sum(quantity) as > sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as su > harge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount) > as avg_disc, count(*) as count_order , shipdate <= 904608000 ;) failed > on ta > T-xaa > c:\GitHub\fastbit\win\Release64\ibis -- total CPU time 3.913 s, total > elapsed time 3.9138 s > clearing data partition xaa > clearing data partition xab > Cleaning up the file manager > Total pages accessed through read(unistd.h) is estimated to be 29351 > fileManager::clear -- starting ... > --- Mon Jun 10 16:04:29 2013 > The number of memory mapped files is 1. (max = 60) > file name: C:\db\fastbit\xab\shipdate > storage @ 0000000001F2C410, 00000000027E0000, 1st 32 bits = 33345660, > 1st 64 bits = 3399cbd033345660 > file descriptor 00000000000005E0 > fmap handle 00000000000005E4 > base address 00000000027E0000 > mapped y opened at Mon Jun 10 16:04:27 2013 > last used at Mon Jun 10 16:04:27 2013 > # of bytes 20000000 # of past acc 1 # of active acc 0 > Size of all mapped files is 20,000,000 > > The number of files read into memory is 0 > > The total size of all files read into memory is 0 > Size of all named storages is 20,000,000 > Size of all unnamed storages is 0 > The total size of all named and unnamed storages is 20,000,000 > The prescribed maximum size is 5,242,880,000 > Number of pages accessed (recorded so far) is 29351 (page size = 8192) > > > > --- Mon Jun 10 16:04:29 2013 > The number of memory mapped files is 0. (max = 60) > Size of all mapped files is 0 > > The number of files read into memory is 0 > > The total size of all files read into memory is 0 > Size of all named storages is 0 > Size of all unnamed storages is 0 > The total size of all named and unnamed storages is 0 > The prescribed maximum size is 5,242,880,000 > Number of pages accessed (recorded so far) is 29351 (page size = 8192) > > > fileManager decommissioned > Started at 16:04:25.25 > Ran for 4000 ms > Press any key to continue . . . > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
