Hi I'm trying to query from 60M rows with 32bit bit ibis , but it fails , I think it is releated to memory , but i can't understand way .
I've attached the output from a debug version . I hope you can find the reason Thanks ,Amihay q.bat FastBit ibis Log messages started on Thu May 16 18:56:51 2013 c:\GitHub\fastbit\win\Debug32\ibis Options summary: batch mode, log level 5, skipping estimation fileManager::ctor found the physical memory size to be 4186902528 bytes fileManager initialization complete -- maxBytes=1910734848, maxOpenFiles=60 DEBUG - WIN32 related macros NTDDI_VERSION=6010000 WINVER=600 HAVE_WIN_ATOMIC32 true HAVE_WIN_ATOMIC64 true util::gatherParts -- examining c:\db\fastbit\lineitem part::readMetaData -- opened c:\db\fastbit\lineitem\-part.txt for reading part[lineitem]::readRIDs -- the file manager failed to read file "c:\db\fastbit\lineitem\-rids". There is no RIDs. Constructed a part named lineitem activeDir = "c:\db\fastbit\lineitem" part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d c:\db\fastbit\lineitem -b | -m orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, 16 columns Column list: comment: (STRING) commitdate: (INT) [1.79769e+308, -1.79769e+308] discount: (FLOAT) [1.79769e+308, -1.79769e+308] linenumber: (INT) [1.79769e+308, -1.79769e+308] linestatus: (KEY) [] orderkey: (INT) [1.79769e+308, -1.79769e+308] partkey: (INT) [1.79769e+308, -1.79769e+308] price: (FLOAT) [1.79769e+308, -1.79769e+308] quantity: (FLOAT) [1.79769e+308, -1.79769e+308] receipdate: (INT) [1.79769e+308, -1.79769e+308] returnflag: (STRING) shipdate: (INT) [1.79769e+308, -1.79769e+308] shipinstuct: (STRING) shipmode: (KEY) [] suppkey: (INT) [1.79769e+308, -1.79769e+308] tax: (FLOAT) [1.79769e+308, -1.79769e+308] Partition[1]: lineitem Query[1]: SELECT returnflag,linestatus,sum(quantity) as sum_qty,sum(price*(1-discount)) as sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount) as avg_disc, count(*) as count_order FROM lineitem WHERE shipdate <= 904608000 ; liga -- constructed table T-lineitem (a simple list of partition: lineitem) from a list of 1 data partition, with 16 columns and 59986052 rows tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) as sum_qty,sum(price*(1-discount)) as sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount) as avg_disc, count(*) as count_order FROM T-lineitem WHERE shipda te <= 904608000 ;" Constructing selectClause @ 0018C654 selectClause::fillNames -- select clause internal details: low-level expressions (names_[8], aggr_[8], atms_[8]): 0: returnflag, returnflag 1: linestatus, linestatus 2: __2, SUM(quantity) 3: __3, SUM((price * (1 - discount))) 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) 5: __5, COUNT(*) 6: __6, SUM(price) 7: __7, SUM(discount) high-level expressions (xnames_[9], xtms_[9]): 0: returnflag, returnflag 1: linestatus, linestatus 2: sum_qty, SUM(quantity) 3: sum_disc_price, SUM((price * (1 - discount))) 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax))) 5: avg_qty, (SUM(quantity) / COUNT(*)) 6: avg_price, (SUM(price) / COUNT(*)) 7: avg_disc, (SUM(discount) / COUNT(*)) 8: count_order, COUNT(*) qExpr::simplify -- input expression shipdate <= 9.04608e+008 replace a compRange with a qRange shipdate <= 9.04608e+008 qExpr::simplify -- output expression (@050A5480) shipdate <= 9.04608e+008 filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- start timer ... countQuery::setWhereClause accepted new query conditions "shipdate <= 9.04608e+008" dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- duration: 0.001 sec(CPU), 0.000323293 sec(elapsed) direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) finished reading index header with nrows=59986052 and bits.size()=3 Constructed in-memory data partition RK2NP0 -- filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- with 8 columns filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- processing a select clause with 8 terms, 2 of which are plain filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- processing data partition lineitem countQuery assigned data partition lineitem countQuery::evaluate -- start timer ... index::create -- attempt to read index type #0 from c:\db\fastbit\lineitem\shipdate.idx for column lineitem.shipdate bin[lineitem.shipdate]::ctor -- initialization completed with 2527 bins for 59986052 rows from a storage object @ 050A6FA0 offset 8 index::create -- the bin index for column lineitem.shipdate was read from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 sec(elapsed) index (equality encoded, binned) for lineitem.shipdate contains 2527 bitvectors for 59986052 objects There are a total 0 set bits out of 59986052 bits -- there are probably NULL values in column shipdate ... (2526 omitted) column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, 9.04597e+008:9.04597e+008) column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, 9.04597e+008:9.04597e+008) index -- sumBins(0, 2436) will operate on 91 out of 2527 bitmaps using the combined option index -- sumBins operated on 91 bitmaps (6751400 in 7740160 out) took 0.042 sec(CPU), 0.0425918 sec(elapsed). column[lineitem.shipdate]::evaluateRange(shipdate <= 9.04608e+008, mask(59986052, 59986052)) completed with ierr = 1 countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, mask.cnt()=59986052) --> 59123858, ierr = 1 countQuery::evaluate -- Select count(*) From lineitem Where shipdate <= 9.04608e+008 --> 59123858 countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 sec(elapsed) bord[RK2NP0]::append -- to process 59123858 rows from partition lineitem, # of existing rows = 0 bord[RK2NP0]::append is to add 59123858 elements to column "__2" from column "quantity" of partition lineitem column[quantity]::selectValuesT got 59123858 values (59123858 wanted) from an in-memory version of file c:\db\fastbit\lineitem\quantity as float column[lineitem.quantity](FLOAT)::selectFloats -- retrieving 59123858 float values took 1.012 sec(CPU), 1.01197 sec(elapsed) part[lineitem]::calculate - starting to evaluate "(price * (1 - discount))" with mask (59123858 out of 59986052) fileManager::storage::ctor failed malloc(472990864), will attempt to free some memory fileManager::unload -- to free up all unused space (totalBytes=1,187,726,660, maxBytes=1,910,734,848) fileManager::unload -- unloading all (2) inactive files fileManager::unload -- unloading file "c:\db\fastbit\lineitem\shipdate.idx" fileManager::unload -- unloading file "c:\db\fastbit\lineitem\quantity" part[lineitem]::calculate -- evaluating (price * (1 - discount)) on 59123858 records (total: 59986052) took 16.565 sec elapsed time and produced 59123858 values bord[RK2NP0]::append -- adding 59123858 elements to column __3 from (price * (1 - discount)) part[lineitem]::calculate - starting to evaluate "((price * (1 - discount)) * (1 + tax))" with mask (59123858 out of 59986052) fileManager::storage::ctor failed malloc(472990864), will attempt to free some memory fileManager::unload -- to free up all unused space (totalBytes=1,433,610,620, maxBytes=1,910,734,848) fileManager::unload -- unloading all (2) inactive files fileManager::unload -- unloading file "c:\db\fastbit\lineitem\discount" clearing data partition RK2NP0 filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 sec(elapsed) Freeing selectClause @ 0018C654 Warning -- table::select absorbed a std::exception (bad allocation), will return a nil pointer --- Thu May 16 18:57:10 2013 The number of memory Press any key to continue . . .
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
