Hi, Amihay, The error message indicates that you are running out of memory. Seems like the intermediate answers you are getting requires too much memory.
John On 5/16/13 9:01 AM, amihay gonen wrote: > Hi I'm trying to query from 60M rows with 32bit bit ibis , but it > fails , I think it is releated to memory , but i can't understand way . > > I've attached the output from a debug version . I hope you can find > the reason > > > Thanks ,Amihay > > > q.bat > > FastBit ibis > Log messages started on Thu May 16 18:56:51 2013 > > c:\GitHub\fastbit\win\Debug32\ibis > Options summary: batch mode, log level 5, skipping estimation > > fileManager::ctor found the physical memory size to be 4186902528 bytes > fileManager initialization complete -- maxBytes=1910734848, > maxOpenFiles=60 > DEBUG - WIN32 related macros > NTDDI_VERSION=6010000 > WINVER=600 > HAVE_WIN_ATOMIC32 true > HAVE_WIN_ATOMIC64 true > util::gatherParts -- examining c:\db\fastbit\lineitem > part::readMetaData -- opened c:\db\fastbit\lineitem\-part.txt for reading > part[lineitem]::readRIDs -- the file manager failed to read file > "c:\db\fastbit\lineitem\-rids". There is no RIDs. > Constructed a part named lineitem > activeDir = "c:\db\fastbit\lineitem" > part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d > c:\db\fastbit\lineitem -b | -m > orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen > t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, 16 columns > Column list: > comment: (STRING) > commitdate: (INT) [1.79769e+308, -1.79769e+308] > discount: (FLOAT) [1.79769e+308, -1.79769e+308] > linenumber: (INT) [1.79769e+308, -1.79769e+308] > linestatus: (KEY) [] > orderkey: (INT) [1.79769e+308, -1.79769e+308] > partkey: (INT) [1.79769e+308, -1.79769e+308] > price: (FLOAT) [1.79769e+308, -1.79769e+308] > quantity: (FLOAT) [1.79769e+308, -1.79769e+308] > receipdate: (INT) [1.79769e+308, -1.79769e+308] > returnflag: (STRING) > shipdate: (INT) [1.79769e+308, -1.79769e+308] > shipinstuct: (STRING) > shipmode: (KEY) [] > suppkey: (INT) [1.79769e+308, -1.79769e+308] > tax: (FLOAT) [1.79769e+308, -1.79769e+308] > > Partition[1]: > lineitem > Query[1]: > SELECT returnflag,linestatus,sum(quantity) as > sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > avg(discount) as avg_disc, count(*) as count_order FROM lineitem > WHERE shipdate <= 904608000 ; > > liga -- constructed table T-lineitem (a simple list of partition: > lineitem) from a list of 1 data partition, with 16 columns and > 59986052 rows > tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) > as sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > avg(discount) as avg_disc, count(*) as count_order FROM T-lineitem > WHERE shipda > te <= 904608000 ;" > Constructing selectClause @ 0018C654 > selectClause::fillNames -- select clause internal details: > low-level expressions (names_[8], aggr_[8], atms_[8]): > 0: returnflag, returnflag > 1: linestatus, linestatus > 2: __2, SUM(quantity) > 3: __3, SUM((price * (1 - discount))) > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) > 5: __5, COUNT(*) > 6: __6, SUM(price) > 7: __7, SUM(discount) > high-level expressions (xnames_[9], xtms_[9]): > 0: returnflag, returnflag > 1: linestatus, linestatus > 2: sum_qty, SUM(quantity) > 3: sum_disc_price, SUM((price * (1 - discount))) > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax))) > 5: avg_qty, (SUM(quantity) / COUNT(*)) > 6: avg_price, (SUM(price) / COUNT(*)) > 7: avg_disc, (SUM(discount) / COUNT(*)) > 8: count_order, COUNT(*) > qExpr::simplify -- input expression shipdate <= 9.04608e+008 > replace a compRange with a qRange shipdate <= 9.04608e+008 > qExpr::simplify -- output expression (@050A5480) shipdate <= 9.04608e+008 > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition > WHERE shipda ...) -- start timer ... > countQuery::setWhereClause accepted new query conditions "shipdate <= > 9.04608e+008" > dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- duration: > 0.001 sec(CPU), 0.000323293 sec(elapsed) > direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) > finished reading index header with nrows=59986052 and bits.size()=3 > Constructed in-memory data partition RK2NP0 -- filter::sift2(SELECT > returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- > with 8 columns > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition > WHERE shipda ...) -- processing a select clause with 8 terms, 2 of > which are plain > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition > WHERE shipda ...) -- processing data partition lineitem > countQuery assigned data partition lineitem > countQuery::evaluate -- start timer ... > index::create -- attempt to read index type #0 from > c:\db\fastbit\lineitem\shipdate.idx for column lineitem.shipdate > bin[lineitem.shipdate]::ctor -- initialization completed with 2527 > bins for 59986052 rows from a storage object @ 050A6FA0 offset 8 > index::create -- the bin index for column lineitem.shipdate was read > from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 sec(elapsed) > index (equality encoded, binned) for lineitem.shipdate contains 2527 > bitvectors for 59986052 objects > There are a total 0 set bits out of 59986052 bits -- there are > probably NULL values in column shipdate > ... (2526 omitted) > > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > 9.04597e+008:9.04597e+008) > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > 9.04597e+008:9.04597e+008) > index -- sumBins(0, 2436) will operate on 91 out of 2527 bitmaps using > the combined option > index -- sumBins operated on 91 bitmaps (6751400 in 7740160 out) took > 0.042 sec(CPU), 0.0425918 sec(elapsed). > column[lineitem.shipdate]::evaluateRange(shipdate <= 9.04608e+008, > mask(59986052, 59986052)) completed with ierr = 1 > countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, > mask.cnt()=59986052) --> 59123858, ierr = 1 > > countQuery::evaluate -- Select count(*) From lineitem Where shipdate > <= 9.04608e+008 --> 59123858 > countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 sec(elapsed) > bord[RK2NP0]::append -- to process 59123858 rows from partition > lineitem, # of existing rows = 0 > bord[RK2NP0]::append is to add 59123858 elements to column "__2" from > column "quantity" of partition lineitem > column[quantity]::selectValuesT got 59123858 values (59123858 wanted) > from an in-memory version of file c:\db\fastbit\lineitem\quantity as float > column[lineitem.quantity](FLOAT)::selectFloats -- retrieving 59123858 > float values took 1.012 sec(CPU), 1.01197 sec(elapsed) > part[lineitem]::calculate - starting to evaluate "(price * (1 - > discount))" with mask (59123858 out of 59986052) > fileManager::storage::ctor failed malloc(472990864), will attempt to > free some memory > fileManager::unload -- to free up all unused space > (totalBytes=1,187,726,660, maxBytes=1,910,734,848) > fileManager::unload -- unloading all (2) inactive files > fileManager::unload -- unloading file > "c:\db\fastbit\lineitem\shipdate.idx" > fileManager::unload -- unloading file "c:\db\fastbit\lineitem\quantity" > part[lineitem]::calculate -- evaluating (price * (1 - discount)) on > 59123858 records (total: 59986052) took 16.565 sec elapsed time and > produced 59123858 values > bord[RK2NP0]::append -- adding 59123858 elements to column __3 from > (price * (1 - discount)) > part[lineitem]::calculate - starting to evaluate "((price * (1 - > discount)) * (1 + tax))" with mask (59123858 out of 59986052) > fileManager::storage::ctor failed malloc(472990864), will attempt to > free some memory > fileManager::unload -- to free up all unused space > (totalBytes=1,433,610,620, maxBytes=1,910,734,848) > fileManager::unload -- unloading all (2) inactive files > fileManager::unload -- unloading file "c:\db\fastbit\lineitem\discount" > clearing data partition RK2NP0 > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition > WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 sec(elapsed) > Freeing selectClause @ 0018C654 > Warning -- table::select absorbed a std::exception (bad allocation), > will return a nil pointer > --- Thu May 16 18:57:10 2013 > The number of memory > Press any key to continue . . . > > > > > > > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
