I've tried to use partitions to over come the memory limits , but i didn't find a way to tell ibis to scan several partition , I think it is because I'm working on windows so gatherparts does'n work (opendir )
am i correct ? On Thu, May 23, 2013 at 12:57 PM, amihay gonen <[email protected]> wrote: > I've found out that I can increase ibis memory using by setting in > configruation file something like that : > fileManager.maxBytes=1648576000 ,but it still doesn't work well. > > I wander if I'll split the data into partitions , will it solved the > memory issues , as each partition will procsse spereatly ? > > Thanks, > Amihay > > > On Fri, May 17, 2013 at 10:14 AM, K. John Wu <[email protected]> wrote: > >> Hi, Amihay, >> >> The error message indicates that you are running out of memory. Seems >> like the intermediate answers you are getting requires too much memory. >> >> John >> >> >> On 5/16/13 9:01 AM, amihay gonen wrote: >> > Hi I'm trying to query from 60M rows with 32bit bit ibis , but it >> > fails , I think it is releated to memory , but i can't understand way . >> > >> > I've attached the output from a debug version . I hope you can find >> > the reason >> > >> > >> > Thanks ,Amihay >> > >> > >> > q.bat >> > >> > FastBit ibis >> > Log messages started on Thu May 16 18:56:51 2013 >> > >> > c:\GitHub\fastbit\win\Debug32\ibis >> > Options summary: batch mode, log level 5, skipping estimation >> > >> > fileManager::ctor found the physical memory size to be 4186902528 bytes >> > fileManager initialization complete -- maxBytes=1910734848, >> > maxOpenFiles=60 >> > DEBUG - WIN32 related macros >> > NTDDI_VERSION=6010000 >> > WINVER=600 >> > HAVE_WIN_ATOMIC32 true >> > HAVE_WIN_ATOMIC64 true >> > util::gatherParts -- examining c:\db\fastbit\lineitem >> > part::readMetaData -- opened c:\db\fastbit\lineitem\-part.txt for >> reading >> > part[lineitem]::readRIDs -- the file manager failed to read file >> > "c:\db\fastbit\lineitem\-rids". There is no RIDs. >> > Constructed a part named lineitem >> > activeDir = "c:\db\fastbit\lineitem" >> > part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d >> > c:\db\fastbit\lineitem -b | -m >> > >> orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen >> > t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, 16 columns >> > Column list: >> > comment: (STRING) >> > commitdate: (INT) [1.79769e+308, -1.79769e+308] >> > discount: (FLOAT) [1.79769e+308, -1.79769e+308] >> > linenumber: (INT) [1.79769e+308, -1.79769e+308] >> > linestatus: (KEY) [] >> > orderkey: (INT) [1.79769e+308, -1.79769e+308] >> > partkey: (INT) [1.79769e+308, -1.79769e+308] >> > price: (FLOAT) [1.79769e+308, -1.79769e+308] >> > quantity: (FLOAT) [1.79769e+308, -1.79769e+308] >> > receipdate: (INT) [1.79769e+308, -1.79769e+308] >> > returnflag: (STRING) >> > shipdate: (INT) [1.79769e+308, -1.79769e+308] >> > shipinstuct: (STRING) >> > shipmode: (KEY) [] >> > suppkey: (INT) [1.79769e+308, -1.79769e+308] >> > tax: (FLOAT) [1.79769e+308, -1.79769e+308] >> > >> > Partition[1]: >> > lineitem >> > Query[1]: >> > SELECT returnflag,linestatus,sum(quantity) as >> > sum_qty,sum(price*(1-discount)) as >> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as >> > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, >> > avg(discount) as avg_disc, count(*) as count_order FROM lineitem >> > WHERE shipdate <= 904608000 ; >> > >> > liga -- constructed table T-lineitem (a simple list of partition: >> > lineitem) from a list of 1 data partition, with 16 columns and >> > 59986052 rows >> > tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) >> > as sum_qty,sum(price*(1-discount)) as >> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as >> > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, >> > avg(discount) as avg_disc, count(*) as count_order FROM T-lineitem >> > WHERE shipda >> > te <= 904608000 ;" >> > Constructing selectClause @ 0018C654 >> > selectClause::fillNames -- select clause internal details: >> > low-level expressions (names_[8], aggr_[8], atms_[8]): >> > 0: returnflag, returnflag >> > 1: linestatus, linestatus >> > 2: __2, SUM(quantity) >> > 3: __3, SUM((price * (1 - discount))) >> > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) >> > 5: __5, COUNT(*) >> > 6: __6, SUM(price) >> > 7: __7, SUM(discount) >> > high-level expressions (xnames_[9], xtms_[9]): >> > 0: returnflag, returnflag >> > 1: linestatus, linestatus >> > 2: sum_qty, SUM(quantity) >> > 3: sum_disc_price, SUM((price * (1 - discount))) >> > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax))) >> > 5: avg_qty, (SUM(quantity) / COUNT(*)) >> > 6: avg_price, (SUM(price) / COUNT(*)) >> > 7: avg_disc, (SUM(discount) / COUNT(*)) >> > 8: count_order, COUNT(*) >> > qExpr::simplify -- input expression shipdate <= 9.04608e+008 >> > replace a compRange with a qRange shipdate <= 9.04608e+008 >> > qExpr::simplify -- output expression (@050A5480) shipdate <= >> 9.04608e+008 >> > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition >> > WHERE shipda ...) -- start timer ... >> > countQuery::setWhereClause accepted new query conditions "shipdate <= >> > 9.04608e+008" >> > dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- duration: >> > 0.001 sec(CPU), 0.000323293 sec(elapsed) >> > direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) >> > finished reading index header with nrows=59986052 and bits.size()=3 >> > Constructed in-memory data partition RK2NP0 -- filter::sift2(SELECT >> > returnflag,linestatu ... FROM 1 data partition WHERE shipda ...) -- >> > with 8 columns >> > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition >> > WHERE shipda ...) -- processing a select clause with 8 terms, 2 of >> > which are plain >> > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition >> > WHERE shipda ...) -- processing data partition lineitem >> > countQuery assigned data partition lineitem >> > countQuery::evaluate -- start timer ... >> > index::create -- attempt to read index type #0 from >> > c:\db\fastbit\lineitem\shipdate.idx for column lineitem.shipdate >> > bin[lineitem.shipdate]::ctor -- initialization completed with 2527 >> > bins for 59986052 rows from a storage object @ 050A6FA0 offset 8 >> > index::create -- the bin index for column lineitem.shipdate was read >> > from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 sec(elapsed) >> > index (equality encoded, binned) for lineitem.shipdate contains 2527 >> > bitvectors for 59986052 objects >> > There are a total 0 set bits out of 59986052 bits -- there are >> > probably NULL values in column shipdate >> > ... (2526 omitted) >> > >> > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= >> > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, >> > 9.04597e+008:9.04597e+008) >> > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= >> > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, >> > 9.04597e+008:9.04597e+008) >> > index -- sumBins(0, 2436) will operate on 91 out of 2527 bitmaps using >> > the combined option >> > index -- sumBins operated on 91 bitmaps (6751400 in 7740160 out) took >> > 0.042 sec(CPU), 0.0425918 sec(elapsed). >> > column[lineitem.shipdate]::evaluateRange(shipdate <= 9.04608e+008, >> > mask(59986052, 59986052)) completed with ierr = 1 >> > countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, >> > mask.cnt()=59986052) --> 59123858, ierr = 1 >> > >> > countQuery::evaluate -- Select count(*) From lineitem Where shipdate >> > <= 9.04608e+008 --> 59123858 >> > countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 sec(elapsed) >> > bord[RK2NP0]::append -- to process 59123858 rows from partition >> > lineitem, # of existing rows = 0 >> > bord[RK2NP0]::append is to add 59123858 elements to column "__2" from >> > column "quantity" of partition lineitem >> > column[quantity]::selectValuesT got 59123858 values (59123858 wanted) >> > from an in-memory version of file c:\db\fastbit\lineitem\quantity as >> float >> > column[lineitem.quantity](FLOAT)::selectFloats -- retrieving 59123858 >> > float values took 1.012 sec(CPU), 1.01197 sec(elapsed) >> > part[lineitem]::calculate - starting to evaluate "(price * (1 - >> > discount))" with mask (59123858 out of 59986052) >> > fileManager::storage::ctor failed malloc(472990864), will attempt to >> > free some memory >> > fileManager::unload -- to free up all unused space >> > (totalBytes=1,187,726,660, maxBytes=1,910,734,848) >> > fileManager::unload -- unloading all (2) inactive files >> > fileManager::unload -- unloading file >> > "c:\db\fastbit\lineitem\shipdate.idx" >> > fileManager::unload -- unloading file "c:\db\fastbit\lineitem\quantity" >> > part[lineitem]::calculate -- evaluating (price * (1 - discount)) on >> > 59123858 records (total: 59986052) took 16.565 sec elapsed time and >> > produced 59123858 values >> > bord[RK2NP0]::append -- adding 59123858 elements to column __3 from >> > (price * (1 - discount)) >> > part[lineitem]::calculate - starting to evaluate "((price * (1 - >> > discount)) * (1 + tax))" with mask (59123858 out of 59986052) >> > fileManager::storage::ctor failed malloc(472990864), will attempt to >> > free some memory >> > fileManager::unload -- to free up all unused space >> > (totalBytes=1,433,610,620, maxBytes=1,910,734,848) >> > fileManager::unload -- unloading all (2) inactive files >> > fileManager::unload -- unloading file "c:\db\fastbit\lineitem\discount" >> > clearing data partition RK2NP0 >> > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data partition >> > WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 sec(elapsed) >> > Freeing selectClause @ 0018C654 >> > Warning -- table::select absorbed a std::exception (bad allocation), >> > will return a nil pointer >> > --- Thu May 16 18:57:10 2013 >> > The number of memory >> > Press any key to continue . . . >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > FastBit-users mailing list >> > [email protected] >> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > >> > >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
