Hi, Amihay, Glad to see that you are getting it to work.
The function that recursively visit all subdirectories is using opendir which is not available on windows environment - at least not the way I am using them.. You will need to explicitly name each data directory on the command through the -d option or listing the data directories in a RC file. John On 5/23/13 7:43 AM, amihay gonen wrote: > I've tried to use partitions to over come the memory limits , but i > didn't find a way to tell ibis to scan several partition , I think it > is because I'm working on windows so gatherparts does'n work (opendir ) > > am i correct ? > > > On Thu, May 23, 2013 at 12:57 PM, amihay gonen <[email protected] > <mailto:[email protected]>> wrote: > > I've found out that I can increase ibis memory using by setting in > configruation file something like that : > fileManager.maxBytes=1648576000 ,but it still doesn't work well. > > I wander if I'll split the data into partitions , will it solved > the memory issues , as each partition will procsse spereatly ? > > Thanks, > Amihay > > > On Fri, May 17, 2013 at 10:14 AM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Amihay, > > The error message indicates that you are running out of > memory. Seems > like the intermediate answers you are getting requires too > much memory. > > John > > > On 5/16/13 9:01 AM, amihay gonen wrote: > > Hi I'm trying to query from 60M rows with 32bit bit ibis , > but it > > fails , I think it is releated to memory , but i can't > understand way . > > > > I've attached the output from a debug version . I hope you > can find > > the reason > > > > > > Thanks ,Amihay > > > > > > q.bat > > > > FastBit ibis > > Log messages started on Thu May 16 18:56:51 2013 > > > > c:\GitHub\fastbit\win\Debug32\ibis > > Options summary: batch mode, log level 5, skipping estimation > > > > fileManager::ctor found the physical memory size to be > 4186902528 bytes > > fileManager initialization complete -- maxBytes=1910734848, > > maxOpenFiles=60 > > DEBUG - WIN32 related macros > > NTDDI_VERSION=6010000 > > WINVER=600 > > HAVE_WIN_ATOMIC32 true > > HAVE_WIN_ATOMIC64 true > > util::gatherParts -- examining c:\db\fastbit\lineitem > > part::readMetaData -- opened > c:\db\fastbit\lineitem\-part.txt for reading > > part[lineitem]::readRIDs -- the file manager failed to read file > > "c:\db\fastbit\lineitem\-rids". There is no RIDs. > > Constructed a part named lineitem > > activeDir = "c:\db\fastbit\lineitem" > > part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d > > c:\db\fastbit\lineitem -b | -m > > > > orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen > > t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, > 16 columns > > Column list: > > comment: (STRING) > > commitdate: (INT) [1.79769e+308, -1.79769e+308] > > discount: (FLOAT) [1.79769e+308, -1.79769e+308] > > linenumber: (INT) [1.79769e+308, -1.79769e+308] > > linestatus: (KEY) [] > > orderkey: (INT) [1.79769e+308, -1.79769e+308] > > partkey: (INT) [1.79769e+308, -1.79769e+308] > > price: (FLOAT) [1.79769e+308, -1.79769e+308] > > quantity: (FLOAT) [1.79769e+308, -1.79769e+308] > > receipdate: (INT) [1.79769e+308, -1.79769e+308] > > returnflag: (STRING) > > shipdate: (INT) [1.79769e+308, -1.79769e+308] > > shipinstuct: (STRING) > > shipmode: (KEY) [] > > suppkey: (INT) [1.79769e+308, -1.79769e+308] > > tax: (FLOAT) [1.79769e+308, -1.79769e+308] > > > > Partition[1]: > > lineitem > > Query[1]: > > SELECT returnflag,linestatus,sum(quantity) as > > sum_qty,sum(price*(1-discount)) as > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > > avg(discount) as avg_disc, count(*) as count_order FROM > lineitem > > WHERE shipdate <= 904608000 ; > > > > liga -- constructed table T-lineitem (a simple list of > partition: > > lineitem) from a list of 1 data partition, with 16 columns and > > 59986052 rows > > tableSelect -- processing "SELECT > returnflag,linestatus,sum(quantity) > > as sum_qty,sum(price*(1-discount)) as > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > > avg(discount) as avg_disc, count(*) as count_order FROM > T-lineitem > > WHERE shipda > > te <= 904608000 ;" > > Constructing selectClause @ 0018C654 > > selectClause::fillNames -- select clause internal details: > > low-level expressions (names_[8], aggr_[8], atms_[8]): > > 0: returnflag, returnflag > > 1: linestatus, linestatus > > 2: __2, SUM(quantity) > > 3: __3, SUM((price * (1 - discount))) > > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) > > 5: __5, COUNT(*) > > 6: __6, SUM(price) > > 7: __7, SUM(discount) > > high-level expressions (xnames_[9], xtms_[9]): > > 0: returnflag, returnflag > > 1: linestatus, linestatus > > 2: sum_qty, SUM(quantity) > > 3: sum_disc_price, SUM((price * (1 - discount))) > > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + > tax))) > > 5: avg_qty, (SUM(quantity) / COUNT(*)) > > 6: avg_price, (SUM(price) / COUNT(*)) > > 7: avg_disc, (SUM(discount) / COUNT(*)) > > 8: count_order, COUNT(*) > > qExpr::simplify -- input expression shipdate <= 9.04608e+008 > > replace a compRange with a qRange shipdate <= 9.04608e+008 > > qExpr::simplify -- output expression (@050A5480) shipdate <= > 9.04608e+008 > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > partition > > WHERE shipda ...) -- start timer ... > > countQuery::setWhereClause accepted new query conditions > "shipdate <= > > 9.04608e+008" > > dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- > duration: > > 0.001 sec(CPU), 0.000323293 sec(elapsed) > > direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) > > finished reading index header with nrows=59986052 and > bits.size()=3 > > Constructed in-memory data partition RK2NP0 -- > filter::sift2(SELECT > > returnflag,linestatu ... FROM 1 data partition WHERE shipda > ...) -- > > with 8 columns > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > partition > > WHERE shipda ...) -- processing a select clause with 8 > terms, 2 of > > which are plain > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > partition > > WHERE shipda ...) -- processing data partition lineitem > > countQuery assigned data partition lineitem > > countQuery::evaluate -- start timer ... > > index::create -- attempt to read index type #0 from > > c:\db\fastbit\lineitem\shipdate.idx for column lineitem.shipdate > > bin[lineitem.shipdate]::ctor -- initialization completed > with 2527 > > bins for 59986052 rows from a storage object @ 050A6FA0 offset 8 > > index::create -- the bin index for column lineitem.shipdate > was read > > from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 > sec(elapsed) > > index (equality encoded, binned) for lineitem.shipdate > contains 2527 > > bitvectors for 59986052 objects > > There are a total 0 set bits out of 59986052 bits -- there are > > probably NULL values in column shipdate > > ... (2526 omitted) > > > > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > > 9.04597e+008:9.04597e+008) > > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > > 9.04597e+008:9.04597e+008) > > index -- sumBins(0, 2436) will operate on 91 out of 2527 > bitmaps using > > the combined option > > index -- sumBins operated on 91 bitmaps (6751400 in 7740160 > out) took > > 0.042 sec(CPU), 0.0425918 sec(elapsed). > > column[lineitem.shipdate]::evaluateRange(shipdate <= > 9.04608e+008, > > mask(59986052, 59986052)) completed with ierr = 1 > > countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, > > mask.cnt()=59986052) --> 59123858, ierr = 1 > > > > countQuery::evaluate -- Select count(*) From lineitem Where > shipdate > > <= 9.04608e+008 --> 59123858 > > countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 > sec(elapsed) > > bord[RK2NP0]::append -- to process 59123858 rows from partition > > lineitem, # of existing rows = 0 > > bord[RK2NP0]::append is to add 59123858 elements to column > "__2" from > > column "quantity" of partition lineitem > > column[quantity]::selectValuesT got 59123858 values > (59123858 wanted) > > from an in-memory version of file > c:\db\fastbit\lineitem\quantity as float > > column[lineitem.quantity](FLOAT)::selectFloats -- retrieving > 59123858 > > float values took 1.012 sec(CPU), 1.01197 sec(elapsed) > > part[lineitem]::calculate - starting to evaluate "(price * (1 - > > discount))" with mask (59123858 out of 59986052) > > fileManager::storage::ctor failed malloc(472990864), will > attempt to > > free some memory > > fileManager::unload -- to free up all unused space > > (totalBytes=1,187,726,660, maxBytes=1,910,734,848) > > fileManager::unload -- unloading all (2) inactive files > > fileManager::unload -- unloading file > > "c:\db\fastbit\lineitem\shipdate.idx" > > fileManager::unload -- unloading file > "c:\db\fastbit\lineitem\quantity" > > part[lineitem]::calculate -- evaluating (price * (1 - > discount)) on > > 59123858 records (total: 59986052) took 16.565 sec elapsed > time and > > produced 59123858 values > > bord[RK2NP0]::append -- adding 59123858 elements to column > __3 from > > (price * (1 - discount)) > > part[lineitem]::calculate - starting to evaluate "((price * (1 - > > discount)) * (1 + tax))" with mask (59123858 out of 59986052) > > fileManager::storage::ctor failed malloc(472990864), will > attempt to > > free some memory > > fileManager::unload -- to free up all unused space > > (totalBytes=1,433,610,620, maxBytes=1,910,734,848) > > fileManager::unload -- unloading all (2) inactive files > > fileManager::unload -- unloading file > "c:\db\fastbit\lineitem\discount" > > clearing data partition RK2NP0 > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > partition > > WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 > sec(elapsed) > > Freeing selectClause @ 0018C654 > > Warning -- table::select absorbed a std::exception (bad > allocation), > > will return a nil pointer > > --- Thu May 16 18:57:10 2013 > > The number of memory > > Press any key to continue . . . > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > <mailto:[email protected]> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
