Hi John ,thanks your suggesting. This was I trying in the first place , but I got the following error, which i don't understan why:
C:\Users\agonen>C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\q.bat FastBit ibis Log messages started on Fri May 24 09:09:07 2013 c:\GitHub\fastbit\win\Release32\ibis Options summary: batch mode, log level 3, skipping estimation resource::read -- parsing configuration file "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt" fileManager initialization complete -- maxBytes=734003200, maxOpenFiles=60 util::gatherParts -- examining C:\db\fastbit\xaa Constructed a part named lineitem activeDir = "C:\db\fastbit\xaa" 1000 rows and 15 columns util::gatherParts -- examining C:\db\fastbit\xab Constructed a part named lineitem activeDir = "C:\db\fastbit\xab" 1000 rows and 15 columns *Warning -- util::gatherParts finds the data partition in C:\db\fastbit\xab to have exactly the same name, number of rows, number of columns, and time stamp as the one in C:\db\fastbit\xaa already in memory, discards the new one and keeps the old one* clearing data partition lineitem Partition[1]: lineitem Query[1]: SELECT returnflag,linestatus,sum(quantity) as sum_qty,sum(price*(1-discount)) as sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount) as avg_disc, count(*) as count_order FROM lineitem WHERE shipdate <= 904608000 ; liga -- constructed table T-lineitem (a simple list of partition: lineitem) from a list of 1 data partition, with 15 columns and 1000 rows tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) as sum_qty,sum(price*(1-discount)) as sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount) as avg_disc, count(*) as count_order FROM T-lineitem WHERE shipda te <= 904608000 ;" selectClause::fillNames -- select clause internal details: low-level expressions (names_[8], aggr_[8], atms_[8]): 0: returnflag, returnflag 1: linestatus, linestatus 2: __2, SUM(quantity) 3: __3, SUM((price * (1 - discount))) 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) 5: __5, COUNT(*) 6: __6, SUM(price) On Fri, May 24, 2013 at 1:15 AM, K. John Wu <[email protected]> wrote: > Hi, Amihay, > > Glad to see that you are getting it to work. > > The function that recursively visit all subdirectories is using > opendir which is not available on windows environment - at least not > the way I am using them.. You will need to explicitly name each data > directory on the command through the -d option or listing the data > directories in a RC file. > > John > > > On 5/23/13 7:43 AM, amihay gonen wrote: > > I've tried to use partitions to over come the memory limits , but i > > didn't find a way to tell ibis to scan several partition , I think it > > is because I'm working on windows so gatherparts does'n work (opendir ) > > > > am i correct ? > > > > > > On Thu, May 23, 2013 at 12:57 PM, amihay gonen <[email protected] > > <mailto:[email protected]>> wrote: > > > > I've found out that I can increase ibis memory using by setting in > > configruation file something like that : > > fileManager.maxBytes=1648576000 ,but it still doesn't work well. > > > > I wander if I'll split the data into partitions , will it solved > > the memory issues , as each partition will procsse spereatly ? > > > > Thanks, > > Amihay > > > > > > On Fri, May 17, 2013 at 10:14 AM, K. John Wu <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, Amihay, > > > > The error message indicates that you are running out of > > memory. Seems > > like the intermediate answers you are getting requires too > > much memory. > > > > John > > > > > > On 5/16/13 9:01 AM, amihay gonen wrote: > > > Hi I'm trying to query from 60M rows with 32bit bit ibis , > > but it > > > fails , I think it is releated to memory , but i can't > > understand way . > > > > > > I've attached the output from a debug version . I hope you > > can find > > > the reason > > > > > > > > > Thanks ,Amihay > > > > > > > > > q.bat > > > > > > FastBit ibis > > > Log messages started on Thu May 16 18:56:51 2013 > > > > > > c:\GitHub\fastbit\win\Debug32\ibis > > > Options summary: batch mode, log level 5, skipping estimation > > > > > > fileManager::ctor found the physical memory size to be > > 4186902528 bytes > > > fileManager initialization complete -- maxBytes=1910734848, > > > maxOpenFiles=60 > > > DEBUG - WIN32 related macros > > > NTDDI_VERSION=6010000 > > > WINVER=600 > > > HAVE_WIN_ATOMIC32 true > > > HAVE_WIN_ATOMIC64 true > > > util::gatherParts -- examining c:\db\fastbit\lineitem > > > part::readMetaData -- opened > > c:\db\fastbit\lineitem\-part.txt for reading > > > part[lineitem]::readRIDs -- the file manager failed to read > file > > > "c:\db\fastbit\lineitem\-rids". There is no RIDs. > > > Constructed a part named lineitem > > > activeDir = "c:\db\fastbit\lineitem" > > > part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d > > > c:\db\fastbit\lineitem -b | -m > > > > > > orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen > > > t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, > > 16 columns > > > Column list: > > > comment: (STRING) > > > commitdate: (INT) [1.79769e+308, -1.79769e+308] > > > discount: (FLOAT) [1.79769e+308, -1.79769e+308] > > > linenumber: (INT) [1.79769e+308, -1.79769e+308] > > > linestatus: (KEY) [] > > > orderkey: (INT) [1.79769e+308, -1.79769e+308] > > > partkey: (INT) [1.79769e+308, -1.79769e+308] > > > price: (FLOAT) [1.79769e+308, -1.79769e+308] > > > quantity: (FLOAT) [1.79769e+308, -1.79769e+308] > > > receipdate: (INT) [1.79769e+308, -1.79769e+308] > > > returnflag: (STRING) > > > shipdate: (INT) [1.79769e+308, -1.79769e+308] > > > shipinstuct: (STRING) > > > shipmode: (KEY) [] > > > suppkey: (INT) [1.79769e+308, -1.79769e+308] > > > tax: (FLOAT) [1.79769e+308, -1.79769e+308] > > > > > > Partition[1]: > > > lineitem > > > Query[1]: > > > SELECT returnflag,linestatus,sum(quantity) as > > > sum_qty,sum(price*(1-discount)) as > > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > > > avg(discount) as avg_disc, count(*) as count_order FROM > > lineitem > > > WHERE shipdate <= 904608000 ; > > > > > > liga -- constructed table T-lineitem (a simple list of > > partition: > > > lineitem) from a list of 1 data partition, with 16 columns and > > > 59986052 rows > > > tableSelect -- processing "SELECT > > returnflag,linestatus,sum(quantity) > > > as sum_qty,sum(price*(1-discount)) as > > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > > > avg(discount) as avg_disc, count(*) as count_order FROM > > T-lineitem > > > WHERE shipda > > > te <= 904608000 ;" > > > Constructing selectClause @ 0018C654 > > > selectClause::fillNames -- select clause internal details: > > > low-level expressions (names_[8], aggr_[8], atms_[8]): > > > 0: returnflag, returnflag > > > 1: linestatus, linestatus > > > 2: __2, SUM(quantity) > > > 3: __3, SUM((price * (1 - discount))) > > > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) > > > 5: __5, COUNT(*) > > > 6: __6, SUM(price) > > > 7: __7, SUM(discount) > > > high-level expressions (xnames_[9], xtms_[9]): > > > 0: returnflag, returnflag > > > 1: linestatus, linestatus > > > 2: sum_qty, SUM(quantity) > > > 3: sum_disc_price, SUM((price * (1 - discount))) > > > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + > > tax))) > > > 5: avg_qty, (SUM(quantity) / COUNT(*)) > > > 6: avg_price, (SUM(price) / COUNT(*)) > > > 7: avg_disc, (SUM(discount) / COUNT(*)) > > > 8: count_order, COUNT(*) > > > qExpr::simplify -- input expression shipdate <= 9.04608e+008 > > > replace a compRange with a qRange shipdate <= 9.04608e+008 > > > qExpr::simplify -- output expression (@050A5480) shipdate <= > > 9.04608e+008 > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- start timer ... > > > countQuery::setWhereClause accepted new query conditions > > "shipdate <= > > > 9.04608e+008" > > > dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- > > duration: > > > 0.001 sec(CPU), 0.000323293 sec(elapsed) > > > > direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) > > > finished reading index header with nrows=59986052 and > > bits.size()=3 > > > Constructed in-memory data partition RK2NP0 -- > > filter::sift2(SELECT > > > returnflag,linestatu ... FROM 1 data partition WHERE shipda > > ...) -- > > > with 8 columns > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- processing a select clause with 8 > > terms, 2 of > > > which are plain > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- processing data partition lineitem > > > countQuery assigned data partition lineitem > > > countQuery::evaluate -- start timer ... > > > index::create -- attempt to read index type #0 from > > > c:\db\fastbit\lineitem\shipdate.idx for column > lineitem.shipdate > > > bin[lineitem.shipdate]::ctor -- initialization completed > > with 2527 > > > bins for 59986052 rows from a storage object @ 050A6FA0 offset > 8 > > > index::create -- the bin index for column lineitem.shipdate > > was read > > > from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 > > sec(elapsed) > > > index (equality encoded, binned) for lineitem.shipdate > > contains 2527 > > > bitvectors for 59986052 objects > > > There are a total 0 set bits out of 59986052 bits -- there are > > > probably NULL values in column shipdate > > > ... (2526 omitted) > > > > > > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > > > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > > > 9.04597e+008:9.04597e+008) > > > column[lineitem.shipdate](INT)::bin::locate -- expr(shipdate <= > > > 9.04608e+008) -> [0:0, 2436:2436) (6.94303e+008:6.94303e+008, > > > 9.04597e+008:9.04597e+008) > > > index -- sumBins(0, 2436) will operate on 91 out of 2527 > > bitmaps using > > > the combined option > > > index -- sumBins operated on 91 bitmaps (6751400 in 7740160 > > out) took > > > 0.042 sec(CPU), 0.0425918 sec(elapsed). > > > column[lineitem.shipdate]::evaluateRange(shipdate <= > > 9.04608e+008, > > > mask(59986052, 59986052)) completed with ierr = 1 > > > countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, > > > mask.cnt()=59986052) --> 59123858, ierr = 1 > > > > > > countQuery::evaluate -- Select count(*) From lineitem Where > > shipdate > > > <= 9.04608e+008 --> 59123858 > > > countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 > > sec(elapsed) > > > bord[RK2NP0]::append -- to process 59123858 rows from partition > > > lineitem, # of existing rows = 0 > > > bord[RK2NP0]::append is to add 59123858 elements to column > > "__2" from > > > column "quantity" of partition lineitem > > > column[quantity]::selectValuesT got 59123858 values > > (59123858 wanted) > > > from an in-memory version of file > > c:\db\fastbit\lineitem\quantity as float > > > column[lineitem.quantity](FLOAT)::selectFloats -- retrieving > > 59123858 > > > float values took 1.012 sec(CPU), 1.01197 sec(elapsed) > > > part[lineitem]::calculate - starting to evaluate "(price * (1 - > > > discount))" with mask (59123858 out of 59986052) > > > fileManager::storage::ctor failed malloc(472990864), will > > attempt to > > > free some memory > > > fileManager::unload -- to free up all unused space > > > (totalBytes=1,187,726,660, maxBytes=1,910,734,848) > > > fileManager::unload -- unloading all (2) inactive files > > > fileManager::unload -- unloading file > > > "c:\db\fastbit\lineitem\shipdate.idx" > > > fileManager::unload -- unloading file > > "c:\db\fastbit\lineitem\quantity" > > > part[lineitem]::calculate -- evaluating (price * (1 - > > discount)) on > > > 59123858 records (total: 59986052) took 16.565 sec elapsed > > time and > > > produced 59123858 values > > > bord[RK2NP0]::append -- adding 59123858 elements to column > > __3 from > > > (price * (1 - discount)) > > > part[lineitem]::calculate - starting to evaluate "((price * (1 > - > > > discount)) * (1 + tax))" with mask (59123858 out of 59986052) > > > fileManager::storage::ctor failed malloc(472990864), will > > attempt to > > > free some memory > > > fileManager::unload -- to free up all unused space > > > (totalBytes=1,433,610,620, maxBytes=1,910,734,848) > > > fileManager::unload -- unloading all (2) inactive files > > > fileManager::unload -- unloading file > > "c:\db\fastbit\lineitem\discount" > > > clearing data partition RK2NP0 > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 > > sec(elapsed) > > > Freeing selectClause @ 0018C654 > > > Warning -- table::select absorbed a std::exception (bad > > allocation), > > > will return a nil pointer > > > --- Thu May 16 18:57:10 2013 > > > The number of memory > > > Press any key to continue . . . > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > FastBit-users mailing list > > > [email protected] > > <mailto:[email protected]> > > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
