Hi, Amihay, Internally, FastBit tracks the data partitions by name. It appears that you have given the two partitions exactly the same name (and the two partitions also happen have the same number of rows and same number of columns). FastBit suspects that you have given the same data partition twice and therefore is telling you that it will only use one of them. Here are a couple of suggestions to work around this problem.
- given each data partition a different name yourself. - give no name to any of the partitions, in which case, the name of the directory will be used - and therefore each partition will have a different name - I could add something else to the test of whether the new partition is the same as an existing one, which might take a while to happen.. Let me know if you would rather that I take the last option and feel free to suggest something.. John On 5/23/13 11:11 PM, amihay gonen wrote: > Hi John ,thanks your suggesting. > This was I trying in the first place , but I got the following error, > which i don't understan why: > > C:\Users\agonen>C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\q.bat > > FastBit ibis > Log messages started on Fri May 24 09:09:07 2013 > > c:\GitHub\fastbit\win\Release32\ibis > Options summary: batch mode, log level 3, skipping estimation > > resource::read -- parsing configuration file > "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt" > fileManager initialization complete -- maxBytes=734003200, maxOpenFiles=60 > util::gatherParts -- examining C:\db\fastbit\xaa > Constructed a part named lineitem > activeDir = "C:\db\fastbit\xaa" > 1000 rows and 15 columns > util::gatherParts -- examining C:\db\fastbit\xab > Constructed a part named lineitem > activeDir = "C:\db\fastbit\xab" > 1000 rows and 15 columns > *Warning -- util::gatherParts finds the data partition in > C:\db\fastbit\xab to have exactly the same name, number of rows, > number of columns, and time stamp as the one in C:\db\fastbit\xaa > already in memory, discards the new one and keeps the old one* > clearing data partition lineitem > Partition[1]: > lineitem > Query[1]: > SELECT returnflag,linestatus,sum(quantity) as > sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > avg(discount) as avg_disc, count(*) as count_order FROM lineitem > WHERE shipdate <= 904608000 ; > > liga -- constructed table T-lineitem (a simple list of partition: > lineitem) from a list of 1 data partition, with 15 columns and 1000 rows > tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity) > as sum_qty,sum(price*(1-discount)) as > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price, > avg(discount) as avg_disc, count(*) as count_order FROM T-lineitem > WHERE shipda > te <= 904608000 ;" > selectClause::fillNames -- select clause internal details: > low-level expressions (names_[8], aggr_[8], atms_[8]): > 0: returnflag, returnflag > 1: linestatus, linestatus > 2: __2, SUM(quantity) > 3: __3, SUM((price * (1 - discount))) > 4: __4, SUM(((price * (1 - discount)) * (1 + tax))) > 5: __5, COUNT(*) > 6: __6, SUM(price) > > > On Fri, May 24, 2013 at 1:15 AM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Amihay, > > Glad to see that you are getting it to work. > > The function that recursively visit all subdirectories is using > opendir which is not available on windows environment - at least not > the way I am using them.. You will need to explicitly name each data > directory on the command through the -d option or listing the data > directories in a RC file. > > John > > > On 5/23/13 7:43 AM, amihay gonen wrote: > > I've tried to use partitions to over come the memory limits , but i > > didn't find a way to tell ibis to scan several partition , I > think it > > is because I'm working on windows so gatherparts does'n work > (opendir ) > > > > am i correct ? > > > > > > On Thu, May 23, 2013 at 12:57 PM, amihay gonen > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > I've found out that I can increase ibis memory using by > setting in > > configruation file something like that : > > fileManager.maxBytes=1648576000 ,but it still doesn't work well. > > > > I wander if I'll split the data into partitions , will it solved > > the memory issues , as each partition will procsse spereatly ? > > > > Thanks, > > Amihay > > > > > > On Fri, May 17, 2013 at 10:14 AM, K. John Wu <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Hi, Amihay, > > > > The error message indicates that you are running out of > > memory. Seems > > like the intermediate answers you are getting requires too > > much memory. > > > > John > > > > > > On 5/16/13 9:01 AM, amihay gonen wrote: > > > Hi I'm trying to query from 60M rows with 32bit bit ibis , > > but it > > > fails , I think it is releated to memory , but i can't > > understand way . > > > > > > I've attached the output from a debug version . I hope you > > can find > > > the reason > > > > > > > > > Thanks ,Amihay > > > > > > > > > q.bat > > > > > > FastBit ibis > > > Log messages started on Thu May 16 18:56:51 2013 > > > > > > c:\GitHub\fastbit\win\Debug32\ibis > > > Options summary: batch mode, log level 5, skipping > estimation > > > > > > fileManager::ctor found the physical memory size to be > > 4186902528 bytes > > > fileManager initialization complete -- > maxBytes=1910734848, > > > maxOpenFiles=60 > > > DEBUG - WIN32 related macros > > > NTDDI_VERSION=6010000 > > > WINVER=600 > > > HAVE_WIN_ATOMIC32 true > > > HAVE_WIN_ATOMIC64 true > > > util::gatherParts -- examining c:\db\fastbit\lineitem > > > part::readMetaData -- opened > > c:\db\fastbit\lineitem\-part.txt for reading > > > part[lineitem]::readRIDs -- the file manager failed to > read file > > > "c:\db\fastbit\lineitem\-rids". There is no RIDs. > > > Constructed a part named lineitem > > > activeDir = "c:\db\fastbit\lineitem" > > > part: lineitem (c:\GitHub\fastbit\win\Release\ardea -d > > > c:\db\fastbit\lineitem -b | -m > > > > > > > orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,commen > > > t:text -t c:\db\row_data\lineitem.tbl) with 59986052 rows, > > 16 columns > > > Column list: > > > comment: (STRING) > > > commitdate: (INT) [1.79769e+308, -1.79769e+308] > > > discount: (FLOAT) [1.79769e+308, -1.79769e+308] > > > linenumber: (INT) [1.79769e+308, -1.79769e+308] > > > linestatus: (KEY) [] > > > orderkey: (INT) [1.79769e+308, -1.79769e+308] > > > partkey: (INT) [1.79769e+308, -1.79769e+308] > > > price: (FLOAT) [1.79769e+308, -1.79769e+308] > > > quantity: (FLOAT) [1.79769e+308, -1.79769e+308] > > > receipdate: (INT) [1.79769e+308, -1.79769e+308] > > > returnflag: (STRING) > > > shipdate: (INT) [1.79769e+308, -1.79769e+308] > > > shipinstuct: (STRING) > > > shipmode: (KEY) [] > > > suppkey: (INT) [1.79769e+308, -1.79769e+308] > > > tax: (FLOAT) [1.79769e+308, -1.79769e+308] > > > > > > Partition[1]: > > > lineitem > > > Query[1]: > > > SELECT returnflag,linestatus,sum(quantity) as > > > sum_qty,sum(price*(1-discount)) as > > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > > sum_charge,avg(quantity) as avg_qty, avg(price) as > avg_price, > > > avg(discount) as avg_disc, count(*) as count_order FROM > > lineitem > > > WHERE shipdate <= 904608000 ; > > > > > > liga -- constructed table T-lineitem (a simple list of > > partition: > > > lineitem) from a list of 1 data partition, with 16 > columns and > > > 59986052 rows > > > tableSelect -- processing "SELECT > > returnflag,linestatus,sum(quantity) > > > as sum_qty,sum(price*(1-discount)) as > > > sum_disc_price,sum(price*(1-discount)*(1+tax)) as > > > sum_charge,avg(quantity) as avg_qty, avg(price) as > avg_price, > > > avg(discount) as avg_disc, count(*) as count_order FROM > > T-lineitem > > > WHERE shipda > > > te <= 904608000 ;" > > > Constructing selectClause @ 0018C654 > > > selectClause::fillNames -- select clause internal details: > > > low-level expressions (names_[8], aggr_[8], atms_[8]): > > > 0: returnflag, returnflag > > > 1: linestatus, linestatus > > > 2: __2, SUM(quantity) > > > 3: __3, SUM((price * (1 - discount))) > > > 4: __4, SUM(((price * (1 - discount)) * (1 + > tax))) > > > 5: __5, COUNT(*) > > > 6: __6, SUM(price) > > > 7: __7, SUM(discount) > > > high-level expressions (xnames_[9], xtms_[9]): > > > 0: returnflag, returnflag > > > 1: linestatus, linestatus > > > 2: sum_qty, SUM(quantity) > > > 3: sum_disc_price, SUM((price * (1 - discount))) > > > 4: sum_charge, SUM(((price * (1 - discount)) > * (1 + > > tax))) > > > 5: avg_qty, (SUM(quantity) / COUNT(*)) > > > 6: avg_price, (SUM(price) / COUNT(*)) > > > 7: avg_disc, (SUM(discount) / COUNT(*)) > > > 8: count_order, COUNT(*) > > > qExpr::simplify -- input expression shipdate <= > 9.04608e+008 > > > replace a compRange with a qRange shipdate <= 9.04608e+008 > > > qExpr::simplify -- output expression (@050A5480) > shipdate <= > > 9.04608e+008 > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- start timer ... > > > countQuery::setWhereClause accepted new query conditions > > "shipdate <= > > > 9.04608e+008" > > > dictionary::read(c:\db\fastbit\lineitem\linestatus.dic) -- > > duration: > > > 0.001 sec(CPU), 0.000323293 sec(elapsed) > > > > direkte[linestatus]::read(c:\db\fastbit\lineitem\linestatus.idx) > > > finished reading index header with nrows=59986052 and > > bits.size()=3 > > > Constructed in-memory data partition RK2NP0 -- > > filter::sift2(SELECT > > > returnflag,linestatu ... FROM 1 data partition WHERE > shipda > > ...) -- > > > with 8 columns > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- processing a select clause with 8 > > terms, 2 of > > > which are plain > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- processing data partition lineitem > > > countQuery assigned data partition lineitem > > > countQuery::evaluate -- start timer ... > > > index::create -- attempt to read index type #0 from > > > c:\db\fastbit\lineitem\shipdate.idx for column > lineitem.shipdate > > > bin[lineitem.shipdate]::ctor -- initialization completed > > with 2527 > > > bins for 59986052 rows from a storage object @ > 050A6FA0 offset 8 > > > index::create -- the bin index for column > lineitem.shipdate > > was read > > > from c:\db\fastbit\lineitem in 0.007 sec(CPU), 0.00701494 > > sec(elapsed) > > > index (equality encoded, binned) for lineitem.shipdate > > contains 2527 > > > bitvectors for 59986052 objects > > > There are a total 0 set bits out of 59986052 bits -- > there are > > > probably NULL values in column shipdate > > > ... (2526 omitted) > > > > > > column[lineitem.shipdate](INT)::bin::locate -- > expr(shipdate <= > > > 9.04608e+008) -> [0:0, 2436:2436) > (6.94303e+008:6.94303e+008, > > > 9.04597e+008:9.04597e+008) > > > column[lineitem.shipdate](INT)::bin::locate -- > expr(shipdate <= > > > 9.04608e+008) -> [0:0, 2436:2436) > (6.94303e+008:6.94303e+008, > > > 9.04597e+008:9.04597e+008) > > > index -- sumBins(0, 2436) will operate on 91 out of 2527 > > bitmaps using > > > the combined option > > > index -- sumBins operated on 91 bitmaps (6751400 in > 7740160 > > out) took > > > 0.042 sec(CPU), 0.0425918 sec(elapsed). > > > column[lineitem.shipdate]::evaluateRange(shipdate <= > > 9.04608e+008, > > > mask(59986052, 59986052)) completed with ierr = 1 > > > countQuery::doEvaluate(050A5300: shipdate <= 9.04608e+008, > > > mask.cnt()=59986052) --> 59123858, ierr = 1 > > > > > > countQuery::evaluate -- Select count(*) From lineitem > Where > > shipdate > > > <= 9.04608e+008 --> 59123858 > > > countQuery::evaluate -- duration: 0.168 sec(CPU), 0.16737 > > sec(elapsed) > > > bord[RK2NP0]::append -- to process 59123858 rows from > partition > > > lineitem, # of existing rows = 0 > > > bord[RK2NP0]::append is to add 59123858 elements to column > > "__2" from > > > column "quantity" of partition lineitem > > > column[quantity]::selectValuesT got 59123858 values > > (59123858 wanted) > > > from an in-memory version of file > > c:\db\fastbit\lineitem\quantity as float > > > column[lineitem.quantity](FLOAT)::selectFloats -- > retrieving > > 59123858 > > > float values took 1.012 sec(CPU), 1.01197 sec(elapsed) > > > part[lineitem]::calculate - starting to evaluate > "(price * (1 - > > > discount))" with mask (59123858 out of 59986052) > > > fileManager::storage::ctor failed malloc(472990864), will > > attempt to > > > free some memory > > > fileManager::unload -- to free up all unused space > > > (totalBytes=1,187,726,660, maxBytes=1,910,734,848) > > > fileManager::unload -- unloading all (2) inactive files > > > fileManager::unload -- unloading file > > > "c:\db\fastbit\lineitem\shipdate.idx" > > > fileManager::unload -- unloading file > > "c:\db\fastbit\lineitem\quantity" > > > part[lineitem]::calculate -- evaluating (price * (1 - > > discount)) on > > > 59123858 records (total: 59986052) took 16.565 sec elapsed > > time and > > > produced 59123858 values > > > bord[RK2NP0]::append -- adding 59123858 elements to column > > __3 from > > > (price * (1 - discount)) > > > part[lineitem]::calculate - starting to evaluate > "((price * (1 - > > > discount)) * (1 + tax))" with mask (59123858 out of > 59986052) > > > fileManager::storage::ctor failed malloc(472990864), will > > attempt to > > > free some memory > > > fileManager::unload -- to free up all unused space > > > (totalBytes=1,433,610,620, maxBytes=1,910,734,848) > > > fileManager::unload -- unloading all (2) inactive files > > > fileManager::unload -- unloading file > > "c:\db\fastbit\lineitem\discount" > > > clearing data partition RK2NP0 > > > filter::sift2(SELECT returnflag,linestatu ... FROM 1 data > > partition > > > WHERE shipda ...) -- duration: 18.219 sec(CPU), 18.2193 > > sec(elapsed) > > > Freeing selectClause @ 0018C654 > > > Warning -- table::select absorbed a std::exception (bad > > allocation), > > > will return a nil pointer > > > --- Thu May 16 18:57:10 2013 > > > The number of memory > > > Press any key to continue . . . > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > FastBit-users mailing list > > > [email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>> > > > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
