Hi, Amihey,
I think that I have got rid of all the memory leaks I can spot now.
The updated code is in SVN Revision 642 (and tar ball marked 1.3.7).
Please given them a try and see if they fix the problem you are
experiencing.
Thanks for the test cases.
John
On 6/12/13 1:10 AM, amihay gonen wrote:
> Hi john , Thanks for your suggestion.
>
> it is still failing , although i give this
> value fileManager.maxBytes=10240000000 in the configuration file.
>
> I'm running the following command :
> c:\GitHub\fastbit\win\Release64\ibis -b "shipdate:<binning
> precision=3/>" -c
> C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt -o c
> :\db\fastbit\q2.lst -v=5 -d C:\db\fastbit\xaa
>
>
> can it be a data problem ?
>
> See output to screen :
>
> FastBit ibis
> Log messages started on Wed Jun 12 11:06:47 2013
>
> c:\GitHub\fastbit\win\Release64\ibis
> Options summary: batch mode, log level 5, building indexes
>
> resource::read -- parsing configuration file
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> util::gatherParts -- examining C:\db\fastbit\xaa
> part::readMetaData -- opened C:\db\fastbit\xaa\-part.txt for reading
> part[xaa]::readRIDs -- the file manager failed to read file
> "C:\db\fastbit\xaa\-rids". There is no RIDs.
> Constructed a part named xaa
> activeDir = "C:\db\fastbit\xaa"
> part: xaa (c:\GitHub\fastbit\win\Release\ardea -d c:\db\fastbit\xaa -b
> | -m
> orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:f
> loat,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key
> -t c:\db\row_dat
> a\xaa) with 5000000 rows, 15 columns
> Column list:
> commitdate: (INT) [0, 0]
> discount: (FLOAT) [0, 0.1]
> linenumber: (INT) [1, 7]
> linestatus: linestatus = F, O, (KEY) []
> orderkey: (INT) [1, 4.99882e+006]
> partkey: (INT) [1, 2e+006]
> price: (FLOAT) [901, 104949]
> quantity: (FLOAT) [1, 50]
> receipdate: (INT) [0, 0]
> returnflag: returnflag (STRING)
> shipdate: (INT) [6.94303e+008, 9.12463e+008]
> shipinstuct: shipinstuct (STRING)
> shipmode: (KEY) []
> suppkey: (INT) [1, 100000]
> tax: (FLOAT) [0, 0.08]
>
> Partition[1]:
> xaa
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type CATEGORY,
> only support int, uint, float, double
> c:\GitHub\fastbit\win\Release64\ibis: start building indexes
> (nthreads=1, indexingOptions= {shipdate:<binning precision=3/>}) ...
> index::create -- attempt to read index type #21 from
> C:\db\fastbit\xaa\commitdate.idx for column xaa.commitdate
> direkte[commitdate]::read(0000000001FD18B0) finished reading index
> header with nrows=5000000 and bits.size()=1
> index::create -- the direct index for column xaa.commitdate was read
> from C:\db\fastbit\xaa in 0 sec(CPU), 0.000609086 sec(elapsed)
> The direct bitmap index for commitdate contains 1 bit vector
>
> 0 5000000 44
>
>
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\discount.idx for column xaa.discount
> bin[xaa.discount]::ctor -- initialization completed with 13 bins for
> 5000000 rows from a storage object @ 0000000001FD8E40 offset 8
> index::create -- the bin index for column xaa.discount was read from
> C:\db\fastbit\xaa in 0.003 sec(CPU), 0.00300951 sec(elapsed)
> index (equality encoded, binned) for xaa.discount contains 13
> bitvectors for 5000000 objects
> There are a total 0 set bits out of 5000000 bits -- there are probably
> NULL values in column discount
> ... (12 omitted)
>
> index::create -- attempt to read index type #21 from
> C:\db\fastbit\xaa\linenumber.idx for column xaa.linenumber
> direkte[linenumber]::read(0000000001FD8EA0) finished reading index
> header with nrows=5000000 and bits.size()=8
> index::create -- the direct index for column xaa.linenumber was read
> from C:\db\fastbit\xaa in 0 sec(CPU), 0.000524216 sec(elapsed)
> The direct bitmap index for linenumber contains 8 bit vectors
>
>
>
> dictionary::read(C:\db\fastbit\xaa\linestatus.dic) -- duration: 0
> sec(CPU), 0.000130265 sec(elapsed)
> direkte[linestatus]::read(C:\db\fastbit\xaa\linestatus.idx) finished
> reading index header with nrows=5000000 and bits.size()=3
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\orderkey.idx for column xaa.orderkey
> bin[xaa.orderkey]::ctor -- initialization completed with 10002 bins
> for 5000000 rows from a storage object @ 0000000001FD8F00 offset 8
> index::create -- the bin index for column xaa.orderkey was read from
> C:\db\fastbit\xaa in 0.005 sec(CPU), 0.00417399 sec(elapsed)
> index (equality encoded, binned) for xaa.orderkey contains 10002
> bitvectors for 5000000 objects
> 0: 0 (..., 0) [1.79769e+308, -1.79769e+308]
> 1: 501 [0, 512) [1, 487]
> 2: 497 [512, 999) [512, 998]
> 3: 500 [999, 1510) [999, 1509]
> 4: 500 [1510, 1991) [1510, 1990]
> 5: 502 [1991, 2471) [1991, 2470]
> 6: 499 [2471, 2976) [2471, 2951]
> 7: 501 [2976, 3461) [2976, 3460]
> 8: 495 [3461, 3937) [3461, 3936]
> 9: 501 [3937, 4455) [3937, 4454]
> 10: 501 [4455, 4961) [4455, 4960]
> 11: 497 [4961, 5446) [4961, 5445]
> 12: 500 [5446, 5985) [5446, 5984]
> 13: 504 [5985, 6466) [5985, 6465]
> 14: 500 [6466, 6947) [6466, 6946]
> 15: 500 [6947, 7491) [6947, 7490]
> 16: 502 [7491, 8007) [7491, 8006]
> 17: 500 [8007, 8513) [8007, 8512]
> 18: 505 [8513, 8998) [8513, 8997]
> 19: 499 [8998, 9536) [8998, 9511]
> 20: 501 [9536, 10053) [9536, 10052]
> 21: 502 [10053, 10561) [10053, 10560]
> 22: 498 [10561, 11043) [10561, 11042]
> 23: 502 [11043, 11557) [11043, 11556]
> 24: 499 [11557, 12037) [11557, 12036]
> 25: 500 [12037, 12513) [12037, 12512]
> 26: 501 [12513, 12999) [12513, 12998]
> 27: 498 [12999, 13474) [12999, 13473]
> 28: 502 [13474, 13953) [13474, 13952]
> 29: 496 [13953, 14435) [13953, 14434]
> 30: 502 [14435, 14949) [14435, 14948]
> 31: 498 [14949, 15491) [14949, 15490]
> ... (9970 omitted)
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\shipdate.idx for column xaa.shipdate
> bin[xaa.shipdate]::ctor -- initialization completed with 2527 bins for
> 5000000 rows from a storage object @ 0000000001FD8F60 offset 8
> index::create -- the bin index for column xaa.shipdate was read from
> C:\db\fastbit\xaa in 0.002 sec(CPU), 0.000822641 sec(elapsed)
> index (equality encoded, binned) for xaa.shipdate contains 2527
> bitvectors for 5000000 objects
> There are a total 0 set bits out of 5000000 bits -- there are probably
> NULL values in column shipdate
> ... (2526 omitted)
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type CATEGORY,
> only support int, uint, float, double
> dictionary::read(C:\db\fastbit\xaa\shipmode.dic) -- duration: 0
> sec(CPU), 0.000180002 sec(elapsed)
> direkte[shipmode]::read(C:\db\fastbit\xaa\shipmode.idx) finished
> reading index header with nrows=5000000 and bits.size()=2467
> index::create -- attempt to build a new index with spec `default' on
> data from directory C:\db\fastbit\xaa for column xaa.suppkey
> direket[xaa.suppkey]::construct0<int> -- starting to process
> C:\db\fastbit\xaa\suppkey
> Started at 11:06:47.90
> Ran for 4930 ms
> Press any key to continue . . .
>
>
> On Wed, Jun 12, 2013 at 9:23 AM, K. John Wu <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Amihay,
>
> Looks like you have run out of memory while building the index for
> shipping date and the cleanup function failed to do its job, which led
> to the query evaluation procedure to fail.
>
> A way to work around this would be to build the index in shipping date
> before doing the query. You can try something like
>
> ibis -b "shipdate:<binning precision=3/>"
>
> John
>
>
> On 6/10/13 6:09 AM, amihay gonen wrote:
> > Hi , I'm trying to use fastbit ibis 64bit.
> >
> > I've allocate 5G to fastbit (use memory mapped file)
> > should be enough , but it still failed ,
> >
> > What am i doing wrong ?
> >
> > see bellow for the output
> >
> >
> > c:\GitHub\fastbit\win\Release64\ibis
> > Options summary: batch mode, log level 3, skipping estimation
> >
> > resource::read -- parsing configuration file
> >
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > resource::read -- parsing configuration file
> >
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > fileManager initialization complete -- maxBytes=5242880000,
> > maxOpenFiles=60
> > util::gatherParts -- examining C:\db\fastbit\xaa
> > Constructed a part named xaa
> > activeDir = "C:\db\fastbit\xaa"
> > 5000000 rows and 15 columns
> > util::gatherParts -- examining C:\db\fastbit\xab
> > Constructed a part named xab
> > activeDir = "C:\db\fastbit\xab"
> > 5000000 rows and 15 columns
> > Partitions[2]:
> > xaa
> > xab
> > Query[1]:
> > SELECT returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg
> > antity) as avg_qty, avg(price) as avg_price, avg(discount) as
> > avg_disc, count(*) as count_order WHERE shipdate <= 904608000 ;
> >
> > liga -- constructed table T-xaa (a simple list of partitions:
> xaa and
> > xab) from a list of 2 data partitions, with 15 columns and
> 10000000 rows
> > tableSelect -- processing "SELECT
> returnflag,linestatus,sum(quantity)
> > as sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(
> > ax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as
> avg_price,
> > avg(discount) as avg_disc, count(*) as count_order FROM T-xaa WHERE
> > shipdate
> > 904608000 ;"
> > selectClause::fillNames -- select clause internal details:
> > low-level expressions (names_[8], aggr_[8], atms_[8]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: __2, SUM(quantity)
> > 3: __3, SUM((price * (1 - discount)))
> > 4: __4, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: __5, COUNT(*)
> > 6: __6, SUM(price)
> > 7: __7, SUM(discount)
> > high-level expressions (xnames_[9], xtms_[9]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: sum_qty, SUM(quantity)
> > 3: sum_disc_price, SUM((price * (1 - discount)))
> > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: avg_qty, (SUM(quantity) / COUNT(*))
> > 6: avg_price, (SUM(price) / COUNT(*))
> > 7: avg_disc, (SUM(discount) / COUNT(*))
> > 8: count_order, COUNT(*)
> > countQuery::setWhereClause accepted new query conditions
> "shipdate <=
> > 9.04608e+008"
> > Constructed in-memory data partition hW04N1 -- filter::sift2S(SELECT
> > returnflag,linestatu ... FROM 2 data partitions WHERE shipd ...) --
> > with 8 col
> > s
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- processing data partition xaa
> > countQuery assigned data partition xaa
> > countQuery::evaluate -- start timer ...
> > bin[xaa.shipdate]::ctor -- initialization completed with 2527
> bins for
> > 5000000 rows from a storage object @ 0000000001F2C5F0 offset 8
> > index::create -- the bin index for column xaa.shipdate was read from
> > C:\db\fastbit\xaa in 0.002 sec(CPU), 0.00239134 sec(elapsed)
> > countQuery::evaluate -- Select count(*) From xaa Where shipdate <=
> > 9.04608e+008 --> 4927856
> > countQuery::evaluate -- duration: 0.009 sec(CPU), 0.00776298
> sec(elapsed)
> > part[xaa]::calculate - starting to evaluate "(price * (1 -
> discount))"
> > with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating (price * (1 - discount)) on
> 4927856
> > records (total: 5000000) took 0.146679 sec elapsed time and produced
> > 4927856
> > lues
> > bord[hW04N1]::append -- adding 4927856 elements to column __3 from
> > (price * (1 - discount))
> > part[xaa]::calculate - starting to evaluate "((price * (1 -
> discount))
> > * (1 + tax))" with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating ((price * (1 - discount)) * (1 +
> > tax)) on 4927856 records (total: 5000000) took 0.222345 sec elapsed
> > time and pr
> > ced 4927856 values
> > bord[hW04N1]::append -- adding 4927856 elements to column __4 from
> > ((price * (1 - discount)) * (1 + tax))
> > text[xaa.returnflag]::selectStrings -- duration: 0.597 sec(CPU),
> > 0.597355 sec(elapsed)
> > bundles::ctor created a colValues for "returnflag" as cols[1] with
> > size 4927856
> > bundles::ctor created a colValues for "linestatus" as cols[2] with
> > size 4927856
> > bundles::ctor created a colValues for "quantity" as cols[3] with
> size
> > 4927856
> > bundles::ctor created a colValues for "(price * (1 - discount))" as
> > cols[4] with size 4927856
> > bundles::ctor created a colValues for "((price * (1 - discount))
> * (1
> > + tax))" as cols[5] with size 4927856
> > bundles::ctor created a colValues for "price" as cols[6] with
> size 4927856
> > bundles::ctor created a colValues for "discount" as cols[7] with
> size
> > 4927856
> > Constructed in-memory data partition ELH5k -- Select returnflag,
> > linestatus, SUM(quantity), SUM((price * (1 - discount))),
> SUM(((price
> > * (1 - disco
> > )) * (1 + tax))), COUNT(*), SUM(price), SUM(discount) From hW04N1 --
> > with 4 rows and 8 columns
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- processing data partition xab
> > countQuery changing data partition from xaa to xab
> > countQuery::evaluate -- start timer ...
> > column[xab.shipdate](INT)::index::mapValues -- generated histogram
> > (2526 distinct values)
> > fileManager::storage::ctor failed malloc(112), will attempt to free
> > some memory
> > fileManager::storage::ctor failed malloc(28672), will attempt to
> free
> > some memory
> > Warning -- column[shipdate]::bin::ctor encountered an exception,
> > cleaning up ...
> > Warning -- index::create(shipdate) received an std::exception -- bad
> > allocation
> > Warning -- index[xab.shipdate]::create failed to create an index of
> > type default
> > column[xab.shipdate]::evaluateRange(shipdate <= 9.04608e+008)
> received
> > a std::exception -- bad allocation
> > part::doScan -- evaluating shipdate <= 9.04608e+008 on 5000000 int
> > values (total: 5000000) took 0.0192468 sec elapsed time and produced
> > 4928205 hit
> > countQuery::evaluate -- Select count(*) From xab Where shipdate <=
> > 9.04608e+008 --> 4928205
> > countQuery::evaluate -- duration: 0.523 sec(CPU), 0.523351
> sec(elapsed)
> > part[xab]::calculate - starting to evaluate "(price * (1 -
> discount))"
> > with mask (4928205 out of 5000000)
> > clearing data partition ELH5k
> > clearing data partition hW04N1
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- duration: 3.897 sec(CPU), 3.89773 sec(elapsed)
> > Warning -- table::select absorbed a std::exception (bad allocation),
> > will return a nil pointer
> > tableSelect:: select(returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as su
> > harge,avg(quantity) as avg_qty, avg(price) as avg_price,
> avg(discount)
> > as avg_disc, count(*) as count_order , shipdate <= 904608000 ;)
> failed
> > on ta
> > T-xaa
> > c:\GitHub\fastbit\win\Release64\ibis -- total CPU time 3.913 s,
> total
> > elapsed time 3.9138 s
> > clearing data partition xaa
> > clearing data partition xab
> > Cleaning up the file manager
> > Total pages accessed through read(unistd.h) is estimated to be 29351
> > fileManager::clear -- starting ...
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 1. (max = 60)
> > file name: C:\db\fastbit\xab\shipdate
> > storage @ 0000000001F2C410, 00000000027E0000, 1st 32 bits =
> 33345660,
> > 1st 64 bits = 3399cbd033345660
> > file descriptor 00000000000005E0
> > fmap handle 00000000000005E4
> > base address 00000000027E0000
> > mapped y opened at Mon Jun 10 16:04:27 2013
> > last used at Mon Jun 10 16:04:27 2013
> > # of bytes 20000000 # of past acc 1 # of
> active acc 0
> > Size of all mapped files is 20,000,000
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 20,000,000
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 20,000,000
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size =
> 8192)
> >
> >
> >
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 0. (max = 60)
> > Size of all mapped files is 0
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 0
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 0
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size =
> 8192)
> >
> >
> > fileManager decommissioned
> > Started at 16:04:25.25
> > Ran for 4000 ms
> > Press any key to continue . . .
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected] <mailto:[email protected]>
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
>
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
