Hi, Amilay,
The log file you shown only contains warning about the function
actualMinMax not able to handle text columns. This should not cause
any other problems. It should have built the indexes correctly.
Have you tried to answer the query you were using last time? Got the
same errors/warnings?
John
On 6/12/13 1:10 AM, amihay gonen wrote:
> Hi john , Thanks for your suggestion.
>
> it is still failing , although i give this
> value fileManager.maxBytes=10240000000 in the configuration file.
>
> I'm running the following command :
> c:\GitHub\fastbit\win\Release64\ibis -b "shipdate:<binning
> precision=3/>" -c
> C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt -o c
> :\db\fastbit\q2.lst -v=5 -d C:\db\fastbit\xaa
>
>
> can it be a data problem ?
>
> See output to screen :
>
> FastBit ibis
> Log messages started on Wed Jun 12 11:06:47 2013
>
> c:\GitHub\fastbit\win\Release64\ibis
> Options summary: batch mode, log level 5, building indexes
>
> resource::read -- parsing configuration file
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> util::gatherParts -- examining C:\db\fastbit\xaa
> part::readMetaData -- opened C:\db\fastbit\xaa\-part.txt for reading
> part[xaa]::readRIDs -- the file manager failed to read file
> "C:\db\fastbit\xaa\-rids". There is no RIDs.
> Constructed a part named xaa
> activeDir = "C:\db\fastbit\xaa"
> part: xaa (c:\GitHub\fastbit\win\Release\ardea -d c:\db\fastbit\xaa -b
> | -m
> orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:f
> loat,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key
> -t c:\db\row_dat
> a\xaa) with 5000000 rows, 15 columns
> Column list:
> commitdate: (INT) [0, 0]
> discount: (FLOAT) [0, 0.1]
> linenumber: (INT) [1, 7]
> linestatus: linestatus = F, O, (KEY) []
> orderkey: (INT) [1, 4.99882e+006]
> partkey: (INT) [1, 2e+006]
> price: (FLOAT) [901, 104949]
> quantity: (FLOAT) [1, 50]
> receipdate: (INT) [0, 0]
> returnflag: returnflag (STRING)
> shipdate: (INT) [6.94303e+008, 9.12463e+008]
> shipinstuct: shipinstuct (STRING)
> shipmode: (KEY) []
> suppkey: (INT) [1, 100000]
> tax: (FLOAT) [0, 0.08]
>
> Partition[1]:
> xaa
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type CATEGORY,
> only support int, uint, float, double
> c:\GitHub\fastbit\win\Release64\ibis: start building indexes
> (nthreads=1, indexingOptions= {shipdate:<binning precision=3/>}) ...
> index::create -- attempt to read index type #21 from
> C:\db\fastbit\xaa\commitdate.idx for column xaa.commitdate
> direkte[commitdate]::read(0000000001FD18B0) finished reading index
> header with nrows=5000000 and bits.size()=1
> index::create -- the direct index for column xaa.commitdate was read
> from C:\db\fastbit\xaa in 0 sec(CPU), 0.000609086 sec(elapsed)
> The direct bitmap index for commitdate contains 1 bit vector
>
> 0 5000000 44
>
>
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\discount.idx for column xaa.discount
> bin[xaa.discount]::ctor -- initialization completed with 13 bins for
> 5000000 rows from a storage object @ 0000000001FD8E40 offset 8
> index::create -- the bin index for column xaa.discount was read from
> C:\db\fastbit\xaa in 0.003 sec(CPU), 0.00300951 sec(elapsed)
> index (equality encoded, binned) for xaa.discount contains 13
> bitvectors for 5000000 objects
> There are a total 0 set bits out of 5000000 bits -- there are probably
> NULL values in column discount
> ... (12 omitted)
>
> index::create -- attempt to read index type #21 from
> C:\db\fastbit\xaa\linenumber.idx for column xaa.linenumber
> direkte[linenumber]::read(0000000001FD8EA0) finished reading index
> header with nrows=5000000 and bits.size()=8
> index::create -- the direct index for column xaa.linenumber was read
> from C:\db\fastbit\xaa in 0 sec(CPU), 0.000524216 sec(elapsed)
> The direct bitmap index for linenumber contains 8 bit vectors
>
>
>
> dictionary::read(C:\db\fastbit\xaa\linestatus.dic) -- duration: 0
> sec(CPU), 0.000130265 sec(elapsed)
> direkte[linestatus]::read(C:\db\fastbit\xaa\linestatus.idx) finished
> reading index header with nrows=5000000 and bits.size()=3
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\orderkey.idx for column xaa.orderkey
> bin[xaa.orderkey]::ctor -- initialization completed with 10002 bins
> for 5000000 rows from a storage object @ 0000000001FD8F00 offset 8
> index::create -- the bin index for column xaa.orderkey was read from
> C:\db\fastbit\xaa in 0.005 sec(CPU), 0.00417399 sec(elapsed)
> index (equality encoded, binned) for xaa.orderkey contains 10002
> bitvectors for 5000000 objects
> 0: 0 (..., 0) [1.79769e+308, -1.79769e+308]
> 1: 501 [0, 512) [1, 487]
> 2: 497 [512, 999) [512, 998]
> 3: 500 [999, 1510) [999, 1509]
> 4: 500 [1510, 1991) [1510, 1990]
> 5: 502 [1991, 2471) [1991, 2470]
> 6: 499 [2471, 2976) [2471, 2951]
> 7: 501 [2976, 3461) [2976, 3460]
> 8: 495 [3461, 3937) [3461, 3936]
> 9: 501 [3937, 4455) [3937, 4454]
> 10: 501 [4455, 4961) [4455, 4960]
> 11: 497 [4961, 5446) [4961, 5445]
> 12: 500 [5446, 5985) [5446, 5984]
> 13: 504 [5985, 6466) [5985, 6465]
> 14: 500 [6466, 6947) [6466, 6946]
> 15: 500 [6947, 7491) [6947, 7490]
> 16: 502 [7491, 8007) [7491, 8006]
> 17: 500 [8007, 8513) [8007, 8512]
> 18: 505 [8513, 8998) [8513, 8997]
> 19: 499 [8998, 9536) [8998, 9511]
> 20: 501 [9536, 10053) [9536, 10052]
> 21: 502 [10053, 10561) [10053, 10560]
> 22: 498 [10561, 11043) [10561, 11042]
> 23: 502 [11043, 11557) [11043, 11556]
> 24: 499 [11557, 12037) [11557, 12036]
> 25: 500 [12037, 12513) [12037, 12512]
> 26: 501 [12513, 12999) [12513, 12998]
> 27: 498 [12999, 13474) [12999, 13473]
> 28: 502 [13474, 13953) [13474, 13952]
> 29: 496 [13953, 14435) [13953, 14434]
> 30: 502 [14435, 14949) [14435, 14948]
> 31: 498 [14949, 15491) [14949, 15490]
> ... (9970 omitted)
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> index::create -- attempt to read index type #0 from
> C:\db\fastbit\xaa\shipdate.idx for column xaa.shipdate
> bin[xaa.shipdate]::ctor -- initialization completed with 2527 bins for
> 5000000 rows from a storage object @ 0000000001FD8F60 offset 8
> index::create -- the bin index for column xaa.shipdate was read from
> C:\db\fastbit\xaa in 0.002 sec(CPU), 0.000822641 sec(elapsed)
> index (equality encoded, binned) for xaa.shipdate contains 2527
> bitvectors for 5000000 objects
> There are a total 0 set bits out of 5000000 bits -- there are probably
> NULL values in column shipdate
> ... (2526 omitted)
>
> Warning -- column::actualMinMax can not handle column type TEXT, only
> support int, uint, float, double
> Warning -- column::actualMinMax can not handle column type CATEGORY,
> only support int, uint, float, double
> dictionary::read(C:\db\fastbit\xaa\shipmode.dic) -- duration: 0
> sec(CPU), 0.000180002 sec(elapsed)
> direkte[shipmode]::read(C:\db\fastbit\xaa\shipmode.idx) finished
> reading index header with nrows=5000000 and bits.size()=2467
> index::create -- attempt to build a new index with spec `default' on
> data from directory C:\db\fastbit\xaa for column xaa.suppkey
> direket[xaa.suppkey]::construct0<int> -- starting to process
> C:\db\fastbit\xaa\suppkey
> Started at 11:06:47.90
> Ran for 4930 ms
> Press any key to continue . . .
>
>
> On Wed, Jun 12, 2013 at 9:23 AM, K. John Wu <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Amihay,
>
> Looks like you have run out of memory while building the index for
> shipping date and the cleanup function failed to do its job, which led
> to the query evaluation procedure to fail.
>
> A way to work around this would be to build the index in shipping date
> before doing the query. You can try something like
>
> ibis -b "shipdate:<binning precision=3/>"
>
> John
>
>
> On 6/10/13 6:09 AM, amihay gonen wrote:
> > Hi , I'm trying to use fastbit ibis 64bit.
> >
> > I've allocate 5G to fastbit (use memory mapped file)
> > should be enough , but it still failed ,
> >
> > What am i doing wrong ?
> >
> > see bellow for the output
> >
> >
> > c:\GitHub\fastbit\win\Release64\ibis
> > Options summary: batch mode, log level 3, skipping estimation
> >
> > resource::read -- parsing configuration file
> >
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > resource::read -- parsing configuration file
> >
> "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > fileManager initialization complete -- maxBytes=5242880000,
> > maxOpenFiles=60
> > util::gatherParts -- examining C:\db\fastbit\xaa
> > Constructed a part named xaa
> > activeDir = "C:\db\fastbit\xaa"
> > 5000000 rows and 15 columns
> > util::gatherParts -- examining C:\db\fastbit\xab
> > Constructed a part named xab
> > activeDir = "C:\db\fastbit\xab"
> > 5000000 rows and 15 columns
> > Partitions[2]:
> > xaa
> > xab
> > Query[1]:
> > SELECT returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg
> > antity) as avg_qty, avg(price) as avg_price, avg(discount) as
> > avg_disc, count(*) as count_order WHERE shipdate <= 904608000 ;
> >
> > liga -- constructed table T-xaa (a simple list of partitions:
> xaa and
> > xab) from a list of 2 data partitions, with 15 columns and
> 10000000 rows
> > tableSelect -- processing "SELECT
> returnflag,linestatus,sum(quantity)
> > as sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(
> > ax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as
> avg_price,
> > avg(discount) as avg_disc, count(*) as count_order FROM T-xaa WHERE
> > shipdate
> > 904608000 ;"
> > selectClause::fillNames -- select clause internal details:
> > low-level expressions (names_[8], aggr_[8], atms_[8]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: __2, SUM(quantity)
> > 3: __3, SUM((price * (1 - discount)))
> > 4: __4, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: __5, COUNT(*)
> > 6: __6, SUM(price)
> > 7: __7, SUM(discount)
> > high-level expressions (xnames_[9], xtms_[9]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: sum_qty, SUM(quantity)
> > 3: sum_disc_price, SUM((price * (1 - discount)))
> > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: avg_qty, (SUM(quantity) / COUNT(*))
> > 6: avg_price, (SUM(price) / COUNT(*))
> > 7: avg_disc, (SUM(discount) / COUNT(*))
> > 8: count_order, COUNT(*)
> > countQuery::setWhereClause accepted new query conditions
> "shipdate <=
> > 9.04608e+008"
> > Constructed in-memory data partition hW04N1 -- filter::sift2S(SELECT
> > returnflag,linestatu ... FROM 2 data partitions WHERE shipd ...) --
> > with 8 col
> > s
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- processing data partition xaa
> > countQuery assigned data partition xaa
> > countQuery::evaluate -- start timer ...
> > bin[xaa.shipdate]::ctor -- initialization completed with 2527
> bins for
> > 5000000 rows from a storage object @ 0000000001F2C5F0 offset 8
> > index::create -- the bin index for column xaa.shipdate was read from
> > C:\db\fastbit\xaa in 0.002 sec(CPU), 0.00239134 sec(elapsed)
> > countQuery::evaluate -- Select count(*) From xaa Where shipdate <=
> > 9.04608e+008 --> 4927856
> > countQuery::evaluate -- duration: 0.009 sec(CPU), 0.00776298
> sec(elapsed)
> > part[xaa]::calculate - starting to evaluate "(price * (1 -
> discount))"
> > with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating (price * (1 - discount)) on
> 4927856
> > records (total: 5000000) took 0.146679 sec elapsed time and produced
> > 4927856
> > lues
> > bord[hW04N1]::append -- adding 4927856 elements to column __3 from
> > (price * (1 - discount))
> > part[xaa]::calculate - starting to evaluate "((price * (1 -
> discount))
> > * (1 + tax))" with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating ((price * (1 - discount)) * (1 +
> > tax)) on 4927856 records (total: 5000000) took 0.222345 sec elapsed
> > time and pr
> > ced 4927856 values
> > bord[hW04N1]::append -- adding 4927856 elements to column __4 from
> > ((price * (1 - discount)) * (1 + tax))
> > text[xaa.returnflag]::selectStrings -- duration: 0.597 sec(CPU),
> > 0.597355 sec(elapsed)
> > bundles::ctor created a colValues for "returnflag" as cols[1] with
> > size 4927856
> > bundles::ctor created a colValues for "linestatus" as cols[2] with
> > size 4927856
> > bundles::ctor created a colValues for "quantity" as cols[3] with
> size
> > 4927856
> > bundles::ctor created a colValues for "(price * (1 - discount))" as
> > cols[4] with size 4927856
> > bundles::ctor created a colValues for "((price * (1 - discount))
> * (1
> > + tax))" as cols[5] with size 4927856
> > bundles::ctor created a colValues for "price" as cols[6] with
> size 4927856
> > bundles::ctor created a colValues for "discount" as cols[7] with
> size
> > 4927856
> > Constructed in-memory data partition ELH5k -- Select returnflag,
> > linestatus, SUM(quantity), SUM((price * (1 - discount))),
> SUM(((price
> > * (1 - disco
> > )) * (1 + tax))), COUNT(*), SUM(price), SUM(discount) From hW04N1 --
> > with 4 rows and 8 columns
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- processing data partition xab
> > countQuery changing data partition from xaa to xab
> > countQuery::evaluate -- start timer ...
> > column[xab.shipdate](INT)::index::mapValues -- generated histogram
> > (2526 distinct values)
> > fileManager::storage::ctor failed malloc(112), will attempt to free
> > some memory
> > fileManager::storage::ctor failed malloc(28672), will attempt to
> free
> > some memory
> > Warning -- column[shipdate]::bin::ctor encountered an exception,
> > cleaning up ...
> > Warning -- index::create(shipdate) received an std::exception -- bad
> > allocation
> > Warning -- index[xab.shipdate]::create failed to create an index of
> > type default
> > column[xab.shipdate]::evaluateRange(shipdate <= 9.04608e+008)
> received
> > a std::exception -- bad allocation
> > part::doScan -- evaluating shipdate <= 9.04608e+008 on 5000000 int
> > values (total: 5000000) took 0.0192468 sec elapsed time and produced
> > 4928205 hit
> > countQuery::evaluate -- Select count(*) From xab Where shipdate <=
> > 9.04608e+008 --> 4928205
> > countQuery::evaluate -- duration: 0.523 sec(CPU), 0.523351
> sec(elapsed)
> > part[xab]::calculate - starting to evaluate "(price * (1 -
> discount))"
> > with mask (4928205 out of 5000000)
> > clearing data partition ELH5k
> > clearing data partition hW04N1
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data
> partitions
> > WHERE shipd ...) -- duration: 3.897 sec(CPU), 3.89773 sec(elapsed)
> > Warning -- table::select absorbed a std::exception (bad allocation),
> > will return a nil pointer
> > tableSelect:: select(returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as su
> > harge,avg(quantity) as avg_qty, avg(price) as avg_price,
> avg(discount)
> > as avg_disc, count(*) as count_order , shipdate <= 904608000 ;)
> failed
> > on ta
> > T-xaa
> > c:\GitHub\fastbit\win\Release64\ibis -- total CPU time 3.913 s,
> total
> > elapsed time 3.9138 s
> > clearing data partition xaa
> > clearing data partition xab
> > Cleaning up the file manager
> > Total pages accessed through read(unistd.h) is estimated to be 29351
> > fileManager::clear -- starting ...
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 1. (max = 60)
> > file name: C:\db\fastbit\xab\shipdate
> > storage @ 0000000001F2C410, 00000000027E0000, 1st 32 bits =
> 33345660,
> > 1st 64 bits = 3399cbd033345660
> > file descriptor 00000000000005E0
> > fmap handle 00000000000005E4
> > base address 00000000027E0000
> > mapped y opened at Mon Jun 10 16:04:27 2013
> > last used at Mon Jun 10 16:04:27 2013
> > # of bytes 20000000 # of past acc 1 # of
> active acc 0
> > Size of all mapped files is 20,000,000
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 20,000,000
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 20,000,000
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size =
> 8192)
> >
> >
> >
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 0. (max = 60)
> > Size of all mapped files is 0
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 0
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 0
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size =
> 8192)
> >
> >
> > fileManager decommissioned
> > Started at 16:04:25.25
> > Ran for 4000 ms
> > Press any key to continue . . .
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected] <mailto:[email protected]>
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
>
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
