Hi john , Thanks for your suggestion.
it is still failing , although i give this
value fileManager.maxBytes=10240000000 in the configuration file.
I'm running the following command :
c:\GitHub\fastbit\win\Release64\ibis -b "shipdate:<binning precision=3/>"
-c C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt -o c
:\db\fastbit\q2.lst -v=5 -d C:\db\fastbit\xaa
can it be a data problem ?
See output to screen :
FastBit ibis
Log messages started on Wed Jun 12 11:06:47 2013
c:\GitHub\fastbit\win\Release64\ibis
Options summary: batch mode, log level 5, building indexes
resource::read -- parsing configuration file
"C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
util::gatherParts -- examining C:\db\fastbit\xaa
part::readMetaData -- opened C:\db\fastbit\xaa\-part.txt for reading
part[xaa]::readRIDs -- the file manager failed to read file
"C:\db\fastbit\xaa\-rids". There is no RIDs.
Constructed a part named xaa
activeDir = "C:\db\fastbit\xaa"
part: xaa (c:\GitHub\fastbit\win\Release\ardea -d c:\db\fastbit\xaa -b | -m
orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity:float,price:f
loat,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key
-t c:\db\row_dat
a\xaa) with 5000000 rows, 15 columns
Column list:
commitdate: (INT) [0, 0]
discount: (FLOAT) [0, 0.1]
linenumber: (INT) [1, 7]
linestatus: linestatus = F, O, (KEY) []
orderkey: (INT) [1, 4.99882e+006]
partkey: (INT) [1, 2e+006]
price: (FLOAT) [901, 104949]
quantity: (FLOAT) [1, 50]
receipdate: (INT) [0, 0]
returnflag: returnflag (STRING)
shipdate: (INT) [6.94303e+008, 9.12463e+008]
shipinstuct: shipinstuct (STRING)
shipmode: (KEY) []
suppkey: (INT) [1, 100000]
tax: (FLOAT) [0, 0.08]
Partition[1]:
xaa
Warning -- column::actualMinMax can not handle column type TEXT, only
support int, uint, float, double
Warning -- column::actualMinMax can not handle column type TEXT, only
support int, uint, float, double
Warning -- column::actualMinMax can not handle column type CATEGORY, only
support int, uint, float, double
c:\GitHub\fastbit\win\Release64\ibis: start building indexes (nthreads=1,
indexingOptions= {shipdate:<binning precision=3/>}) ...
index::create -- attempt to read index type #21 from
C:\db\fastbit\xaa\commitdate.idx for column xaa.commitdate
direkte[commitdate]::read(0000000001FD18B0) finished reading index header
with nrows=5000000 and bits.size()=1
index::create -- the direct index for column xaa.commitdate was read from
C:\db\fastbit\xaa in 0 sec(CPU), 0.000609086 sec(elapsed)
The direct bitmap index for commitdate contains 1 bit vector
0 5000000 44
index::create -- attempt to read index type #0 from
C:\db\fastbit\xaa\discount.idx for column xaa.discount
bin[xaa.discount]::ctor -- initialization completed with 13 bins for
5000000 rows from a storage object @ 0000000001FD8E40 offset 8
index::create -- the bin index for column xaa.discount was read from
C:\db\fastbit\xaa in 0.003 sec(CPU), 0.00300951 sec(elapsed)
index (equality encoded, binned) for xaa.discount contains 13 bitvectors
for 5000000 objects
There are a total 0 set bits out of 5000000 bits -- there are probably NULL
values in column discount
... (12 omitted)
index::create -- attempt to read index type #21 from
C:\db\fastbit\xaa\linenumber.idx for column xaa.linenumber
direkte[linenumber]::read(0000000001FD8EA0) finished reading index header
with nrows=5000000 and bits.size()=8
index::create -- the direct index for column xaa.linenumber was read from
C:\db\fastbit\xaa in 0 sec(CPU), 0.000524216 sec(elapsed)
The direct bitmap index for linenumber contains 8 bit vectors
dictionary::read(C:\db\fastbit\xaa\linestatus.dic) -- duration: 0 sec(CPU),
0.000130265 sec(elapsed)
direkte[linestatus]::read(C:\db\fastbit\xaa\linestatus.idx) finished
reading index header with nrows=5000000 and bits.size()=3
index::create -- attempt to read index type #0 from
C:\db\fastbit\xaa\orderkey.idx for column xaa.orderkey
bin[xaa.orderkey]::ctor -- initialization completed with 10002 bins for
5000000 rows from a storage object @ 0000000001FD8F00 offset 8
index::create -- the bin index for column xaa.orderkey was read from
C:\db\fastbit\xaa in 0.005 sec(CPU), 0.00417399 sec(elapsed)
index (equality encoded, binned) for xaa.orderkey contains 10002 bitvectors
for 5000000 objects
0: 0 (..., 0) [1.79769e+308, -1.79769e+308]
1: 501 [0, 512) [1, 487]
2: 497 [512, 999) [512, 998]
3: 500 [999, 1510) [999, 1509]
4: 500 [1510, 1991) [1510, 1990]
5: 502 [1991, 2471) [1991, 2470]
6: 499 [2471, 2976) [2471, 2951]
7: 501 [2976, 3461) [2976, 3460]
8: 495 [3461, 3937) [3461, 3936]
9: 501 [3937, 4455) [3937, 4454]
10: 501 [4455, 4961) [4455, 4960]
11: 497 [4961, 5446) [4961, 5445]
12: 500 [5446, 5985) [5446, 5984]
13: 504 [5985, 6466) [5985, 6465]
14: 500 [6466, 6947) [6466, 6946]
15: 500 [6947, 7491) [6947, 7490]
16: 502 [7491, 8007) [7491, 8006]
17: 500 [8007, 8513) [8007, 8512]
18: 505 [8513, 8998) [8513, 8997]
19: 499 [8998, 9536) [8998, 9511]
20: 501 [9536, 10053) [9536, 10052]
21: 502 [10053, 10561) [10053, 10560]
22: 498 [10561, 11043) [10561, 11042]
23: 502 [11043, 11557) [11043, 11556]
24: 499 [11557, 12037) [11557, 12036]
25: 500 [12037, 12513) [12037, 12512]
26: 501 [12513, 12999) [12513, 12998]
27: 498 [12999, 13474) [12999, 13473]
28: 502 [13474, 13953) [13474, 13952]
29: 496 [13953, 14435) [13953, 14434]
30: 502 [14435, 14949) [14435, 14948]
31: 498 [14949, 15491) [14949, 15490]
... (9970 omitted)
Warning -- column::actualMinMax can not handle column type TEXT, only
support int, uint, float, double
index::create -- attempt to read index type #0 from
C:\db\fastbit\xaa\shipdate.idx for column xaa.shipdate
bin[xaa.shipdate]::ctor -- initialization completed with 2527 bins for
5000000 rows from a storage object @ 0000000001FD8F60 offset 8
index::create -- the bin index for column xaa.shipdate was read from
C:\db\fastbit\xaa in 0.002 sec(CPU), 0.000822641 sec(elapsed)
index (equality encoded, binned) for xaa.shipdate contains 2527 bitvectors
for 5000000 objects
There are a total 0 set bits out of 5000000 bits -- there are probably NULL
values in column shipdate
... (2526 omitted)
Warning -- column::actualMinMax can not handle column type TEXT, only
support int, uint, float, double
Warning -- column::actualMinMax can not handle column type CATEGORY, only
support int, uint, float, double
dictionary::read(C:\db\fastbit\xaa\shipmode.dic) -- duration: 0 sec(CPU),
0.000180002 sec(elapsed)
direkte[shipmode]::read(C:\db\fastbit\xaa\shipmode.idx) finished reading
index header with nrows=5000000 and bits.size()=2467
index::create -- attempt to build a new index with spec `default' on data
from directory C:\db\fastbit\xaa for column xaa.suppkey
direket[xaa.suppkey]::construct0<int> -- starting to process
C:\db\fastbit\xaa\suppkey
Started at 11:06:47.90
Ran for 4930 ms
Press any key to continue . . .
On Wed, Jun 12, 2013 at 9:23 AM, K. John Wu <[email protected]> wrote:
> Hi, Amihay,
>
> Looks like you have run out of memory while building the index for
> shipping date and the cleanup function failed to do its job, which led
> to the query evaluation procedure to fail.
>
> A way to work around this would be to build the index in shipping date
> before doing the query. You can try something like
>
> ibis -b "shipdate:<binning precision=3/>"
>
> John
>
>
> On 6/10/13 6:09 AM, amihay gonen wrote:
> > Hi , I'm trying to use fastbit ibis 64bit.
> >
> > I've allocate 5G to fastbit (use memory mapped file)
> > should be enough , but it still failed ,
> >
> > What am i doing wrong ?
> >
> > see bellow for the output
> >
> >
> > c:\GitHub\fastbit\win\Release64\ibis
> > Options summary: batch mode, log level 3, skipping estimation
> >
> > resource::read -- parsing configuration file
> > "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > resource::read -- parsing configuration file
> > "C:\GitHub\pythia\tests\fastbit\scripts\tpch_queries\ibis_config.txt"
> > fileManager initialization complete -- maxBytes=5242880000,
> > maxOpenFiles=60
> > util::gatherParts -- examining C:\db\fastbit\xaa
> > Constructed a part named xaa
> > activeDir = "C:\db\fastbit\xaa"
> > 5000000 rows and 15 columns
> > util::gatherParts -- examining C:\db\fastbit\xab
> > Constructed a part named xab
> > activeDir = "C:\db\fastbit\xab"
> > 5000000 rows and 15 columns
> > Partitions[2]:
> > xaa
> > xab
> > Query[1]:
> > SELECT returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as sum_charge,avg
> > antity) as avg_qty, avg(price) as avg_price, avg(discount) as
> > avg_disc, count(*) as count_order WHERE shipdate <= 904608000 ;
> >
> > liga -- constructed table T-xaa (a simple list of partitions: xaa and
> > xab) from a list of 2 data partitions, with 15 columns and 10000000 rows
> > tableSelect -- processing "SELECT returnflag,linestatus,sum(quantity)
> > as sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(
> > ax)) as sum_charge,avg(quantity) as avg_qty, avg(price) as avg_price,
> > avg(discount) as avg_disc, count(*) as count_order FROM T-xaa WHERE
> > shipdate
> > 904608000 ;"
> > selectClause::fillNames -- select clause internal details:
> > low-level expressions (names_[8], aggr_[8], atms_[8]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: __2, SUM(quantity)
> > 3: __3, SUM((price * (1 - discount)))
> > 4: __4, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: __5, COUNT(*)
> > 6: __6, SUM(price)
> > 7: __7, SUM(discount)
> > high-level expressions (xnames_[9], xtms_[9]):
> > 0: returnflag, returnflag
> > 1: linestatus, linestatus
> > 2: sum_qty, SUM(quantity)
> > 3: sum_disc_price, SUM((price * (1 - discount)))
> > 4: sum_charge, SUM(((price * (1 - discount)) * (1 + tax)))
> > 5: avg_qty, (SUM(quantity) / COUNT(*))
> > 6: avg_price, (SUM(price) / COUNT(*))
> > 7: avg_disc, (SUM(discount) / COUNT(*))
> > 8: count_order, COUNT(*)
> > countQuery::setWhereClause accepted new query conditions "shipdate <=
> > 9.04608e+008"
> > Constructed in-memory data partition hW04N1 -- filter::sift2S(SELECT
> > returnflag,linestatu ... FROM 2 data partitions WHERE shipd ...) --
> > with 8 col
> > s
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions
> > WHERE shipd ...) -- processing data partition xaa
> > countQuery assigned data partition xaa
> > countQuery::evaluate -- start timer ...
> > bin[xaa.shipdate]::ctor -- initialization completed with 2527 bins for
> > 5000000 rows from a storage object @ 0000000001F2C5F0 offset 8
> > index::create -- the bin index for column xaa.shipdate was read from
> > C:\db\fastbit\xaa in 0.002 sec(CPU), 0.00239134 sec(elapsed)
> > countQuery::evaluate -- Select count(*) From xaa Where shipdate <=
> > 9.04608e+008 --> 4927856
> > countQuery::evaluate -- duration: 0.009 sec(CPU), 0.00776298 sec(elapsed)
> > part[xaa]::calculate - starting to evaluate "(price * (1 - discount))"
> > with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating (price * (1 - discount)) on 4927856
> > records (total: 5000000) took 0.146679 sec elapsed time and produced
> > 4927856
> > lues
> > bord[hW04N1]::append -- adding 4927856 elements to column __3 from
> > (price * (1 - discount))
> > part[xaa]::calculate - starting to evaluate "((price * (1 - discount))
> > * (1 + tax))" with mask (4927856 out of 5000000)
> > part[xaa]::calculate -- evaluating ((price * (1 - discount)) * (1 +
> > tax)) on 4927856 records (total: 5000000) took 0.222345 sec elapsed
> > time and pr
> > ced 4927856 values
> > bord[hW04N1]::append -- adding 4927856 elements to column __4 from
> > ((price * (1 - discount)) * (1 + tax))
> > text[xaa.returnflag]::selectStrings -- duration: 0.597 sec(CPU),
> > 0.597355 sec(elapsed)
> > bundles::ctor created a colValues for "returnflag" as cols[1] with
> > size 4927856
> > bundles::ctor created a colValues for "linestatus" as cols[2] with
> > size 4927856
> > bundles::ctor created a colValues for "quantity" as cols[3] with size
> > 4927856
> > bundles::ctor created a colValues for "(price * (1 - discount))" as
> > cols[4] with size 4927856
> > bundles::ctor created a colValues for "((price * (1 - discount)) * (1
> > + tax))" as cols[5] with size 4927856
> > bundles::ctor created a colValues for "price" as cols[6] with size
> 4927856
> > bundles::ctor created a colValues for "discount" as cols[7] with size
> > 4927856
> > Constructed in-memory data partition ELH5k -- Select returnflag,
> > linestatus, SUM(quantity), SUM((price * (1 - discount))), SUM(((price
> > * (1 - disco
> > )) * (1 + tax))), COUNT(*), SUM(price), SUM(discount) From hW04N1 --
> > with 4 rows and 8 columns
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions
> > WHERE shipd ...) -- processing data partition xab
> > countQuery changing data partition from xaa to xab
> > countQuery::evaluate -- start timer ...
> > column[xab.shipdate](INT)::index::mapValues -- generated histogram
> > (2526 distinct values)
> > fileManager::storage::ctor failed malloc(112), will attempt to free
> > some memory
> > fileManager::storage::ctor failed malloc(28672), will attempt to free
> > some memory
> > Warning -- column[shipdate]::bin::ctor encountered an exception,
> > cleaning up ...
> > Warning -- index::create(shipdate) received an std::exception -- bad
> > allocation
> > Warning -- index[xab.shipdate]::create failed to create an index of
> > type default
> > column[xab.shipdate]::evaluateRange(shipdate <= 9.04608e+008) received
> > a std::exception -- bad allocation
> > part::doScan -- evaluating shipdate <= 9.04608e+008 on 5000000 int
> > values (total: 5000000) took 0.0192468 sec elapsed time and produced
> > 4928205 hit
> > countQuery::evaluate -- Select count(*) From xab Where shipdate <=
> > 9.04608e+008 --> 4928205
> > countQuery::evaluate -- duration: 0.523 sec(CPU), 0.523351 sec(elapsed)
> > part[xab]::calculate - starting to evaluate "(price * (1 - discount))"
> > with mask (4928205 out of 5000000)
> > clearing data partition ELH5k
> > clearing data partition hW04N1
> > filter::sift2S(SELECT returnflag,linestatu ... FROM 2 data partitions
> > WHERE shipd ...) -- duration: 3.897 sec(CPU), 3.89773 sec(elapsed)
> > Warning -- table::select absorbed a std::exception (bad allocation),
> > will return a nil pointer
> > tableSelect:: select(returnflag,linestatus,sum(quantity) as
> > sum_qty,sum(price*(1-discount)) as
> > sum_disc_price,sum(price*(1-discount)*(1+tax)) as su
> > harge,avg(quantity) as avg_qty, avg(price) as avg_price, avg(discount)
> > as avg_disc, count(*) as count_order , shipdate <= 904608000 ;) failed
> > on ta
> > T-xaa
> > c:\GitHub\fastbit\win\Release64\ibis -- total CPU time 3.913 s, total
> > elapsed time 3.9138 s
> > clearing data partition xaa
> > clearing data partition xab
> > Cleaning up the file manager
> > Total pages accessed through read(unistd.h) is estimated to be 29351
> > fileManager::clear -- starting ...
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 1. (max = 60)
> > file name: C:\db\fastbit\xab\shipdate
> > storage @ 0000000001F2C410, 00000000027E0000, 1st 32 bits = 33345660,
> > 1st 64 bits = 3399cbd033345660
> > file descriptor 00000000000005E0
> > fmap handle 00000000000005E4
> > base address 00000000027E0000
> > mapped y opened at Mon Jun 10 16:04:27 2013
> > last used at Mon Jun 10 16:04:27 2013
> > # of bytes 20000000 # of past acc 1 # of active acc 0
> > Size of all mapped files is 20,000,000
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 20,000,000
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 20,000,000
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size = 8192)
> >
> >
> >
> > --- Mon Jun 10 16:04:29 2013
> > The number of memory mapped files is 0. (max = 60)
> > Size of all mapped files is 0
> >
> > The number of files read into memory is 0
> >
> > The total size of all files read into memory is 0
> > Size of all named storages is 0
> > Size of all unnamed storages is 0
> > The total size of all named and unnamed storages is 0
> > The prescribed maximum size is 5,242,880,000
> > Number of pages accessed (recorded so far) is 29351 (page size = 8192)
> >
> >
> > fileManager decommissioned
> > Started at 16:04:25.25
> > Ran for 4000 ms
> > Press any key to continue . . .
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected]
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
