Hi, Ghousia,
My apologies for the slow response. If you are still interested,
would you mind download the latest code from SVN and try to see if
java/milky.java can reproduce the problem?
The attached log file is from a run with milky.java with my
replication of your test data (which seems to be slightly smaller than
yours).
Thanks.
John
PS: You can check out the latest code from SVN with the following command
svn checkout https://codeforge.lbl.gov/anonscm/fastbit
If you have already checked out a copy some time ago, you only need to
do 'svn update' in the working directory.
On 3/14/11 6:23 AM, Ghousia wrote:
Hi John,
Thanks for the reply,
We are still working on the sample data available with fastbit
examples. To reach 5G partitions, we are concatenating the data
available in test0.csv under tests directory of Fastbit. Once we
achieve the desired performance with fastbit we will start
experimenting Fastbit with our business data.
Kindly find below answers to your questions:
1. How much memory do you have on the test machine? how much memory is
allotted to FastBit?
We are running the fastbit on a dedicated Redhat linux el5 machine
without much overhead from other processes. Test machines have a
memory of 24G, with around 8G allotted to Fastbit . (This is set
through java argument)
2. How many results (rows) are you retrieving? what type of values
are involved? what is the expected sizes of them?
Around 4497792 rows out of total<> rows are being
retrieved per partition. We are fetching columns with integers types.
Each query results in 5MB data.
3. Are you keeping the values in Java code? How much memory do you
expect to use?
We are not keeping the values in java code. Just to analyze this
issue in detail, we are throwing away the result. Once we get pass
this issue, we will start utilizing the results values further in our
code.
Many Thanks,
Ghousia.
FastBit.set_messsage_level(2) returned 0
starting to process directory ../../t/0
Completed construction of an part named t0 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/0"
419430400 rows and 3 columns
query[Cs-1JJqH643----0]::setPartition -- new data patition name t0
query[Cs-1JJqH643----0]::setWhereClause -- where "a=0"
column[t0.a](BYTE)::readIndex -- the basic index was read (../../t/0) in 0.001
sec(CPU), 0.046906 sec(elapsed)
query[Cs-1JJqH643----0]::evaluate -- time to compute the 4194304 hits: 0.549916
sec(CPU), 0.691084 sec(elapsed).
query[Cs-1JJqH643----0]::evaluate -- user kewu FROM t0 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/0 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/0 took 7227 ms
starting to process directory ../../t/1
Completed construction of an part named t1 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/1"
419430400 rows and 3 columns
query[Cs-1JJqH64Z----1]::setPartition -- new data patition name t1
query[Cs-1JJqH64Z----1]::setWhereClause -- where "a=0"
column[t1.a](BYTE)::readIndex -- the basic index was read (../../t/1) in 0.001
sec(CPU), 0.249024 sec(elapsed)
query[Cs-1JJqH64Z----1]::evaluate -- time to compute the 4194304 hits: 0.540917
sec(CPU), 0.858713 sec(elapsed).
query[Cs-1JJqH64Z----1]::evaluate -- user kewu FROM t1 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/1 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/1 took 6900 ms
starting to process directory ../../t/2
Completed construction of an part named t2 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/2"
419430400 rows and 3 columns
query[Cs-1JJqH65-----2]::setPartition -- new data patition name t2
query[Cs-1JJqH65-----2]::setWhereClause -- where "a=0"
column[t2.a](BYTE)::readIndex -- the basic index was read (../../t/2) in 0
sec(CPU), 0.0350199 sec(elapsed)
query[Cs-1JJqH65-----2]::evaluate -- time to compute the 4194304 hits: 0.541917
sec(CPU), 0.642127 sec(elapsed).
query[Cs-1JJqH65-----2]::evaluate -- user kewu FROM t2 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/2 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/2 took 6801 ms
starting to process directory ../../t/3
Completed construction of an part named t3 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/3"
419430400 rows and 3 columns
query[Cs-1JJqH65R----3]::setPartition -- new data patition name t3
query[Cs-1JJqH65R----3]::setWhereClause -- where "a=0"
column[t3.a](BYTE)::readIndex -- the basic index was read (../../t/3) in 0
sec(CPU), 0.506515 sec(elapsed)
query[Cs-1JJqH65R----3]::evaluate -- time to compute the 4194304 hits: 0.543917
sec(CPU), 1.1402 sec(elapsed).
query[Cs-1JJqH65R----3]::evaluate -- user kewu FROM t3 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/3 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/3 took 7276 ms
starting to process directory ../../t/4
Completed construction of an part named t4 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/4"
419430400 rows and 3 columns
query[Cs-1JJqH65w----4]::setPartition -- new data patition name t4
query[Cs-1JJqH65w----4]::setWhereClause -- where "a=0"
column[t4.a](BYTE)::readIndex -- the basic index was read (../../t/4) in 0
sec(CPU), 0.368333 sec(elapsed)
query[Cs-1JJqH65w----4]::evaluate -- time to compute the 4194304 hits: 0.53092
sec(CPU), 0.963928 sec(elapsed).
query[Cs-1JJqH65w----4]::evaluate -- user kewu FROM t4 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/4 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/4 took 6990 ms
starting to process directory ../../t/5
Completed construction of an part named t5 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/5"
419430400 rows and 3 columns
query[Cs-1JJqH66N----5]::setPartition -- new data patition name t5
query[Cs-1JJqH66N----5]::setWhereClause -- where "a=0"
column[t5.a](BYTE)::readIndex -- the basic index was read (../../t/5) in 0
sec(CPU), 0.0585041 sec(elapsed)
query[Cs-1JJqH66N----5]::evaluate -- time to compute the 4194304 hits: 0.520921
sec(CPU), 0.639361 sec(elapsed).
query[Cs-1JJqH66N----5]::evaluate -- user kewu FROM t5 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/5 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/5 took 7447 ms
starting to process directory ../../t/6
Completed construction of an part named t6 --
C:\Home\John\src\ibis\win\Debug\ardea.exe -d t0 -m a:b, b:f, c:s -t test0.csv
activeDir = "../../t/6"
419430400 rows and 3 columns
query[Cs-1JJqH66s----6]::setPartition -- new data patition name t6
query[Cs-1JJqH66s----6]::setWhereClause -- where "a=0"
column[t6.a](BYTE)::readIndex -- the basic index was read (../../t/6) in 0
sec(CPU), 0.39732 sec(elapsed)
query[Cs-1JJqH66s----6]::evaluate -- time to compute the 4194304 hits: 0.530919
sec(CPU), 1.0143 sec(elapsed).
query[Cs-1JJqH66s----6]::evaluate -- user kewu FROM t6 WHERE a=0 ==> 4194304
hits.
applying "a=0" on data in ../../t/6 produced 4194304 hits
a[4194304]: 0 0 0 0 ... (4194300 skipped)
b[4194304]: 0.0 0.0 0.0 0.0 ... (4194300 skipped)
FastBit.destroy_query returned 0
Processing directory ../../t/6 took 6846 ms
fileManager::clear -- completed with 392 bytes of storage remain in memory
after removing all managed objects
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
