Thanks for spending time considering my suggestions.
When i started using scid i was amazed by the incredible speed of
position searching (compared to Chessbase 9 that i used at that time).
However you had to wait until the search was completed before making
another move, and i hate to wait computer programs.
So i thought: ok, i have plenty of free RAM, let's add an option to load
the full database into RAM.
Very clever indeed: my code made searching SLOWER!
So i profiled the code and concluded that the real bottleneck is
decoding the moves (and updating the complex Position class).
In the end i gave up (changing the database format is a huge step) and
wrote the trick in the actual scid code that automatically interrupt the
tree search if the user change the board position.
However maybe i made some errors and my measurements was wrong: any more
test/info is certainly welcome.
But you need to test many different positions and you can't disable the
optimizations if you want meaningful results.
Let's say we have a big database file (filename.big) > 200MB.
Let's try this:
(1) Clear the disk cache
# sync; echo 3 > /proc/sys/vm/drop_caches; free -m
(2) Read 100MB
$ dd if=filename.big of=/dev/null count=200000
(3) Clear the disk cache
# sync; echo 3 > /proc/sys/vm/drop_caches; free -m
(4) Read 200MB
$ dd if=filename.big of=/dev/null count=400000
Point (2) will be obviously much faster than point (4)
This explain the results of your test.
However this is not our case.
We are reading small parts of a big file (an average game takes about
100 byte with 1-byte format).
Lets use a simple script.sh:
for i in $(seq 1 1000);
do
let sk=i*400
dd if=filename.big of=/dev/null bs=$1 count=1 skip=$sk 2> /dev/null
done
And try again:
1) Clear the disk cache
# sync; echo 3 > /proc/sys/vm/drop_caches; free -m
2) Read 1000 records of 128 byte
$ time ./script.sh 128
3) Clear the disk cache
# sync; echo 3 > /proc/sys/vm/drop_caches; free -m
4) Read 1000 records of 256 byte
$ time ./script.sh 256
Point (2) will take the same time of Point (4)
That's why i think your conclusions are wrong.
As you said, only the first moves will be decoded, but the game is
entirely read from disk.
So you can try to:
1) decode the full game instead of the first moves and record the
performance drop
2) load into RAM the entire games file and record the performance gain
Then you can compare the two results and draw you conclusions.
Bye,
Fulvio
Gregor Cramer wrote:
- Improve the speed of position searching
Scid use only 1-byte for non-queen moves, and it's very
clever, but very slow to decode.
Nowadays harddisk space is not a problem and a 2-byte for
move format will be more appropriate in my opinion.
Steve is right that file IO is the slowest part. I did an experimental
two byte move decoding, and a test with database Big2010.sci (converted
from Big2010.cbh) which contains 4.462.803 games. After I did the move
1.e4 the position search (in opening tree window) took about 20 seconds.
The one byte version needs only about 10 seconds. The result: a two byte
decoding is slowing down the position search by factor 2.
Scid use only 1-byte for non-queen moves, and it's very clever,
but very slow to decode.
This is not true, the decoding is fast, although the two byte decoding
is faster. (By the way, the ChessBase decoding is slow).
scid use a very clever tree_search algorithm that read on average
less than 5% of the sg4 database.
Yes, this is true, but this means that a faster decoding cannot have
a significant impact. But the disk IO has a very significant impact.
Furthermore on average only a few moves will be decoded in a game,
the tree search algorithm is indeed clever.
Gregor
------------------------------------------------------------------------------
