Hi all,
I've updated screed with a couple of fixes. You may recall that screed
(http://github.com/ctb/screed/tree/master) is a FileSequenceDB
replacement intended for short-read sequence databases; you can see some
example code here:
http://github.com/ctb/screed/blob/372bded41875421dfb92f965dc0e0110294b9721/doc/screed.txt
The summary? screed DBs can now be pickled, and I've figured out the
memory consumption & load time issue, too. Plus, screed DBs are faster
than SequenceFileDBs for random seeks.
You can get screed from github,
http://github.com/ctb/screed/tree/master
--
Details:
First, the screed dbs can now be pickled, and hence also saved into
worldbase. Yay! I just had to override __reduce__.
Next, I debugged the memory consumption & load-time problem that
Namshin was experiencing (2 or more seconds on load!). There were two
different problems:
a) screed was loading the entire index of records into memory twice.
This was basically braindead behavior that I fixed with some ifstream
magic.
b) screed is loading the entire index of records into memory. This
behavior is basically a tradeoff between memory and speed: if you don't
do this, you have to do two disk seeks in order to retrieve a record;
if you DO preload the index, you only have to do one.
So I did some benchmarking.
The results are OK. For retrieval of 100k random sequences (randomly
picking a key, then retrieving actual sequence), I got the following
numbers very consistently:
pygr/SequenceFileDB/bsddb: 21.9s total (19.4s/user 2.5s/sys)
screed/ScreedSequenceDB/no preload: 14.7s total (10.7s/user 3.5s/sys)
screed/ScreedSequenceDB/preload: 12.8s total (10.1s/user 2.5s/sys)
So, screed can be close to twice as fast as the bsddb-based
SequenceFileDB, IF you preload the entire record index (which will
consume 8 bytes * number of sequences). If you don't preload, you take
significant hit (~20%) in disk usage -- the 'sys' number above increases
-- but it's still significantly faster.
Enjoy. I'm going to start using this myself with some next-gen
sequencing data & we'll see how it goes.
cheers,
--titus
p.s. Istvan, I still don't have a Windows native version going; sorry.
Hopefully that'll be next up when Alex returns.
--
C. Titus Brown, [email protected]
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"pygr-dev" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/pygr-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
