Frank A. Stevenson wrote: > I wrote a table sorting function, ... > > Tarball is available for inspection at: http://traxme.net/a5/ (native > endianness is assumed when reading the tables) > > F > I have done more research on efficient table storage. My initial suggestion of using 1 million distict files would simply overwhelm a ext2 forametted flash device with inodes, so I had to reduce the number of files to a more manageble 4096. I have updated my tables sorting code accordingly (on the website for now, will add to svn later ) I have also added code that does the lookup into the compressed tables (not finished)
The interesting figures from a 8GB Kingston datatraveler with 5MB/sec write speed are: 1000 lookups in a smaller table (40million chains) can be performed in 2.1 seconds from a cold device (newly inserted, with no locally cached inodes) - access speed is not an issue :-) Time to sort a complete table would be limited to the write speed of the device (~15 minutes). It is also interesting to note that the suggested table size should fit in < 4GB in compressed format. So an even multiples of tables can easily be stored on devices in commonly available sizes. The big advantage with flash based storage is that it won't bog down your machine with a torrent of disk requests when doing th lookup work. F _______________________________________________ A51 mailing list [email protected] http://lists.lists.reflextor.com/cgi-bin/mailman/listinfo/a51
