Basically, all the design criteria for a datstore index are the same as for a general database indice.
> 1. How do you efficiently search indices (remember that datastore > indices are like the flat filesystems on old bitty boxen)? B+ tree, or a hash structure. > 2. How do you efficiently add items to datastore indices (you can't > go and write an entire half gigabyte index to file each time you > add a single file (yes, this is an extreme example, but it *will* > happen))? Any modern 2nd-level storage algo handles this. > 3. How do you efficiently look for files to displace from the > datastore when you add new files (yeah, like the other stuff in > this list, it doesn't matter when you have a few hundred files in > the datastore, but it soon becomes very significant when you have > half a million files)? LRU stack seperate from the index. > 4. How do you break up files in the datastore and the datastore > indices to get around file size limits imposed by filesystems? what? > 5. Binary formats make datastores *easier* to work with (you don't > have to rewrite the entire datastore each time you change the > status of a file in the datastore), but you have to watch out for > stuff like endianness. Duh. > All of these issues have not been addressed yet, and they will be > significant problems once Freenet reaches primetime. I'm planning on > not procrastinating on solving these problems, but instead solving > them from the first version of nfreenetd. All of this stuff makes > writing a Freenet node daemon far more complex and difficult than it > appears at face value. I have to challenge this assumption. Datastores aren't meant to be this multi-gigabyte repository. Running a site like flys in the face of the whole idea behind Freenet and distributed datastores. You don't *need* to run these enormous centralized/webserver like nodes. In fact, I think a lot of your criticisms really don't make much sense. You're not designing a webserver. It doesnt *have* to scale to enterprise levels. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20000803/cb1376bd/attachment.pgp>
