On Fri, Jan 24, 2003 at 07:40:53AM +0000, Gordan Bobic wrote: > On Friday 24 Jan 2003 12:01 am, Matthew Toseland wrote: > > > > > Fproxy uses 256kB to 1MB chunks, but other clients could use other > > > > sizes. That is however the recommended range for most uses. > > > > > > ... > > > > > > > Splitfiles therefore can fail if too many of the chunks are no longer > > > > fetchable. > > > > > > Is it possible to use smaller chunks? Can you give me a link to a > > > document that explains how to control the use of FEC via fproxy? For > > > example, can I force the use of FEC for files smaller than 1 MB? > > > > No. Your application would not use Fproxy anyway, it would probably use > > a library to talk directly to the node using FCP. What language are you > > considering? > > Perl for maintainance utilities and browser-side JavaScript for client-side > operation. > > And yes, I know that JS will make the anonymity filter complain, but > ultimately, JS in browser is no more of an anonymity issue than any other way > to create Freenet application. Hmm. Interesting. Have you considered writing it in java as a servlet? Surely you do not want to make users have to click anonymity warnings every five seconds? :). Eventually, we will implement somse sort of server side scripting... with a couple of limits it could be reasonably safe. We might just allow some client-side stuff, but only "trusted recipes". Help with this would be greatly appreciated :) > > Once I have a prototype read-write implementation written in javascript, I > will probably port it to Perl DBI. > > With the browser-based approach, fproxy is just about all I can do, as > everything has to be done using HTTP requests. > > In the case of a Perl DBI library, there are more options, but the HTTP access > libraries are already there to make things a bit easier. > > > > > You do know that Freenet is lossy, right? Content which is not accessed > > > > very much will eventually expire. > > > > > > Yes, this is why I am thinking about using DBR. There would potentially > > > be a number of nodes that would once per day retrieve the data, compact > > > it into bigger files, and re-insert it for the next day. This would be > > > equivalent to vacuum (PostgreSQL) and optimize (MySQL) commands. > > > > > > The daily operation data would involve inserting many small files (one > > > file per record in a table, one file per delete flag, etc.) > > > > > > This would all be gathered, compacted, and re-inserted. Any indices would > > > also get re-generated in the same way. > > > > Hmm. Interesting. > > No idea if it will work, but I don't think there's any way to find out other > than write a prototype. :-) > > > > > FEC divides the file into segments of up to 128 chunks (I think). > > > > It then creates 64 check blocks for the 128 chunks (obviously fewer if > > > > fewer original chunks), and inserts the lot, along with a file > > > > specifying the CHKs of all the different chunks inserted for each > > > > segment. > > > > > > Doesn't that mean that with maximum chunk size of 1 MB, this limits the > > > file size to 128 MB? Or did I misunderstand the maximum chunk size, and > > > it is purely a matter of caching as a factor of the store size? > > > > No. After 128MB, we use more than one segment. Within each segment, we > > need any 128 of the 192 chunks to reconstruct the file. > > Ah, OK, I understand now. > > > > Is FEC fixed at 50% redundancy, or can the amount of redundancy be > > > controlled (e.g. reduced to 25%, if requested)? Or has it been tried and > > > tested that around 50% gives best results? > > > > Hmm. At the moment it is hard-coded. At some point we may change this. > > The original libraries support other amounts. > > OK. > > Thank you for your help. I think I have a better idea of how to go about > writing my app now. :-) > > Gordan > > _______________________________________________ > Tech mailing list > [EMAIL PROTECTED] > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/tech >
-- Matthew Toseland [EMAIL PROTECTED][EMAIL PROTECTED] Full time freenet hacker. http://freenetproject.org/ Freenet Distribution Node (temporary) at http://amphibian.dyndns.org:8889/eI49KUM9vEY/ ICTHUS.
msg01063/pgp00000.pgp
Description: PGP signature
