[freenet-dev] FCP FEC Proposal -- in message body
Gianni Johansson writes: > It's good to make the distinction because if you manage to get all the data > blocks you don't need to decode at all. Prefering data blocks to check blocks will restrict the usefulness of FEC, as it will make the check blocks less popular, thus less fetchable, in extreme degenerating to non-FEC splitfiles. I'm with Fish in the belief that all blocks should be requested in random order, and with equal probability. Decoding is a really cheap operation relative to the fetching of the blocks from Freenet. We shouldn't shun it and make FEC less efficient in the long run. > Client writers just don't want to deal with it. It's certainly a class harder than fetching a monolithic key. I also found that accurate documentation more accessible than the source is nonexistent. > FEC support has been in CVS for almost a year and you are the first > person who has attempted to write a client besides me. Well, could have something to do with Freenet being largely unusable for retrieval of bigger files in that time period. With the current comeback, more freesites are a-coming, with more FEC splitfiles, so the need for clients supporting that rises as well. The FCP access will surely be a good thing. As long as direct access is still possible ... and I don't see why that should be removed. -- Robbe -- next part -- A non-text attachment was scrubbed... Name: signature.ng Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/9f22f076/attachment.pgp>
[freenet-dev] Cleaning up fproxy
Gianni Johansson writes: > / -- with no arguments would map to Ian's infolet main page >-- with a valid URI, maps to a leaner and meaner fproxy-lite servlet that > handles normal requests and inserts but redirects splitfile > inserts /requests to he servlets below. > /servlet/splitfile_insert -- inserts a FEC encoded splitfile > /servlet/splitfile_retrieve -- retrieves a fec encoded splitfile > /servlet/cool_thing_no_one_has_thought_of yet -- it's easy to add new stuff Sounds good. Except I'd like to have the URLs as short as possible, before they're set in stone. Nobody cares that this is a servlet internally, so let's just use /x/ or /_/ or /./ as the special prefix. Or (I already suggested that) drop the prefix and restrict servlets to have names easily distinguishable from freenet keys (we will lose the ability to make /gpl.txt mean /KSK at gpl.txt, but KSKs aren't as hip as they were in our youth, anyway). We will also have to think about access from freesite pages. Will links to /_/some_service be allowed? Redirects? Triggered javascript? This may make sense for some servlets and not for others. We could put the safe class under /s/ and unsafe ones under /u/. There could be a generalised version of the __CHECKED_HTTP_ handler named /s/u/, so that surfing to /s/u/dangerous will bring up a confirmation page, that links you to the real /u/dangerous. __CHECKED_HTTP__freenetproject.org will get /s/u/http/freenetproject.org The safeness of a servlet need not be encoded in its name (and if it isn't it is easier to reconfigure locally). Every servlet could register whether it is safe (overridable from the config file). fproxy will not barf on links to unsafe servlets, but rewrite them to checked links on the fly. -- Robbe -- next part -- A non-text attachment was scrubbed... Name: signature.ng Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/3e6a72f3/attachment.pgp>
[freenet-dev] FCP FEC Proposal -- in message body
Most of this sounds pretty good. Firstly, a stupid question - is there any reason to seperate "data blocks" and "check blocks"? As far as the FEC encoder/decoder knows, they're just blocks, right? I mean, that's the whole *point* of it (you need any k blocks of n in order to decode a file). Would make things simpler, conceptually, I think. This of course is based on assumptions from FEC implentations that I have seen, where the block size is constant... obviously, if your FEC implentation makes the distinction, then I guess it makes sense (translation from fishish: yeah, we need to support checkblocks for certain algorythms, but they don't make sense for onion's) Anyhow, the other point I wished to make, is that from looking at your information, it seems like it would be far more convinent still to just call the onionnetworks library direclty - okay, yeah, I see the usefulness of this for providing access to people who don't have/don't want to have bindings to this, but it just seems like an unnessesary layer of abstraction to me. But perhaps I am on crack. > III. Changes to SplitFile metadata format. > > 0) Deprecate the BlockSize field, since check blocks are not necessarily the > same size as data blocks and blocks may be different sizes across segments. I strongly disagree with this - if we want to support this case, it is better to then have a seperate set of metadata for each segment and specify the block/check size in each one. This information is very useful to have for reasons of memory allocation and the like. Other than that, however, All of that being said, this all looks okay to me on my initial reading. - fish p.s. I included the following in the original draft of this email, but considered it offtopic and hence seperated it out from the main email. However, I include it here because it is interesting and semi-relevent: > For a given maximum block size, some FEC algorithms can only > practically handle files up to a certain maximum size. The design > uses segmentation to handle this case. Large files are divided into > smaller segments and FEC is only done on a per segment basis. This > compromise provides a least limited redundancy for large files. Should we be perhaps looking for a library which doens't suffer from these problems as much as onion's library, however? The thing is, the whole usefulness of FEC is for big files, you know... I know that we do have to deal with reality instead of would-be-nice's, however, but it is something to think about. The other problem is, that as you stripe like this, the amount of redundancy is, of course, reduced significantly, however you already knew that. However, people writing bad algorythms for downloading files (block 1,2,3,4 etc in order) could make this problem even worse. (As a side note to this, I have been wondering if an AWT based freenet download manager would be a useful thing to code/have... any thoughts on this? Heh, of course, this would require me to learn how to communicate with freenet from java, but i can't be that hard, can it? :-p) Anyhow, I'm sure you knew all of this... just restating it for my own benefit, don't mind me :). I'll look into the alternative libraries myself over the next few days... there's nothing to stop us having two encoders, given that the facilities are there, and let the best codec win :-p. ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Fproxy insert
On Thu, Sep 12, 2002 at 10:31:58AM +0200, Marco A. Calamari wrote: > None follow Ian's call for opinion about the > fproxy insertion capability removal. > > My opinion about it is that was a bad(TM) idea. > > I cannot imagine a good technical reason for that. > > From a normal user point of view, this fact transform > a read/write media in a readonly media; something that > the RIAA and other organization from the Dark Side > can just dream of. > However, you were never able to insert a *site* by fproxy. This would be a useful feature, no? :) > It is unreasonable that a normal user install and understand > other programs just to have the basic feature of writing a file > on Freenet; maybe the only good change would be to put a size > limit on that. > We don't need to put a size limit on. We only need to change some HTML. Ian seems reasonably adept at this, he broke it, and I'm busy with more important things :). > An artist friend of mine, that was partecipating to "Ars Electronica" > exibition in Lintz, showing an installation that use a Freenet gateway, > had major problems because she lost the ability to insert > without warning; I downgrade a node for her, but this was just > a stop-gap solution. > > So I strongly suggest to reenable some kind of Fproxy in fred, > at least as option. > > FWIW. Ciao. Marco > > > -- > * Marco A. Calamari marco at freenetproject.org * > > il Progetto Freenet - segui il coniglio bianco > the Freenet Project - follow the white rabbit > > > -- Matthew Toseland mtoseland at blueyonder.co.uk amphibian at sourceforge.net Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/bbda5481/attachment.pgp>
[freenet-dev] FCP FEC Proposal -- in message body
On Thu, Sep 12, 2002 at 07:37:32PM +1000, fish wrote: > > Most of this sounds pretty good. > > Firstly, a stupid question - is there any reason to seperate "data blocks" > and "check blocks"? As far as the FEC encoder/decoder knows, they're just > blocks, right? I mean, that's the whole *point* of it (you need any k > blocks of n in order to decode a file). Would make things simpler, > conceptually, I think. This of course is based on assumptions from FEC > implentations that I have seen, where the block size is > constant... obviously, if your FEC implentation makes the distinction, > then I guess it makes sense (translation from fishish: yeah, we need to > support checkblocks for certain algorythms, but they don't make sense for > onion's) > > Anyhow, the other point I wished to make, is that from looking at your > information, it seems like it would be far more convinent still to just > call the onionnetworks library direclty - okay, yeah, I see the usefulness > of this for providing access to people who don't have/don't want to have > bindings to this, but it just seems like an unnessesary layer of > abstraction to me. But perhaps I am on crack. Hmmm, maybe. It's not well documented afaics? Anyway, higher level commands (with reasonable status reporting, and some sort of keepalives or a way to reconnect to a running command, and it only terminate when explicitly told to), would make clients easier to write, but this looks useful. > > > III. Changes to SplitFile metadata format. > > > > 0) Deprecate the BlockSize field, since check blocks are not necessarily the > > same size as data blocks and blocks may be different sizes across segments. > > I strongly disagree with this - if we want to support this case, it is > better to then have a seperate set of metadata for each segment and > specify the block/check size in each one. This information is very useful > to have for reasons of memory allocation and the like. > > Other than that, however, All of that being said, this all looks okay to > me on my initial reading. > > - fish > > p.s. I included the following in the original draft of this email, but > considered it offtopic and hence seperated it out from the main > email. However, I include it here because it is interesting and > semi-relevent: > > > For a given maximum block size, some FEC algorithms can only > > practically handle files up to a certain maximum size. The design > > uses segmentation to handle this case. Large files are divided into > > smaller segments and FEC is only done on a per segment basis. This > > compromise provides a least limited redundancy for large files. > > Should we be perhaps looking for a library which doens't suffer from these > problems as much as onion's library, however? The thing is, the whole > usefulness of FEC is for big files, you know... I know that we do have to > deal with reality instead of would-be-nice's, however, but it is something > to think about. > > The other problem is, that as you stripe like this, the amount of > redundancy is, of course, reduced significantly, however you already knew > that. However, people writing bad algorythms for downloading files (block > 1,2,3,4 etc in order) could make this problem even worse. (As a side > note to this, I have been wondering if an AWT based freenet download > manager would be a useful thing to code/have... any thoughts on > this? Heh, of course, this would require me to learn how to communicate > with freenet from java, but i can't be that hard, can it? :-p) Please do. Do NOT use swing. And include more detail on the status of the individual blocks, eg bytes downloaded/total progress bar, time this block idle for etc. Would be very cool. > > Anyhow, I'm sure you knew all of this... just restating it for my own > benefit, don't mind me :). I'll look into the alternative libraries > myself over the next few days... there's nothing to stop us having two > encoders, given that the facilities are there, and let the best codec win > :-p. > > > -- Matthew Toseland mtoseland at blueyonder.co.uk amphibian at sourceforge.net Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/fa75bf71/attachment.pgp>
[freenet-dev] Cleaning up fproxy
On Thu, Sep 12, 2002 at 02:24:28AM -0400, Gianni Johansson wrote: > I think we should break froxy up into separate servlets bound together on the > same port with Tavin's MultipleHttpServletContainer. The hard cross port > redirects are ugly. Once everything is running on the same port redirects > are less ugly. > > / -- with no arguments would map to Ian's infolet main page >-- with a valid URI, maps to a leaner and meaner fproxy-lite servlet that > handles normal requests and inserts but redirects splitfile > inserts /requests to he servlets below. > /servlet/splitfile_insert -- inserts a FEC encoded splitfile > /servlet/splitfile_retrieve -- retrieves a fec encoded splitfile > /servlet/cool_thing_no_one_has_thought_of yet -- it's easy to add new stuff We might want to make checked_http a separate servlet too. > > This would make the code more modular and more easily expandable. > > Think webcam servlet. edition freesite checker servlet etc. etc. > > -- gj -- Matthew Toseland mtoseland at blueyonder.co.uk amphibian at sourceforge.net Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/b780a2ef/attachment.pgp>
[freenet-dev] Fproxy insert
On Thursday 12 September 2002 08:42, you wrote: > However, you were never able to insert a *site* by fproxy. This would be > a useful feature, no? :) This is exactly the kind of feature that doesn't belong in a bloated monolithic fproxy, but which could easily be implemented as a separate servlet. --gj ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] FCP FEC Proposal -- in message body
On Thursday 12 September 2002 08:40, you wrote: > > Anyhow, the other point I wished to make, is that from looking at your > > information, it seems like it would be far more convinent still to just > > call the onionnetworks library direclty - okay, yeah, I see the > > usefulness of this for providing access to people who don't have/don't > > want to have bindings to this, but it just seems like an unnessesary > > layer of abstraction to me. But perhaps I am on crack. > > Hmmm, maybe. It's not well documented afaics? Anyway, higher level > commands (with reasonable status reporting, and some sort of keepalives > or a way to reconnect to a running command, and it only terminate when > explicitly told to), would make clients easier to write, but this looks > useful. Status messages might be a useful addition. "keepalive", "reconnect" == stateful. You don't want to do that. One of the reasons that FCP is easy is that it is stateless. --gj ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] FCP FEC Proposal -- in message body
On Thursday 12 September 2002 05:37, you wrote: > Most of this sounds pretty good. > > Firstly, a stupid question - is there any reason to seperate "data blocks" > and "check blocks"? As far as the FEC encoder/decoder knows, they're just > blocks, right? I mean, that's the whole *point* of it (you need any k > blocks of n in order to decode a file). Would make things simpler, > conceptually, I think. This of course is based on assumptions from FEC > implentations that I have seen, where the block size is > constant... obviously, if your FEC implentation makes the distinction, > then I guess it makes sense (translation from fishish: yeah, we need to > support checkblocks for certain algorythms, but they don't make sense for > onion's) It's good to make the distinction because if you manage to get all the data blocks you don't need to decode at all. > > Anyhow, the other point I wished to make, is that from looking at your > information, it seems like it would be far more convinent still to just > call the onionnetworks library direclty - okay, yeah, I see the usefulness > of this for providing access to people who don't have/don't want to have > bindings to this, but it just seems like an unnessesary layer of > abstraction to me. But perhaps I am on crack. Client writers just don't want to deal with it. FEC support has been in CVS for almost a year and you are the first person who has attempted to write a client besides me. > > > III. Changes to SplitFile metadata format. > > > > 0) Deprecate the BlockSize field, since check blocks are not necessarily > > the same size as data blocks and blocks may be different sizes across > > segments. > > I strongly disagree with this - if we want to support this case, it is > better to then have a seperate set of metadata for each segment and > specify the block/check size in each one. This information is very useful > to have for reasons of memory allocation and the like. You can still get this info out of the SegmentHeader messages if you want it. I am leery of bloating the SplitFile metadata any more. > > Other than that, however, All of that being said, this all looks okay to > me on my initial reading. > > - fish > > p.s. I included the following in the original draft of this email, but > considered it offtopic and hence seperated it out from the main > email. However, I include it here because it is interesting and > > semi-relevent: > > For a given maximum block size, some FEC algorithms can only > > practically handle files up to a certain maximum size. The design > > uses segmentation to handle this case. Large files are divided into > > smaller segments and FEC is only done on a per segment basis. This > > compromise provides a least limited redundancy for large files. > > Should we be perhaps looking for a library which doens't suffer from these > problems as much as onion's library, however? The glass is half full man. The math works for files out to about 1G no problem even with segmentation for the current implementation. The OnionNetworks code works well for moderately sized files now! We should make it easily available. I will keep a similar plugin architecture so that new encoder/decoder implementations can be used if/when they are written. > The thing is, the whole > usefulness of FEC is for big files, you know... I know that we do have to > deal with reality instead of would-be-nice's, however, but it is something > to think about. > Segmenting reduces redundancy not striping. (see above) > The other problem is, that as you stripe like this, the amount of > redundancy is, of course, reduced significantly, however you already knew > that. However, people writing bad algorythms for downloading files (block > 1,2,3,4 etc in order) could make this problem even worse. (As a side > note to this, I have been wondering if an AWT based freenet download > manager would be a useful thing to code/have... any thoughts on > this? Heh, of course, this would require me to learn how to communicate > with freenet from java, but i can't be that hard, can it? :-p) > > Anyhow, I'm sure you knew all of this... just restating it for my own > benefit, don't mind me :). I'll look into the alternative libraries > myself over the next few days... there's nothing to stop us having two > encoders, given that the facilities are there, and let the best codec win. Cool. > > :-p. > > ___ > devl mailing list > devl at freenetproject.org > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Fproxy insert
None follow Ian's call for opinion about the fproxy insertion capability removal. My opinion about it is that was a bad(TM) idea. I cannot imagine a good technical reason for that. From a normal user point of view, this fact transform a read/write media in a readonly media; something that the RIAA and other organization from the Dark Side can just dream of. It is unreasonable that a normal user install and understand other programs just to have the basic feature of writing a file on Freenet; maybe the only good change would be to put a size limit on that. An artist friend of mine, that was partecipating to "Ars Electronica" exibition in Lintz, showing an installation that use a Freenet gateway, had major problems because she lost the ability to insert without warning; I downgrade a node for her, but this was just a stop-gap solution. So I strongly suggest to reenable some kind of Fproxy in fred, at least as option. FWIW. Ciao. Marco -- * Marco A. Calamari marco at freenetproject.org * il Progetto Freenet - segui il coniglio bianco the Freenet Project - follow the white rabbit ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Fproxy insert
On Thu, Sep 12, 2002 at 10:31:58AM +0200, Marco A. Calamari wrote: > I cannot imagine a good technical reason for that. It was a usability reason, FProxy should be a tool for retrieval of information from Freenet, there are much better tools for inserting information into Freenet. > From a normal user point of view, this fact transform > a read/write media in a readonly media; something that > the RIAA and other organization from the Dark Side > can just dream of. Not at all, it just means that they can't insert with FProxy, I honestly doubt many people were actually using FProxy for inserting stuff (more likely they will use fcptools, freeweb, or frost). > It is unreasonable that a normal user install and understand > other programs just to have the basic feature of writing a file > on Freenet; maybe the only good change would be to put a size > limit on that. Writing a file on Freenet isn't of much use unless you have a mechanism to tell people where the file is. This means that Frost is probably a better option, or if you want to insert a website, you will be using fcptools anyway. > An artist friend of mine, that was partecipating to "Ars Electronica" > exibition in Lintz, showing an installation that use a Freenet gateway, > had major problems because she lost the ability to insert > without warning; I downgrade a node for her, but this was just > a stop-gap solution. Well, insert functionality is back for the time being. Ian. -- Ian Clarkeian at freenetproject.org Founder & Coordinator, The Freenet Projecthttp://freenetproject.org/ Chief Technology Officer, Uprizer Inc. http://www.uprizer.com/ Personal Homepage http://locut.us/ -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/55b2a7fb/attachment.pgp>
[freenet-dev] Insert functionality returns to FProxy
As of this email, FProxy can insert again (I have updated the snapshot). Ian. -- Ian Clarkeian at freenetproject.org Founder & Coordinator, The Freenet Projecthttp://freenetproject.org/ Chief Technology Officer, Uprizer Inc. http://www.uprizer.com/ Personal Homepage http://locut.us/ -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/b0d0c055/attachment.pgp>
[freenet-dev] FCP FEC Proposal -- in message body
FCP FEC Proposal rev. 1.0 giannijohansson at attbi.com 20020912 I. INTRODUCTION: This proposal presents a set of new FCP commands that can be used to encode and decode files using forward error correction (FEC). FEC is a way of encoding packetized data files with extra error recovery information which can be used to reconstruct lost packets. In this document I will refer to the packets containing data as "data blocks" and those containing error recovery information as "check blocks". One of the objectives of this design is to separate FEC encoding and decoding from inserting and retrieving the data and check blocks to/from Freenet. By separating encoding and decoding from insertion and retrieval I sidestep the problem of having to hold FCP connections open while waiting for large amounts of data to be fetched from / inserted into Freenet. For a given maximum block size, some FEC algorithms can only practically handle files up to a certain maximum size. The design uses segmentation to handle this case. Large files are divided into smaller segments and FEC is only done on a per segment basis. This compromise provides a least limited redundancy for large files. II. Assumptions This proposal doesn't specify any particular FEC algorithm. However the following assumptions are implicit in the design: A. For a given segment with k data blocks and n - k check blocks, it must be possible to decode all k data blocks from any k of n data or check blocks. B. Encoder and decoder implementations must be completely specified by an implementation name and a file length. No other parameters can be required to instantiate the encoder or decoder. C. Within a segment all data blocks must be the same size and all check blocks must be the same size. The check block and data block sizes are not required to be the same however. Smaller trailing blocks must be zero padded to the required length. D. The encoder may ask for extra trailing data blocks. These extra blocks must contain zeros. II. Proposed FCP FEC commands convention: All numbers are hexadecimal A. Helper messages, SegmentHeader and BlockMap A SegmentHeader message contains all the information necessary to FEC encode or decode a segment of a file. SegmentHeaders may contain FEC implementation specific fields. They are guaranteed to contain the documented fields given in the example SegmentHeader message below: SegmentHeader FECAlgorithm=OnionFEC_a_1_2 // The FEC implementation name FileLength=17 // Total file length Offset=0 // Offset from the start of the file BlockCount=6 // Number of data blocks BlockSize=4 // Data block size CheckBlockCount=3 // Number of check blocks CheckBlockSize=3 // Check block size Segments=1// Total number of segments SegmentNum=0 // Index of the current segment BlocksRequired=6 // Blocks required to decode this segment EndMessage Client code should not rely on any undocumented fields. BlockMap messages are used to list the CHKs of the data and check blocks for a segment. Here's an example: BlockMap Block.0=freenet:CHK at p2ISvZPkCwbY62xciJb~KrsOCTsSAwI,jGonMeCCz1GCHde5bc1t~w Block.1=freenet:CHK at 1z8CubDNzLEfNfuTYM4NVJAUxU4SAwI,5cxWki4YzWyKP0s3g9~Vow Block.2=freenet:CHK@~VW7XskmHcJMFlmG6l2c7jkTOnkSAwI,Il2ztTbQImZvVlsnuDq-8Q Block.3=freenet:CHK at A-qK8GWofXd9JOxb4fHfVMHAUawSAwI,2D5~Mm~MjAfup3edGXy6Eg Block.4=freenet:CHK at r-FhUu444LxUIUGi5BMuEVGM4nQSAwI,J7HpLvPscLyW3Sc6Nq2S5g Check.0=freenet:CHK at rLdCwOXO7PAv6BDpm21ThdIwmnkSAwI,4ZX2inJ7gg0EectTxPYRSg Check.1=freenet:CHK at EjEg1UHWsAfHHMQmRbxe2ToY0RQSAwI,xjJCPsCxpnw9lyNI2VBRGA EndMessage B. FECSegmentFile The FECSegmentFile message is used to generate the segment headers necessary to encode a file of a given length with a specified FEC algorithm. FECSegmentFile AlgoName=OnionFEC_a_1_2 FileLength=ABC123 EndMessage If this command is successful one or more SegmentHeader messages are sent in order of ascending SegmentNumber. The client can detect when the last segment has been sent by checking the SegmentNumber and Segments field of each received SegmentHeader. On failure a Failed message is sent. C. FECEncodeSegment The FECEncodeSegment message is used to create check blocks for a segment of a file. The RequestedList field contains a comma delimited list of the requested check blocks. If the list is empty or omitted completely all the check blocks are sent. The SegmentHeader for the requested segment must sent as data in the trailing field of the FECEncodeSegment message, preceding the raw segment data to encode. FECEncodeSegment [RequestedList=0,A,F] DataLength= + Data < SegmentHeader > < raw data > If the encode request is successful, the server sends a BlocksEncoded confirmation message, followed by DataChunk messages for the encoded blocks. Check blocks are
[freenet-dev] Cleaning up fproxy
I think we should break froxy up into separate servlets bound together on the same port with Tavin's MultipleHttpServletContainer. The hard cross port redirects are ugly. Once everything is running on the same port redirects are less ugly. / -- with no arguments would map to Ian's infolet main page -- with a valid URI, maps to a leaner and meaner fproxy-lite servlet that handles normal requests and inserts but redirects splitfile inserts /requests to he servlets below. /servlet/splitfile_insert -- inserts a FEC encoded splitfile /servlet/splitfile_retrieve -- retrieves a fec encoded splitfile /servlet/cool_thing_no_one_has_thought_of yet -- it's easy to add new stuff This would make the code more modular and more easily expandable. Think webcam servlet. edition freesite checker servlet etc. etc. -- gj ___ devl mailing list devl at freenetproject.org http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Adding FCP commands for FEC
Here's a description of the fcp commands I want to add to do FEC encoding/decoding. I have a working implementation with encoder and decoder plugins based on the OnionNetworks Java FEC library. It has JNI native code support for Windows, x86 linux and Solaris. Unfortunately, I don't have FEC downloading working in fproxy with the new code yet. Otherwise I might have been tempted to do a "full speed ahead and damn the torpedos checkin" ;-) Don't get too hung up on the issue of segments. Files under 128M won't get segmented with the my plugin implementation. I played around with FCP level 2 like single call "just do it and tell me when your done" designs. I didn't do it that way, because I don't think that will work for real applications. FEC SplitFile inserting / downloading is a resource intensive process. You need fine grain control at the application level. What do people think about the proposal? --gj p.s. I am not igoring the source issue for the OnionNetworks util code. Somehow that got lost between rev's. Justin is supposed to be coming out with an update release with full source pretty soon. If worse came to worse we could just decompile. We have full human readable source to all the tricky stuff. -- next part -- A non-text attachment was scrubbed... Name: FCP-FEC.txt Type: text/english Size: 9272 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20020912/4af2ff75/attachment.bin>
[freenet-dev] Fproxy insert
None follow Ian's call for opinion about the fproxy insertion capability removal. My opinion about it is that was a bad(TM) idea. I cannot imagine a good technical reason for that. From a normal user point of view, this fact transform a read/write media in a readonly media; something that the RIAA and other organization from the Dark Side can just dream of. It is unreasonable that a normal user install and understand other programs just to have the basic feature of writing a file on Freenet; maybe the only good change would be to put a size limit on that. An artist friend of mine, that was partecipating to Ars Electronica exibition in Lintz, showing an installation that use a Freenet gateway, had major problems because she lost the ability to insert without warning; I downgrade a node for her, but this was just a stop-gap solution. So I strongly suggest to reenable some kind of Fproxy in fred, at least as option. FWIW. Ciao. Marco -- * Marco A. Calamari [EMAIL PROTECTED] * il Progetto Freenet - segui il coniglio bianco the Freenet Project - follow the white rabbit ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] FCP FEC Proposal -- in message body
Most of this sounds pretty good. Firstly, a stupid question - is there any reason to seperate data blocks and check blocks? As far as the FEC encoder/decoder knows, they're just blocks, right? I mean, that's the whole *point* of it (you need any k blocks of n in order to decode a file). Would make things simpler, conceptually, I think. This of course is based on assumptions from FEC implentations that I have seen, where the block size is constant... obviously, if your FEC implentation makes the distinction, then I guess it makes sense (translation from fishish: yeah, we need to support checkblocks for certain algorythms, but they don't make sense for onion's) Anyhow, the other point I wished to make, is that from looking at your information, it seems like it would be far more convinent still to just call the onionnetworks library direclty - okay, yeah, I see the usefulness of this for providing access to people who don't have/don't want to have bindings to this, but it just seems like an unnessesary layer of abstraction to me. But perhaps I am on crack. III. Changes to SplitFile metadata format. 0) Deprecate the BlockSize field, since check blocks are not necessarily the same size as data blocks and blocks may be different sizes across segments. I strongly disagree with this - if we want to support this case, it is better to then have a seperate set of metadata for each segment and specify the block/check size in each one. This information is very useful to have for reasons of memory allocation and the like. Other than that, however, All of that being said, this all looks okay to me on my initial reading. - fish p.s. I included the following in the original draft of this email, but considered it offtopic and hence seperated it out from the main email. However, I include it here because it is interesting and semi-relevent: For a given maximum block size, some FEC algorithms can only practically handle files up to a certain maximum size. The design uses segmentation to handle this case. Large files are divided into smaller segments and FEC is only done on a per segment basis. This compromise provides a least limited redundancy for large files. Should we be perhaps looking for a library which doens't suffer from these problems as much as onion's library, however? The thing is, the whole usefulness of FEC is for big files, you know... I know that we do have to deal with reality instead of would-be-nice's, however, but it is something to think about. The other problem is, that as you stripe like this, the amount of redundancy is, of course, reduced significantly, however you already knew that. However, people writing bad algorythms for downloading files (block 1,2,3,4 etc in order) could make this problem even worse. (As a side note to this, I have been wondering if an AWT based freenet download manager would be a useful thing to code/have... any thoughts on this? Heh, of course, this would require me to learn how to communicate with freenet from java, but i can't be that hard, can it? :-p) Anyhow, I'm sure you knew all of this... just restating it for my own benefit, don't mind me :). I'll look into the alternative libraries myself over the next few days... there's nothing to stop us having two encoders, given that the facilities are there, and let the best codec win :-p. ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Cleaning up fproxy
On Thu, Sep 12, 2002 at 02:24:28AM -0400, Gianni Johansson wrote: I think we should break froxy up into separate servlets bound together on the same port with Tavin's MultipleHttpServletContainer. The hard cross port redirects are ugly. Once everything is running on the same port redirects are less ugly. / -- with no arguments would map to Ian's infolet main page -- with a valid URI, maps to a leaner and meaner fproxy-lite servlet that handles normal requests and inserts but redirects splitfile inserts /requests to he servlets below. /servlet/splitfile_insert -- inserts a FEC encoded splitfile /servlet/splitfile_retrieve -- retrieves a fec encoded splitfile /servlet/cool_thing_no_one_has_thought_of yet -- it's easy to add new stuff We might want to make checked_http a separate servlet too. This would make the code more modular and more easily expandable. Think webcam servlet. edition freesite checker servlet etc. etc. -- gj -- Matthew Toseland [EMAIL PROTECTED] [EMAIL PROTECTED] Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. msg03975/pgp0.pgp Description: PGP signature
Re: [freenet-dev] FCP FEC Proposal -- in message body
On Thu, Sep 12, 2002 at 07:37:32PM +1000, fish wrote: Most of this sounds pretty good. Firstly, a stupid question - is there any reason to seperate data blocks and check blocks? As far as the FEC encoder/decoder knows, they're just blocks, right? I mean, that's the whole *point* of it (you need any k blocks of n in order to decode a file). Would make things simpler, conceptually, I think. This of course is based on assumptions from FEC implentations that I have seen, where the block size is constant... obviously, if your FEC implentation makes the distinction, then I guess it makes sense (translation from fishish: yeah, we need to support checkblocks for certain algorythms, but they don't make sense for onion's) Anyhow, the other point I wished to make, is that from looking at your information, it seems like it would be far more convinent still to just call the onionnetworks library direclty - okay, yeah, I see the usefulness of this for providing access to people who don't have/don't want to have bindings to this, but it just seems like an unnessesary layer of abstraction to me. But perhaps I am on crack. Hmmm, maybe. It's not well documented afaics? Anyway, higher level commands (with reasonable status reporting, and some sort of keepalives or a way to reconnect to a running command, and it only terminate when explicitly told to), would make clients easier to write, but this looks useful. III. Changes to SplitFile metadata format. 0) Deprecate the BlockSize field, since check blocks are not necessarily the same size as data blocks and blocks may be different sizes across segments. I strongly disagree with this - if we want to support this case, it is better to then have a seperate set of metadata for each segment and specify the block/check size in each one. This information is very useful to have for reasons of memory allocation and the like. Other than that, however, All of that being said, this all looks okay to me on my initial reading. - fish p.s. I included the following in the original draft of this email, but considered it offtopic and hence seperated it out from the main email. However, I include it here because it is interesting and semi-relevent: For a given maximum block size, some FEC algorithms can only practically handle files up to a certain maximum size. The design uses segmentation to handle this case. Large files are divided into smaller segments and FEC is only done on a per segment basis. This compromise provides a least limited redundancy for large files. Should we be perhaps looking for a library which doens't suffer from these problems as much as onion's library, however? The thing is, the whole usefulness of FEC is for big files, you know... I know that we do have to deal with reality instead of would-be-nice's, however, but it is something to think about. The other problem is, that as you stripe like this, the amount of redundancy is, of course, reduced significantly, however you already knew that. However, people writing bad algorythms for downloading files (block 1,2,3,4 etc in order) could make this problem even worse. (As a side note to this, I have been wondering if an AWT based freenet download manager would be a useful thing to code/have... any thoughts on this? Heh, of course, this would require me to learn how to communicate with freenet from java, but i can't be that hard, can it? :-p) Please do. Do NOT use swing. And include more detail on the status of the individual blocks, eg bytes downloaded/total progress bar, time this block idle for etc. Would be very cool. Anyhow, I'm sure you knew all of this... just restating it for my own benefit, don't mind me :). I'll look into the alternative libraries myself over the next few days... there's nothing to stop us having two encoders, given that the facilities are there, and let the best codec win :-p. -- Matthew Toseland [EMAIL PROTECTED] [EMAIL PROTECTED] Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. msg03976/pgp0.pgp Description: PGP signature
Re: [freenet-dev] Fproxy insert
On Thu, Sep 12, 2002 at 10:31:58AM +0200, Marco A. Calamari wrote: None follow Ian's call for opinion about the fproxy insertion capability removal. My opinion about it is that was a bad(TM) idea. I cannot imagine a good technical reason for that. From a normal user point of view, this fact transform a read/write media in a readonly media; something that the RIAA and other organization from the Dark Side can just dream of. However, you were never able to insert a *site* by fproxy. This would be a useful feature, no? :) It is unreasonable that a normal user install and understand other programs just to have the basic feature of writing a file on Freenet; maybe the only good change would be to put a size limit on that. We don't need to put a size limit on. We only need to change some HTML. Ian seems reasonably adept at this, he broke it, and I'm busy with more important things :). An artist friend of mine, that was partecipating to Ars Electronica exibition in Lintz, showing an installation that use a Freenet gateway, had major problems because she lost the ability to insert without warning; I downgrade a node for her, but this was just a stop-gap solution. So I strongly suggest to reenable some kind of Fproxy in fred, at least as option. FWIW. Ciao. Marco -- * Marco A. Calamari [EMAIL PROTECTED] * il Progetto Freenet - segui il coniglio bianco the Freenet Project - follow the white rabbit -- Matthew Toseland [EMAIL PROTECTED] [EMAIL PROTECTED] Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/11/02. msg03977/pgp0.pgp Description: PGP signature
Re: [freenet-dev] FCP FEC Proposal -- in message body
On Thursday 12 September 2002 05:37, you wrote: Most of this sounds pretty good. Firstly, a stupid question - is there any reason to seperate data blocks and check blocks? As far as the FEC encoder/decoder knows, they're just blocks, right? I mean, that's the whole *point* of it (you need any k blocks of n in order to decode a file). Would make things simpler, conceptually, I think. This of course is based on assumptions from FEC implentations that I have seen, where the block size is constant... obviously, if your FEC implentation makes the distinction, then I guess it makes sense (translation from fishish: yeah, we need to support checkblocks for certain algorythms, but they don't make sense for onion's) It's good to make the distinction because if you manage to get all the data blocks you don't need to decode at all. Anyhow, the other point I wished to make, is that from looking at your information, it seems like it would be far more convinent still to just call the onionnetworks library direclty - okay, yeah, I see the usefulness of this for providing access to people who don't have/don't want to have bindings to this, but it just seems like an unnessesary layer of abstraction to me. But perhaps I am on crack. Client writers just don't want to deal with it. FEC support has been in CVS for almost a year and you are the first person who has attempted to write a client besides me. III. Changes to SplitFile metadata format. 0) Deprecate the BlockSize field, since check blocks are not necessarily the same size as data blocks and blocks may be different sizes across segments. I strongly disagree with this - if we want to support this case, it is better to then have a seperate set of metadata for each segment and specify the block/check size in each one. This information is very useful to have for reasons of memory allocation and the like. You can still get this info out of the SegmentHeader messages if you want it. I am leery of bloating the SplitFile metadata any more. Other than that, however, All of that being said, this all looks okay to me on my initial reading. - fish p.s. I included the following in the original draft of this email, but considered it offtopic and hence seperated it out from the main email. However, I include it here because it is interesting and semi-relevent: For a given maximum block size, some FEC algorithms can only practically handle files up to a certain maximum size. The design uses segmentation to handle this case. Large files are divided into smaller segments and FEC is only done on a per segment basis. This compromise provides a least limited redundancy for large files. Should we be perhaps looking for a library which doens't suffer from these problems as much as onion's library, however? The glass is half full man. The math works for files out to about 1G no problem even with segmentation for the current implementation. The OnionNetworks code works well for moderately sized files now! We should make it easily available. I will keep a similar plugin architecture so that new encoder/decoder implementations can be used if/when they are written. The thing is, the whole usefulness of FEC is for big files, you know... I know that we do have to deal with reality instead of would-be-nice's, however, but it is something to think about. Segmenting reduces redundancy not striping. (see above) The other problem is, that as you stripe like this, the amount of redundancy is, of course, reduced significantly, however you already knew that. However, people writing bad algorythms for downloading files (block 1,2,3,4 etc in order) could make this problem even worse. (As a side note to this, I have been wondering if an AWT based freenet download manager would be a useful thing to code/have... any thoughts on this? Heh, of course, this would require me to learn how to communicate with freenet from java, but i can't be that hard, can it? :-p) Anyhow, I'm sure you knew all of this... just restating it for my own benefit, don't mind me :). I'll look into the alternative libraries myself over the next few days... there's nothing to stop us having two encoders, given that the facilities are there, and let the best codec win. Cool. :-p. ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] FCP FEC Proposal -- in message body
On Thursday 12 September 2002 08:40, you wrote: Anyhow, the other point I wished to make, is that from looking at your information, it seems like it would be far more convinent still to just call the onionnetworks library direclty - okay, yeah, I see the usefulness of this for providing access to people who don't have/don't want to have bindings to this, but it just seems like an unnessesary layer of abstraction to me. But perhaps I am on crack. Hmmm, maybe. It's not well documented afaics? Anyway, higher level commands (with reasonable status reporting, and some sort of keepalives or a way to reconnect to a running command, and it only terminate when explicitly told to), would make clients easier to write, but this looks useful. Status messages might be a useful addition. keepalive, reconnect == stateful. You don't want to do that. One of the reasons that FCP is easy is that it is stateless. --gj ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Fproxy insert
On Thursday 12 September 2002 08:42, you wrote: However, you were never able to insert a *site* by fproxy. This would be a useful feature, no? :) This is exactly the kind of feature that doesn't belong in a bloated monolithic fproxy, but which could easily be implemented as a separate servlet. --gj ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Insert functionality returns to FProxy
As of this email, FProxy can insert again (I have updated the snapshot). Ian. -- Ian Clarke[EMAIL PROTECTED] Founder Coordinator, The Freenet Projecthttp://freenetproject.org/ Chief Technology Officer, Uprizer Inc. http://www.uprizer.com/ Personal Homepage http://locut.us/ msg03981/pgp0.pgp Description: PGP signature
Re: [freenet-dev] Fproxy insert
On Thu, Sep 12, 2002 at 10:31:58AM +0200, Marco A. Calamari wrote: I cannot imagine a good technical reason for that. It was a usability reason, FProxy should be a tool for retrieval of information from Freenet, there are much better tools for inserting information into Freenet. From a normal user point of view, this fact transform a read/write media in a readonly media; something that the RIAA and other organization from the Dark Side can just dream of. Not at all, it just means that they can't insert with FProxy, I honestly doubt many people were actually using FProxy for inserting stuff (more likely they will use fcptools, freeweb, or frost). It is unreasonable that a normal user install and understand other programs just to have the basic feature of writing a file on Freenet; maybe the only good change would be to put a size limit on that. Writing a file on Freenet isn't of much use unless you have a mechanism to tell people where the file is. This means that Frost is probably a better option, or if you want to insert a website, you will be using fcptools anyway. An artist friend of mine, that was partecipating to Ars Electronica exibition in Lintz, showing an installation that use a Freenet gateway, had major problems because she lost the ability to insert without warning; I downgrade a node for her, but this was just a stop-gap solution. Well, insert functionality is back for the time being. Ian. -- Ian Clarke[EMAIL PROTECTED] Founder Coordinator, The Freenet Projecthttp://freenetproject.org/ Chief Technology Officer, Uprizer Inc. http://www.uprizer.com/ Personal Homepage http://locut.us/ msg03982/pgp0.pgp Description: PGP signature
Re: [freenet-dev] FCP FEC Proposal -- in message body
Gianni Johansson [EMAIL PROTECTED] writes: It's good to make the distinction because if you manage to get all the data blocks you don't need to decode at all. Prefering data blocks to check blocks will restrict the usefulness of FEC, as it will make the check blocks less popular, thus less fetchable, in extreme degenerating to non-FEC splitfiles. I'm with Fish in the belief that all blocks should be requested in random order, and with equal probability. Decoding is a really cheap operation relative to the fetching of the blocks from Freenet. We shouldn't shun it and make FEC less efficient in the long run. Client writers just don't want to deal with it. It's certainly a class harder than fetching a monolithic key. I also found that accurate documentation more accessible than the source is nonexistent. FEC support has been in CVS for almost a year and you are the first person who has attempted to write a client besides me. Well, could have something to do with Freenet being largely unusable for retrieval of bigger files in that time period. With the current comeback, more freesites are a-coming, with more FEC splitfiles, so the need for clients supporting that rises as well. The FCP access will surely be a good thing. As long as direct access is still possible ... and I don't see why that should be removed. -- Robbe signature.ng Description: PGP signature
Re: [freenet-dev] Cleaning up fproxy
Gianni Johansson [EMAIL PROTECTED] writes: / -- with no arguments would map to Ian's infolet main page -- with a valid URI, maps to a leaner and meaner fproxy-lite servlet that handles normal requests and inserts but redirects splitfile inserts /requests to he servlets below. /servlet/splitfile_insert -- inserts a FEC encoded splitfile /servlet/splitfile_retrieve -- retrieves a fec encoded splitfile /servlet/cool_thing_no_one_has_thought_of yet -- it's easy to add new stuff Sounds good. Except I'd like to have the URLs as short as possible, before they're set in stone. Nobody cares that this is a servlet internally, so let's just use /x/ or /_/ or /./ as the special prefix. Or (I already suggested that) drop the prefix and restrict servlets to have names easily distinguishable from freenet keys (we will lose the ability to make /gpl.txt mean [EMAIL PROTECTED], but KSKs aren't as hip as they were in our youth, anyway). We will also have to think about access from freesite pages. Will links to /_/some_service be allowed? Redirects? Triggered javascript? This may make sense for some servlets and not for others. We could put the safe class under /s/ and unsafe ones under /u/. There could be a generalised version of the __CHECKED_HTTP_ handler named /s/u/, so that surfing to /s/u/dangerous will bring up a confirmation page, that links you to the real /u/dangerous. __CHECKED_HTTP__freenetproject.org will get /s/u/http/freenetproject.org The safeness of a servlet need not be encoded in its name (and if it isn't it is easier to reconfigure locally). Every servlet could register whether it is safe (overridable from the config file). fproxy will not barf on links to unsafe servlets, but rewrite them to checked links on the fly. -- Robbe signature.ng Description: PGP signature
Re: [freenet-dev] Fproxy insert
On Thu, 12 Sep 2002, Ian Clarke wrote: Not at all, it just means that they can't insert with FProxy, I honestly doubt many people were actually using FProxy for inserting stuff (more likely they will use fcptools, freeweb, or frost). You'd be surprised, actully, at how many people were using it to insert static files. Writing a file on Freenet isn't of much use unless you have a mechanism to tell people where the file is. This means that Frost is probably a better option, or if you want to insert a website, you will be using fcptools anyway. it's called a telephone, iip, etc ;) - fishy fishy fish ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] FCP FEC Proposal -- in message body
On Thu, 12 Sep 2002, Gianni Johansson wrote: It's good to make the distinction because if you manage to get all the data blocks you don't need to decode at all. bob the angry flower says wrong,m wrong, wrong ,wrong, where did you learn that??! wrong!!! :-p. Seriously, this is a very bad thing to be encouraging, for two reasons: a) the checkblocks will fall oiff freenet - you should never be retrieving this data significantly! (i wrote about that in the documentation i wrote, I won't reepat that here) Client writers just don't want to deal with it. FEC support has been in CVS for almost a year and you are the first person who has attempted to write a client besides me. I think that's got more to do with the fact that there was no documentation, and hence it took me a solid week to work everything out to the point where it worked where I tested it (and yes, before anyone asks, I did try just asking GJ first. 4 times (this was back last march. I also asked about the fproxy changes, to the same lack of result). it seems that any email I send to GJ gets redirected to /dev/null :(). And then I *still* got it wrong and didn't know until fproxy was patched, which I found out about when people started bitching about my tools breaking freenet. Hell, I had to write *documentation* to get your attention :-p. Talk about sinking to a new low :-p. Should we be perhaps looking for a library which doens't suffer from these problems as much as onion's library, however? The glass is half full man. The math works for files out to about 1G no problem even with segmentation for the current implementation. The OnionNetworks code works well for moderately sized files now! We should make it easily available. I agree with this... i wasn't trying to suggest otherwise, it was just random musings. I get emails asking me why it's still hard to insert iso's into freenet :-p (hey, did you know that onionnetwork's c library will bluescreen a winxp box when you feed it an iso? :-p) Segmenting reduces redundancy not striping. (see above) okay, i got the two confused ^_^ - from fish with love. ___ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/devl
