On Tue, Jan 14, 2003 at 08:31:17PM -0600, Edgar Friendly wrote: > Gianni Johansson <giannijohansson at attbi.com> writes: > > > I want to make the following changes to the SplitFile metadata spec: > > > > 0) Optional CHK checksum of the entire file. > > SplitFile.CheckSum=CHK at blahblah blah > > > Why not just put this into the Info.* metadata? There's already this > place for it. It's a good thing to allow authors to put the Checksum > in the metadata, but I'm not certain that it belongs in the > SplitFile.* section. > > > 1) Specification of cipher used for all block CHKs > > SplitFile.BlockCipher=TwoFish > > > I'm opposed to this for the reason that it will make splitfile > processing a node-only procedure, when it rightfully lies outside the > realm of the node's influence. > > > 2) Explicit requirement that block CHKs contain data only. > > i.e. so if you know the cipher and you know the block size and you > > have the data, you should be able to reinsert the block without any further > > information. > > > This is a hard one. It is/should be *extremely* recommended that all > CHKs contain only data and no metadata. I admit that I can't see a > use for splitfile pieces to have any sort of metadata, but that > doesn't mean that there isn't one. If someone inserts a splitfile > with metadata in the CHK blocks, they deserve having their splitfile > be not easily re-insertable, but I don't see any need to label what > they did as "invalid" or as being outside the spec. Non-redundant splitfiles might well be constructed from existing files with content in their own right, which could be accessed directly, with metadata.. but mostly, this metadata is provided by manifests, or SSK redirects; so the only case it makes sense for is assembling files from CHKs which happen to have metadata in them, or from files where we provide the full, original URI rather than the CHK - either it's a non-manifest small SSK, or a DBR'd file... hmm. Bottom line is I see no good reason to make this restriction on non-FEC splitfiles, which I don't think we should deprecate because they have some interesting possible uses (fproxy-side inlining, essentially)... maybe we should warn on large non-redundant splitfiles with lots of chunks though, just to piss off Frost users :). > > > 3) Explicit requirement that trailing blocks are zero padded to the block > > size. > > > I'm against padding trailing blocks; It's needed for FEC, it's a waste > of space otherwise. Why can't the client do the padding with nulls for the last data block, if it's not already done? > > > 4) Explicit segmentation in the SplitFile metadata. > > The FEC encoding stuff already "segments" large files. FEC encoding is only > > done over the data in each segment. I want to make this explicit in the > > SplitFile metadata. i.e. > > > I see what you're doing and why you're doing it, I'm just not sure > that I like having the extra layer for regular splitfiles. Splitfiles > are a really simple thing, and should be kept as simple as possible. > > <SNIP example> > > > > What do people think? > > > > -- gj > > > As an alternative to explicit segmentation, why not just build a > heirarchical splitfile? i.e. have as your main document a > meta-splitfile, where each block is a FEC splitfile. This keeps > normal splitfiles as they should be, while the structure of the > metadata is able to accurately represent what's going on at the lower > levels. It would be good to support splitfiles inside non-redundant splitfiles, at some point. BUT explicit segmentation inside a splitfile makes more sense than splitting it out into another splitfile, simply because of the delay involved in fetching another file in series. > > Thelema > -- > E-mail: thelema314 at swbell.net Raabu and Piisu > GPG 1024D/36352AAB fpr:756D F615 B4F3 BFFC 02C7 84B7 D8D7 6ECE 3635 2AAB >
-- Matthew Toseland toad at amphibian.dyndns.org/amphibian at users.sourceforge.net Freenet/Coldstore open source hacker. Contracted full time by Freenet Project Inc. http://freenetproject.org/ ICTHUS. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20030115/af7e9916/attachment.pgp>
