[freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[guido]

Am Donnerstag 23 April 2009 14:48:43 schrieb Matthew Toseland:
> I would really appreciate input on option 2 i.e. how much of a problem are
> long CHKs?

If CHK key lengths as they are now are not bad enough to keep people from 
using them, then making them 50% longer won't be, either.

Besides, making the pathname of the file a mandatory part of the key is already 
having larger impact on average key lengths then this would.

[freenet-dev] Easy top block duplication: Content MultiplicationKeys

[Daniel Cheng]

On Thu, Apr 23, 2009 at 8:48 PM, Matthew Toseland
 wrote:
> On Thursday 23 April 2009 07:17:36 xor wrote:
>>
>> >
>> > The filename is ignored. This will make the Frost folk happy.
>> >
>> Now that DOES sound good. Especially if you consider that it is sometimes
>> difficult to restore the original filename, for example if someone uploaded
>> the file using Linux and the file name containes characters which are not
>> allowed on windows and you want to re-insert from windows
>
> Uhm, what archaic version of Windows are you using that doesn't have unicode
> support?!
>

Try colon (:), this is not valid in windows file name.
Or just use FAT32.

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[VolodyA! V Anarhist]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Robert Hailey wrote:
> 
> On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:
> 
>> Robert Hailey wrote:
>>>
>>> Perhaps there is an easier solution?
>>>
>>> How about extending the chk logic into an alternate-chk-key (ACK?);  
>>> that simply adds 0.25 to the expected location (for routing and  
>>> storage).
>>>
>>> So when you insert the top block, put it in as a chk and an ack (no  
>>> extra uri's neccesary). When you go to fetch it, if the chk does not  
>>> work, try the ack variant of the same key.
>>
>> At the moment each node on the path can verify that the data sent by
>> previous hop corresponds to what it ought to; How would that work with
>> your proposed solution?
>>
>> NextGen$
> 
> Sorta like this...
> 
> package freenet.keys;
> 
> public class ASKKey extends NodeCHK {
> public double toNormalizedDouble() {
> return (super.toNormalizedDouble()+0.25)%1.0;
> }
> }
> 
> The only difference is where any node would look for it. This would not
> be exposed to the client. My idea is that any chk could be converted to
> an alternate-location-finding-key just by type (which surely would mean
> a different fetch-command, e.g. fetchCHK/fetchACK...).
> 
> There would be no difference in handling, the only difference would be
> how the target-routing-location is identified from the key (the same as
> CHK plus a constant [mod 1.0]). After all, the mapping from the key to
> the small-world location is open to interpretation...
> 
> --
> Robert Hailey

But from the perspective of forwarding nodes top block is just another CHK
block, just like any other one. So you'd have to forward the information that it
is a special block through the network.

   - Volodya

- --
http://freedom.libsyn.com/   Echo of Freedom, Radical Podcast
http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal
http://www.freedomporn.org/  Freedom Porn, anarchist and activist smut

 "None of us are free until all of us are free."~ Mihail Bakunin
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknwyxcACgkQuWy2EFICg+3KVwCfSwZywD9HH9boKzoGQCzSAohK
G74An0yhV48V9B+Cv5jhfWyMUJLqyh4z
=DKa2
-END PGP SIGNATURE-

[freenet-dev] Current uservoice top 5 (20 node barrier)

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 14:53:45 schrieb Arne Babenhauserheide:
> Am Mittwoch 22 April 2009 14:38:29 schrieb Matthew Toseland:
> > I don't know. IMHO 150 is probably too much, have you spoken privately to
> > all these people?
>
> I think all people I know privately, including school and university,
> account for maybe 100 to 120 people. Of them I'd trust about 40 as
> connections :)

To test the 150 people limit for the "monkey space", my wife and I just did a 
test by counting all people we know personally and care about (we were walking 
to the supermarket, so we had some free time :) ). 

I got to over 200, and she got to over 300, so 150 is a bit too low as upper 
boundary, I'd say (the article didn't say 150, by the way, but 100 to 230 for 
95%). 

But of these 200 I'd trust only 50 enough that they'd keep the information 
private that I run freenet - she'd trust about 30 people enough (less tech 
savvy community :) ). 

So for pretty communicative people who mostly know tech geeks 150 to 200 
friends might be a good bet (for most tech geeks the number is likely to be 
lower, though). 

(I hope you don't mind my wording. Geek is no insult for me, and neither is 
Nerd)


Just wanted to give you the data :) 


I have a question, though: Would it help routing if people would exchange refs 
with other people who have similar interests? If yes: Can you guess how much 
it would help? 


Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/a5e0ed6c/attachment.pgp>

[freenet-dev] Our current web interface and its usability

[Mike Bush]

Matthew Toseland wrote:
> On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:
>   
>> On Wed, Apr 22, 2009 at 1:17 PM, xor  wrote:
>>
>> 
>>>  "Node" should really be replaced with "Client" *everywhere* because
>>> client is the common word.
>>>   
>> Is it?  When I talk to non-techies about a "client" they think I'm referring
>> to the person that employs a lawyer.  I think the least confusing term to
>> use in this context may be "software".
>>
>> 
> Very clumbersome. How would you translate "Your node is downloading this page 
> from Freenet" ? "The Freenet software running on your computer is downloading 
> this page from the Freenet network" ?
>   

"The Freenet software running on your computer" is probably what I would use to 
describe what "node" means to non-techy users.
Couldn't it just use "Your computer is downloading this page from Freenet", 
that's what people want to know, just that the stuff they are downloading will 
be on their computer soon.


>>> I also think that some words should be replaced. I did that for a few
>>> things in my last few commits. Further, for example I suggested to replace
>>> "Key" with "Download link" on the downloads & uploads page.
>>>   
>> Yes, or perhaps just "link".
>>
>> Ian.
>> 
>> 
>>
>> ___
>> Devl mailing list
>> Devl at freenetproject.org
>> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

[freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[VolodyA! V Anarhist]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Toseland wrote:
> I would really appreciate input on option 2 i.e. how much of a problem are 
> long CHKs?

If Long CHKs will become too much of a problem, and people won't mind their
content being spoofed, then people will start using KSK...

- Volodya

- --
http://freedom.libsyn.com/   Echo of Freedom, Radical Podcast
http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal
http://www.freedomporn.org/  Freedom Porn, anarchist and activist smut

 "None of us are free until all of us are free."~ Mihail Bakunin
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknwvCUACgkQuWy2EFICg+00GgCgi0KdVgk3trar9NcfGYpNaOd2
7CAAn1mTjWDU5y6DIS3qZzHLaSLMmHdk
=7d0m
-END PGP SIGNATURE-

[freenet-dev] Easy top block duplication: Content MultiplicationKeys

[VolodyA! V Anarhist]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Toseland wrote:
> On Thursday 23 April 2009 07:17:36 xor wrote:
>>> The filename is ignored. This will make the Frost folk happy.
>>>
>> Now that DOES sound good. Especially if you consider that it is sometimes
>> difficult to restore the original filename, for example if someone uploaded
>> the file using Linux and the file name containes characters which are not
>> allowed on windows and you want to re-insert from windows
> 
> Uhm, what archaic version of Windows are you using that doesn't have unicode 
> support?!
> 
> 
> 
> 
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

It's not Windows, it's Windows Explorer that sets extra limitations on the file
name. You can name files pretty much anything right now, but you need software
that'll let you do that.

For example you can write a quick C app which writes to a file starting with the
full stop, but you can't create a new file starting with it in Explorer. Please
don't ask me why.

 - Volodya


- --
http://freedom.libsyn.com/   Echo of Freedom, Radical Podcast
http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal
http://www.freedomporn.org/  Freedom Porn, anarchist and activist smut

 "None of us are free until all of us are free."~ Mihail Bakunin
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknwu68ACgkQuWy2EFICg+0+NACgh33i9gD6Lni14/3PowVt1j4i
dZgAoKbQZWzeWktqRHBso1G3l5aUy7PN
=KLc4
-END PGP SIGNATURE-

[freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Juiceman]

On Thu, Apr 23, 2009 at 8:48 AM, Matthew Toseland
 wrote:
> I would really appreciate input on option 2 i.e. how much of a problem are
> long CHKs?
>
> On Thursday 23 April 2009 03:03:00 Matthew Toseland wrote:
>> Argh, no, this doesn't work, because the pubkey is known, and there is no
> way
>> for the node to verify that the content is in fact valid. An attacker can
> not
>> only cause collisions, he can preemptively block known content by inserting
>> bogus data to where it would be posted.
>>
>> So what does that leave?
>>
>> On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
>> > 1) Duplicate the top block with SSKs:
>> > SSK,3@,,/ would insert e.g.
>> > SSK@,,/-N for a series of N's.
>>
>> Make cryptokey the hash of the content, avoids different-content attacks.
>>
>> > 2) CHKs with extra routing keys:
>> > CHK@
>>
>> Replace crypto key with the hash of the content, and derive the crypto key
> for
>> each block:
>> cryptokey_N = H ( H(data) + "N" )
>>
>> 3) Reinserting the top block when a download has finished, and also at some
>> point after inserting the data. IMHO this may help, but it does not address
>> the core problem. We need redundancy over *different areas of the keyspace*.
>>
>> 4) An FCP command to reinsert a file, given a known top-block key. If we
>> cannot reconstruct as-is, we fail, but if we can, we reinsert that block
>> (exactly as is, like a binary blob), and the rest of the CHK-based blocks
>> under it.
>>
>> Maybe a combination of 1) and 4) ?
>>
>> On first insert:
>> User inserts the data as DSK,3@/filename.
>> The node creates a random pubkey, and hashes the content of the top block to
>> get the cryptokey. It inserts each block, and returns
>> DSK,3@,,/filename
>>
>> The filename could be ignored if we want.
>>
>> On reinsert:
>> The original URI must be specified, and have been downloaded. If it is kept
> on
>> the download queue then we can simply click a button to reinsert.
>>
>> For SSKs:
>> User inserts the data as SSK,3@,,/
>>
>> We insert to SSK@,,/- for N in 0, 1,
> 2.
>>
>> Since the URI is not content-derived, there is the possibility of the
> inserter
>> will insert different content to each slot. AFAICS this cannot be avoided.
>>
>> The major disadvantages are:
>> - When reinserting we MUST know the original URI.
>> - There is no way to heal the alternate top-blocks.
>>
>> IMHO the latter is more serious than the former, but full convergence would
> be
>> ideal. Option 2 has neither of these problems, but does have very long URIs.
>> Mostly keys are copied and pasted, so maybe that isn't a big problem?
>>
>> If people are happy with option 2 (very long CHKs), I can implement it
>> reasonably quickly...
>> >
>> > Neither of these schemes is acceptable IMHO. The former allows for an
>> attacker
>> > to insert different content to different keys, and get some info about
>> > targets that way, and it is non-convergent, not allowing for reinserts.
> The
>> > latter doubles the length of the already way too long CHKs.
>> >
>> > I propose a new key type which is convergent, has URIs shorter than
> existing
>> > CHKs, and any number of duplicate blocks: the Content Multiplication Key
>> (for
>> > want of a better name, alternatives are welcome).
>> >
>> > DETAILS:
>> >
>> > Insert the splitfile normally, except for the top block. The top block
> must
>> > fit inside a 1KB SSK payload.
>> >
>> > Hash the content of the top block:
>> >
>> > hash of content: H(data)
>> >
>> > Create a private key:
>> >
>> > privkey = MAKE_PRIVKEY ( H ( H(data) + "0" ) )
>> >
>> > Create the base crypto key:
>> >
>> > ckey_base = H ( H ( data + "1" ) )
>> >
>> > Create a series of crypto keys:
>> >
>> > ckey_N = H ( ckey_base + "N" )
>> >
>> > Insert to a series of SSKs:
>> >
>> > SSK@,,
>> >
>> > Announce the key:
>> >
>> > UMK,N@,/
>> > (Where N is the number of copies inserted)
>> >
>> > The filename is ignored. This will make the Frost folk happy.
>> >
>>
>>
>>
>
>
>
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
>

Extra long CHK's are no problem at all.  As many have pointed out, we
all just cut and paste or clink a link.  I say go for it.  Option 2
with option 3 would make me very happy.

-- 
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin

[freenet-dev] Current uservoice top 5 (20 node barrier)

[Matthew Toseland]

On Wednesday 22 April 2009 16:52:57 Robert Hailey wrote:
> 
> On Apr 22, 2009, at 7:38 AM, Matthew Toseland wrote:
> 
> > On Wednesday 22 April 2009 02:09:21 Arne Babenhauserheide wrote:
> >> So we get to the question, what a freenet contact is: A friend or an
> >> aquaintance.
> >>
> >> If you look at myspace and similar sites, you'll see people with  
> >> hundreds of
> >> "friends" which in truth are aquaintances.
> >>
> >> Also the question arises, which number of friends will be efficient  
> >> for
> >> freenets algorithm: How many people have similar interest?
> >
> > In terms of routing, the main issues are:
> > - There must be a small-world network. Clearly random automatically  
> > selected
> > participants will not form a small-world network, but acquaintances  
> > probably
> > do. I repeat, randomly selected people through any automated  
> > mechanism WILL
> > BREAK ROUTING!
> 
> Are we even sure of that???
> 
> I know that the whole routing algorithm is based on small world  
> theory. However, if we load up a sim of a large randomly connected  
> network, would freenet not operate on it? Perhaps it would sort out  
> effective and usable locations anyway (simply on graph theory)?

No. It would not work well. We demonstrated this pretty clearly with 
#freenet-refs !
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/3436298c/attachment.pgp>

[freenet-dev] Current uservoice top 5 (20 node barrier)

[Matthew Toseland]

On Wednesday 22 April 2009 18:53:48 Arne Babenhauserheide wrote:
> Am Mittwoch 22 April 2009 15:53:39 schrieb Matthew Toseland:
> > I don't understand why you want to run a jabber server. Surely announcing
> > to your jabber contacts that you are interested in ref exchange would be
> > sufficient, and would be client level?
> 
> I don't mean announcing to your jabber contacts that you'd like to exchange 
> refs (though that would be a nice option, too). 
> 
> I mean having all freenet refs as jabber contacts automatically, and having 
a 
> freenet jabber ID based on your ref. 
> 
> That way communication with peers would become far easier (it can just rely 
on 
> jabbers own encryption - you're open to your peers anyway - and it can 
> automatically use all features people develop for jabber). 
> 
> That way I'd just add my freenet jabber ID in my noderef with 127.0.0.1 as 
> server, and freenet would add all my refs as jabber contacts for that 
freenet 
> jabber ID. 

Encryption is irrelevant, any such traffic should be tunneled with regular 
freenet traffic and thus be invisible from the network. But it might be an 
interesting idea for a plugin.
> 
> Best wishes, 
> Arne
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/8da026bb/attachment.pgp>

[freenet-dev] Our current web interface and its usability

[Ian Clarke]

On Thu, Apr 23, 2009 at 3:05 PM, Robert Hailey
 wrote:
> Yea, but Matthew's language has a more technically-accurate flavor (as
> "your node" implies the distributed nature of freenet, whereas
> "freenet is downloading" makes it sound like a monolithic entity).

Technically accurate flavor is secondary to making newbies feel
comfortable.  Newbies are made comfortable by using plain English, and
avoiding terms of art.

Here is a good resource on examples of jargon versus plain English:

  http://www.plainenglish.co.uk/examples/before_and_after.html

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: ian at uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674

[freenet-dev] Our current web interface and its usability

[Ian Clarke]

On Thu, Apr 23, 2009 at 10:28 AM, Matthew Toseland
 wrote:
>> Is it? ?When I talk to non-techies about a "client" they think I'm referring
>> to the person that employs a lawyer. ?I think the least confusing term to
>> use in this context may be "software".
>>
> Very clumbersome. How would you translate "Your node is downloading this page
> from Freenet" ? "The Freenet software running on your computer is downloading
> this page from the Freenet network" ?

"Freenet is downloading this page from the Freenet network"

Speaking plain English isn't brain surgery (or it shouldn't be)!

Ian.


-- 
Ian Clarke
CEO, Uprizer Labs
Email: ian at uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Florent Daigniere]

Robert Hailey wrote:
> On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:
> 
>> It has recently come to our attention that the problem with data  
>> persistence
>> is usually that the top block has fallen out or that the few nodes  
>> with the
>> block are never online at the same time as the person who wants to  
>> fetch it.
>> Hence we need to duplicate the top block. So far there have been two  
>> schemes:
>> 1) Duplicate the top block with SSKs:
>> SSK,3@,,/ would insert e.g.
>> SSK@,,/-N for a series of N's.
>> 2) CHKs with extra routing keys:
>> CHK@,,,> key>,
>>
>> Neither of these schemes is acceptable IMHO. The former allows for  
>> an attacker
>> to insert different content to different keys, and get some info about
>> targets that way, and it is non-convergent, not allowing for  
>> reinserts. The
>> latter doubles the length of the already way too long CHKs.
>>
>> I propose a new key type which is convergent, has URIs shorter than  
>> existing
>> CHKs, and any number of duplicate blocks: the Content Multiplication  
>> Key (for
>> want of a better name, alternatives are welcome).
> 
> Perhaps there is an easier solution?
> 
> How about extending the chk logic into an alternate-chk-key (ACK?);  
> that simply adds 0.25 to the expected location (for routing and  
> storage).
> 
> So when you insert the top block, put it in as a chk and an ack (no  
> extra uri's neccesary). When you go to fetch it, if the chk does not  
> work, try the ack variant of the same key.

At the moment each node on the path can verify that the data sent by 
previous hop corresponds to what it ought to; How would that work with 
your proposed solution?

NextGen$

[freenet-dev] Non-convergent encryption kills easy filesharing

[Daniel Cheng]

On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
 wrote:
> After a long conversation with p0s, I am fairly sure that our decision at last
> year's summit to use non-convergent encryption for splitfiles (i.e. a
> different set of blocks each time) in order to largely solve our security
> problems will make filesharing on Freenet much less convenient.
>
> Other points:
> - The problem with persistence of data is frequently the top block. This can
> be solved by duplicating it. This is easy for SSKs, but for CHKs would
> require quite long URIs. This may be acceptable given the likely data
> persistence gains.
> - We must deal with the short segments problem.
> - Node references should be tolerant of losing newlines and gaining new ones.
>
> I had hoped that we could release a 0.8.1 with solutions to the data
> persistence problems and non-convergent encryption, but it looks like we will
> need to wait for 0.9 with tunnels and bloom filter sharing. :(

I believe the persistence would be improved with tunnels
  -- blocks can be insert from different paths and cache differently.

[freenet-dev] Our current web interface and its usability

[Matthew Toseland]

On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:
> On Wed, Apr 22, 2009 at 1:17 PM, xor  wrote:
> 
> >  "Node" should really be replaced with "Client" *everywhere* because
> > client is the common word.
> 
> Is it?  When I talk to non-techies about a "client" they think I'm referring
> to the person that employs a lawyer.  I think the least confusing term to
> use in this context may be "software".
> 
Very clumbersome. How would you translate "Your node is downloading this page 
from Freenet" ? "The Freenet software running on your computer is downloading 
this page from the Freenet network" ?
> 
> > I also think that some words should be replaced. I did that for a few
> > things in my last few commits. Further, for example I suggested to replace
> > "Key" with "Download link" on the downloads & uploads page.
> 
> Yes, or perhaps just "link".
> 
> Ian.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/08146157/attachment.pgp>

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Evan Daniel]

On Thu, Apr 23, 2009 at 4:01 PM, Robert Hailey
 wrote:
>
> On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:
>
> Robert Hailey wrote:
>
> Perhaps there is an easier solution?
>
> How about extending the chk logic into an alternate-chk-key (ACK?);
>
> that simply adds 0.25 to the expected location (for routing and
>
> storage).
>
> So when you insert the top block, put it in as a chk and an ack (no
>
> extra uri's neccesary). When you go to fetch it, if the chk does not
>
> work, try the ack variant of the same key.
>
> At the moment each node on the path can verify that the data sent by
> previous hop corresponds to what it ought to; How would that work with
> your proposed solution?
>
> NextGen$
>
> Sorta like this...
> package freenet.keys;
> public class ASKKey extends NodeCHK {
> public double toNormalizedDouble() {
> return (super.toNormalizedDouble()+0.25)%1.0;
> }
> }
> The only difference is where any node would look for it. This would not be
> exposed to the client. My idea is that any chk could be converted to an
> alternate-location-finding-key just by type (which surely would mean a
> different fetch-command, e.g. fetchCHK/fetchACK...).
> There would be no difference in handling, the only difference would be how
> the target-routing-location is identified from the key (the same as CHK plus
> a constant [mod 1.0]). After all, the mapping from the key to the
> small-world location is open to interpretation...
> --
> Robert Hailey

I suggested the obvious extension of this on IRC.  Instead of simple
searching at location + 0.25, you search at location + n/N, where n is
which copy of the block you're looking for, and N is the number of
copies inserted.

Toad didn't like this because it makes top blocks identifiable to
everyone on the routing path, and involves network-level changes.  The
other approaches can be implemented at a higher level as a translation
before handing a normal CHK request to the network.

Evan Daniel

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 3:11 PM, Evan Daniel wrote:
>
> I suggested the obvious extension of this on IRC.  Instead of simple
> searching at location + 0.25, you search at location + n/N, where n is
> which copy of the block you're looking for, and N is the number of
> copies inserted.
>
> Toad didn't like this because it makes top blocks identifiable to
> everyone on the routing path, and involves network-level changes.  The
> other approaches can be implemented at a higher level as a translation
> before handing a normal CHK request to the network.
>
> Evan Daniel

The problem...
How to find a 'top block'... either
(1) we have to already know about it on-fetch (long chk uri)
(2) it needs to be derived from existing fetch data (like my idea)
(3) a new mechanism needs to be made (like ssk-top-blocks/matthew's- 
multiplier idea)

At this point, I'm greatly favoring Matthew's idea of multi-content- 
hash/ssk keys.

However, some alternative names to try on...

Reliable Fetch Key  RFK
Reliable Hash Key   RHK
Multiple Hash Key   MHK
Multi-link Key  MLK
Matthew's Multiplier KeyMMK

On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:
> UMK,N@<H(data)>,/
> (Where N is the number of copies inserted)

And maybe with "N" in the extra data so as to not introduce a new  
syntax... it is still fetch-time data after all!

> MMK@<H(data)>,/


--
Robert Hailey

-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/a9c82dd0/attachment.html>

[freenet-dev] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 15:16:40 schrieb Matthew Toseland:
> Arguably nobody ever types CHKs even now, and copy and paste allows for
> fairly long keys. Thoughts?

You know what I think. 

The length of the key doesn't matter to me, because freesites already hide 
them in links, and otherwise I just copy-paste them. 

I didn't ever watch my downloads long enough to say where exactly they stop. I 
just start them, and if they didn't finish in a week, I remove them again. I 
don't trust my memory enough to say much about the state. Many didn't even 
start, but I'm not sure in which state they were...

Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 3:09 PM, VolodyA! V Anarhist wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Robert Hailey wrote:
>>
>> Sorta like this...
>>
>> package freenet.keys;
>>
>> public class ASKKey extends NodeCHK {
>> public double toNormalizedDouble() {
>> return (super.toNormalizedDouble()+0.25)%1.0;
>> }
>> }
>>
>> The only difference is where any node would look for it. This would  
>> not
>> be exposed to the client. My idea is that any chk could be  
>> converted to
>> an alternate-location-finding-key just by type (which surely would  
>> mean
>> a different fetch-command, e.g. fetchCHK/fetchACK...).
>>
>> There would be no difference in handling, the only difference would  
>> be
>> how the target-routing-location is identified from the key (the  
>> same as
>> CHK plus a constant [mod 1.0]). After all, the mapping from the key  
>> to
>> the small-world location is open to interpretation...
>>
>> --
>> Robert Hailey
>
> But from the perspective of forwarding nodes top block is just  
> another CHK
> block, just like any other one. So you'd have to forward the  
> information that it
> is a special block through the network.
>
>   - Volodya
>

True, I hadn't thought of that.

It is correctable, though, as via an option flag on the uri we can  
inverted the standard fetching procedure to be mostly alternates, and  
failover to the 'normal' routing location. Then the 'specialness' of  
the alternate request would fall back into the noise of standard chk  
fetching (eventually, after enough data is inserted, etc).

--
Robert Hailey

[freenet-dev] Solving "I queued it 2 weeks ago and it's still at0%" : are really long URIs a problem?

[xor]

> -Original Message-
> From: devl-bounces at freenetproject.org 
> [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland
> Sent: Thursday, April 23, 2009 3:17 PM
> To: support at freenetproject.org; Discussion of development issues
> Cc: Ian Clarke
> Subject: [freenet-dev] Solving "I queued it 2 weeks ago and 
> it's still at0%" : are really long URIs a problem?
> 
> Anecdotal evidence suggests that right now at least one third 
> of our content persistence problems boil down to this one 
> bug: "I added it 2 weeks ago and it still hasn't got past 0% 
> (0/1)". A new key type, DHKs (Duplicated Hash Keys), would 
> solve the problem, but the new keys would be twice as long as 
> current CHKs. Is this a problem? I would really appreciate 
> input from users, particularly those who upload and download files:
> - Is it a problem for the keys to be really long (twice as 
> long as current CHKs)? CHKs are copied and pasted, so maybe 
> not a problem?
> - Is it true that a great many downloads get stuck at 0% for 
> a long time, showing 0 blocks of 1 if you mouseover the percentage?

I think you should first of all figure out whether it isn't a db4o
introduced bug or even a problem which has been there before db4o.
Relying on something brand new (the db4o branch) which has been out in the
real world only for some days to decide implementation of a new key type is
really insane IMHO. We don't know whether the db4o branch doesn't have
hundreds of other bugs. And in general the debugging of the node has been
mostly on hold for long time due to your work on db4o.

This would not be the first bug which makes downloads hang at 0% which we
had.

Further I do not understand why you don't want to solve the problem with
tunnels. You told me that tunnels fix three problems:

1. they increase security
2.  well the other benefit is that it allows us to do bloom filter
sharing [17:07:58] 
3. tunnels would fix the top-block-not-reachable problem we are talking
about

So tunnels would fix 3 problems, this fixes 1 and tunnels will still be
necessary some day.

Why not just postpone the fix of the top-block-problem until we have
tunnels?

Why not make the node just automatically re-insert the top block upon
download? Thats easy to implement after all?

Why not debug first?

I don't understand why you would want to chose the annoying alternative of
implementing yet another key type if there are so many alternatives to it.


> 
> Example:
> CHK at 4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8Hxk
> JFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
> 
> -> (something like)
> 
> DHK at 97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8N
> ZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9D
> rDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--
> 8/chaosradio_142.mp3
> 
> GORY DETAILS:
> 
> Currently we use:
> CHK@,,
> 
> (Filenames afterwards are manifests, and therefore impact on the CHK)
> 
> The new key type would be:
> 
> DHK@,,, 3>,/
> 
> (A filename is mandatory, and is always ignored, so does not 
> impact on the rest of the key).
> 
> We might allow any number of routing keys from 2 upwards, for 
> more redundancy at the cost of a longer URI, but IMHO 3 is a 
> good default number.
> 
> You would get such a key when you insert a file as DHK at .
> 
> Arguably nobody ever types CHKs even now, and copy and paste 
> allows for fairly long keys. Thoughts?
> 

[freenet-dev] [freenet-cvs] r26829 - trunk/freenet/src/freenet/node/fcp

[Matthew Toseland]

On Wednesday 15 April 2009 07:43:14 j16sdiz at freenetproject.org wrote:
> Author: j16sdiz
> Date: 2009-04-15 06:43:12 + (Wed, 15 Apr 2009)
> New Revision: 26829
> 
> Modified:
>trunk/freenet/src/freenet/node/fcp/FCPClient.java
> Log:
> revert r26828: req.cacnel() in removeAll() not work as expected

Hmmm, what is the problem here? Clearly we are adding to toKill in order to 
mass-cancel?
> 
> Modified: trunk/freenet/src/freenet/node/fcp/FCPClient.java
> ===
> --- trunk/freenet/src/freenet/node/fcp/FCPClient.java 2009-04-15 06:35:17 
UTC (rev 26828)
> +++ trunk/freenet/src/freenet/node/fcp/FCPClient.java 2009-04-15 06:43:12 
UTC (rev 26829)
> @@ -466,8 +466,6 @@
>   if (persistenceType == ClientRequest.PERSIST_FOREVER)
>   
> container.ext().store(clientRequestsByIdentifier, 2);
>   }
> - for (ClientRequest req : toKill)
> - req.cancel();
>   }
>  
>   public ClientGet getCompletedRequest(FreenetURI key, ObjectContainer 
container) {
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/f49aaee8/attachment.pgp>

[freenet-dev] Easy top block duplication: ContentMultiplicationKeys

[xor]

> -Original Message-
> From: devl-bounces at freenetproject.org 
> [mailto:devl-bounces at freenetproject.org] On Behalf Of Matthew Toseland
> Sent: Thursday, April 23, 2009 2:48 PM
> To: Discussion of development issues
> Subject: Re: [freenet-dev] Easy top block duplication: 
> ContentMultiplicationKeys
> 
> On Thursday 23 April 2009 07:17:36 xor wrote:
> > 
> > >
> > > The filename is ignored. This will make the Frost folk happy.
> > >
> > Now that DOES sound good. Especially if you consider that it is 
> > sometimes difficult to restore the original filename, for 
> example if 
> > someone uploaded the file using Linux and the file name containes 
> > characters which are not allowed on windows and you want to 
> re-insert 
> > from windows
> 
> Uhm, what archaic version of Windows are you using that 
> doesn't have unicode support?!
> 

Linux allows stuff in filenames which are control characters for windows.

[freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

> -Original Message-
> From: arne_bab at web.de [mailto:arne_bab at web.de] 
> Sent: Thursday, April 23, 2009 11:21 AM
> To: devl at freenetproject.org
> Cc: xor
> Subject: Re: [freenet-dev] Non-convergent encryption kills 
> easy filesharing
> 
> Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
> > > -Original Message-
> > > From: devl-bounces at freenetproject.org 
> > > [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne 
> > > Babenhauserheide
> > > Sent: Thursday, April 23, 2009 9:14 AM
> > > To: devl at freenetproject.org
> > > Subject: Re: [freenet-dev] Non-convergent encryption kills easy 
> > > filesharing
> > >
> > > Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > > > block [16:39:31]  duplicating the top block can be done 
> > > > with SSKs very easily [16:39:40]  but with CHKs 
> it requires 
> > > > much longer URIs [16:39:43]  is that a problem?
> > > > [16:40:04]  how much longer?
> > > > [16:40:10]  CHK@,, -> 
> > > > CHK@,,, > > > key>, [16:40:29]  i.e. at least twice as long
> > >
> > > I stopped reading at about the middle, so it might be 
> that I missed 
> > > something, but why do you care if CHKs become longer?
> > >
> > > They are already so long that noone types them (I hope), and I 
> > > really don't care if the key I copy-paste has two times 
> the length.
> >
> > If they are exchanged via Freetalk it will bloat the messages.
> > Further, I was told to implement CHK messages in Freetalk because 
> > messages could not fit into SSK. So now Freetalk uses SSK message 
> > lists to publish CHK URI of the messages... If the CHK URI become 
> > insanely long then the message lists will be bloated very much and 
> > maybe only five messages or so will fit into a SSK message 
> list. Sucks.
> 
> So why don't you take the approach of filesystems and add an 
> option where the message list SSK can contain a link to a CHK 
> which contains all message CHK URIs?

1. Too complex.
2. The problem is only shifted to another place. The problem with message
lists is that they must fit as many message references into a single SSK
block as possible to gurantee high network efficiency. You cannot tell me to
solve the space problem by INCREASING the used space by adding an extra CHK
block.

> 
> (that's what ext2 does with inodes to support almost 
> arbitrarily large files, though each inodes is only 4k in size) 
> 
> If the efficiency of CHKs should rise by more than a factor 
> of 2 when using the large URI instead of non-republishable 
> URIs, freetalk will even profit from this (ping time wise). 
> 
> But I don't know the implementation details of freetalk, so I 
> don't know if this is a possible option for you. 
> 
> Best wishes,
> Arne
> --
> -- Ein W?rfel System: http://1w6.org - einfach saubere 
> (Rollenspiel-) Regeln.
> -- Infinite Hands: http://infinite-hands.draketo.de - singing 
> a part of the history of free software.
> -- My stuff: http://draketo.de - stories, songs, poems, 
> programs and stuff :)
> 
> -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Solving "I queued it 2 weeks ago and it's still at0%" : are really long URIs a problem?

[Matthew Toseland]

understand why you would want to chose the annoying alternative of
> implementing yet another key type if there are so many alternatives to it.

Because none of the alternatives solves the problem.
------ next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/f1f709b7/attachment.pgp>

[freenet-dev] Our current web interface and its usability

[Robert Hailey]

On Apr 23, 2009, at 2:22 PM, Mike Bush wrote:

> Matthew Toseland wrote:
>> On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:
>>
>>> On Wed, Apr 22, 2009 at 1:17 PM, xor  wrote:
>>>
>>>
 "Node" should really be replaced with "Client" *everywhere* because
 client is the common word.

>>> Is it?  When I talk to non-techies about a "client" they think I'm  
>>> referring
>>> to the person that employs a lawyer.  I think the least confusing  
>>> term to
>>> use in this context may be "software".
>>>
>>>
>> Very clumbersome. How would you translate "Your node is downloading  
>> this page
>> from Freenet" ? "The Freenet software running on your computer is  
>> downloading
>> this page from the Freenet network" ?
>>
>
> "The Freenet software running on your computer" is probably what I  
> would use to describe what "node" means to non-techy users.
> Couldn't it just use "Your computer is downloading this page from  
> Freenet", that's what people want to know, just that the stuff they  
> are downloading will be on their computer soon.

Yea, but Matthew's language has a more technically-accurate flavor (as  
"your node" implies the distributed nature of freenet, whereas  
"freenet is downloading" makes it sound like a monolithic entity).

--
Robert Hailey

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:

> Robert Hailey wrote:
>>
>>
>> Perhaps there is an easier solution?
>>
>> How about extending the chk logic into an alternate-chk-key (ACK?);
>> that simply adds 0.25 to the expected location (for routing and
>> storage).
>>
>> So when you insert the top block, put it in as a chk and an ack (no
>> extra uri's neccesary). When you go to fetch it, if the chk does not
>> work, try the ack variant of the same key.
>
> At the moment each node on the path can verify that the data sent by
> previous hop corresponds to what it ought to; How would that work with
> your proposed solution?
>
> NextGen$

Sorta like this...

package freenet.keys;

public class ASKKey extends NodeCHK {
public double toNormalizedDouble() {
return (super.toNormalizedDouble()+0.25)%1.0;
}
}

The only difference is where any node would look for it. This would  
not be exposed to the client. My idea is that any chk could be  
converted to an alternate-location-finding-key just by type (which  
surely would mean a different fetch-command, e.g. fetchCHK/fetchACK...).

There would be no difference in handling, the only difference would be  
how the target-routing-location is identified from the key (the same  
as CHK plus a constant [mod 1.0]). After all, the mapping from the key  
to the small-world location is open to interpretation...

--
Robert Hailey

-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/760595ae/attachment.html>

[freenet-dev] Solving "I queued it 2 weeks ago and it's still at 0%" : are really long URIs a problem?

[Matthew Toseland]

Anecdotal evidence suggests that right now at least one third of our content 
persistence problems boil down to this one bug: "I added it 2 weeks ago and 
it still hasn't got past 0% (0/1)". A new key type, DHKs (Duplicated Hash 
Keys), would solve the problem, but the new keys would be twice as long as 
current CHKs. Is this a problem? I would really appreciate input from users, 
particularly those who upload and download files:
- Is it a problem for the keys to be really long (twice as long as current 
CHKs)? CHKs are copied and pasted, so maybe not a problem?
- Is it true that a great many downloads get stuck at 0% for a long time, 
showing 0 blocks of 1 if you mouseover the percentage?

Example:
CHK at 
4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3

-> (something like)

DHK at 
97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3

GORY DETAILS:

Currently we use:
CHK@,,

(Filenames afterwards are manifests, and therefore impact on the CHK)

The new key type would be:

DHK@/

(A filename is mandatory, and is always ignored, so does not impact on the 
rest of the key).

We might allow any number of routing keys from 2 upwards, for more redundancy 
at the cost of a longer URI, but IMHO 3 is a good default number.

You would get such a key when you insert a file as DHK at .

Arguably nobody ever types CHKs even now, and copy and paste allows for fairly 
long keys. Thoughts?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/6b272130/attachment.pgp>

[freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 09:37:45 Daniel Cheng wrote:
> On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
>  wrote:
> > After a long conversation with p0s, I am fairly sure that our decision at 
last
> > year's summit to use non-convergent encryption for splitfiles (i.e. a
> > different set of blocks each time) in order to largely solve our security
> > problems will make filesharing on Freenet much less convenient.
> >
> > Other points:
> > - The problem with persistence of data is frequently the top block. This 
can
> > be solved by duplicating it. This is easy for SSKs, but for CHKs would
> > require quite long URIs. This may be acceptable given the likely data
> > persistence gains.
> > - We must deal with the short segments problem.
> > - Node references should be tolerant of losing newlines and gaining new 
ones.
> >
> > I had hoped that we could release a 0.8.1 with solutions to the data
> > persistence problems and non-convergent encryption, but it looks like we 
will
> > need to wait for 0.9 with tunnels and bloom filter sharing. :(
> 
> I believe the persistence would be improved with tunnels
>   -- blocks can be insert from different paths and cache differently.

Why? If specialisation works, and IMHO we have no reason to think it doesn't, 
then it will land up on the same nodes. The problem is simply that those 
nodes go offline.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/266dbd1f/attachment.pgp>

[freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

I would really appreciate input on option 2 i.e. how much of a problem are 
long CHKs?

On Thursday 23 April 2009 03:03:00 Matthew Toseland wrote:
> Argh, no, this doesn't work, because the pubkey is known, and there is no 
way 
> for the node to verify that the content is in fact valid. An attacker can 
not 
> only cause collisions, he can preemptively block known content by inserting 
> bogus data to where it would be posted.
> 
> So what does that leave?
> 
> On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
> > 1) Duplicate the top block with SSKs: 
> > SSK,3@,,/ would insert e.g. 
> > SSK@,,/-N for a series of N's.
> 
> Make cryptokey the hash of the content, avoids different-content attacks.
> 
> > 2) CHKs with extra routing keys:
> > CHK@
> 
> Replace crypto key with the hash of the content, and derive the crypto key 
for 
> each block:
> cryptokey_N = H ( H(data) + "N" )
> 
> 3) Reinserting the top block when a download has finished, and also at some 
> point after inserting the data. IMHO this may help, but it does not address 
> the core problem. We need redundancy over *different areas of the keyspace*.
> 
> 4) An FCP command to reinsert a file, given a known top-block key. If we 
> cannot reconstruct as-is, we fail, but if we can, we reinsert that block 
> (exactly as is, like a binary blob), and the rest of the CHK-based blocks 
> under it.
> 
> Maybe a combination of 1) and 4) ?
> 
> On first insert:
> User inserts the data as DSK,3@/filename.
> The node creates a random pubkey, and hashes the content of the top block to 
> get the cryptokey. It inserts each block, and returns 
> DSK,3@,,/filename
> 
> The filename could be ignored if we want.
> 
> On reinsert:
> The original URI must be specified, and have been downloaded. If it is kept 
on 
> the download queue then we can simply click a button to reinsert.
> 
> For SSKs:
> User inserts the data as SSK,3@,,/
> 
> We insert to SSK@,,/- for N in 0, 1, 
2.
> 
> Since the URI is not content-derived, there is the possibility of the 
inserter 
> will insert different content to each slot. AFAICS this cannot be avoided.
> 
> The major disadvantages are:
> - When reinserting we MUST know the original URI.
> - There is no way to heal the alternate top-blocks.
> 
> IMHO the latter is more serious than the former, but full convergence would 
be 
> ideal. Option 2 has neither of these problems, but does have very long URIs. 
> Mostly keys are copied and pasted, so maybe that isn't a big problem?
> 
> If people are happy with option 2 (very long CHKs), I can implement it 
> reasonably quickly...
> > 
> > Neither of these schemes is acceptable IMHO. The former allows for an 
> attacker 
> > to insert different content to different keys, and get some info about 
> > targets that way, and it is non-convergent, not allowing for reinserts. 
The 
> > latter doubles the length of the already way too long CHKs.
> > 
> > I propose a new key type which is convergent, has URIs shorter than 
existing 
> > CHKs, and any number of duplicate blocks: the Content Multiplication Key 
> (for 
> > want of a better name, alternatives are welcome).
> > 
> > DETAILS:
> > 
> > Insert the splitfile normally, except for the top block. The top block 
must 
> > fit inside a 1KB SSK payload. 
> > 
> > Hash the content of the top block:
> > 
> > hash of content: H(data)
> > 
> > Create a private key:
> > 
> > privkey = MAKE_PRIVKEY ( H ( H(data) + "0" ) )
> > 
> > Create the base crypto key:
> > 
> > ckey_base = H ( H ( data + "1" ) )
> > 
> > Create a series of crypto keys:
> > 
> > ckey_N = H ( ckey_base + "N" )
> > 
> > Insert to a series of SSKs:
> > 
> > SSK@,,
> > 
> > Announce the key:
> > 
> > UMK,N@<H(data)>,/
> > (Where N is the number of copies inserted)
> > 
> > The filename is ignored. This will make the Frost folk happy.
> > 
> 
> 
> 


-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/1e1b599b/attachment.pgp>

[freenet-dev] Easy top block duplication: Content MultiplicationKeys

[Matthew Toseland]

On Thursday 23 April 2009 07:17:36 xor wrote:
> 
> >
> > The filename is ignored. This will make the Frost folk happy.
> >
> Now that DOES sound good. Especially if you consider that it is sometimes
> difficult to restore the original filename, for example if someone uploaded
> the file using Linux and the file name containes characters which are not
> allowed on windows and you want to re-insert from windows

Uhm, what archaic version of Windows are you using that doesn't have unicode 
support?!
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/1ce6cd38/attachment.pgp>

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:

> It has recently come to our attention that the problem with data  
> persistence
> is usually that the top block has fallen out or that the few nodes  
> with the
> block are never online at the same time as the person who wants to  
> fetch it.
> Hence we need to duplicate the top block. So far there have been two  
> schemes:
> 1) Duplicate the top block with SSKs:
> SSK,3@,,/ would insert e.g.
> SSK@,,/-N for a series of N's.
> 2) CHKs with extra routing keys:
> CHK@,,, key>,
>
> Neither of these schemes is acceptable IMHO. The former allows for  
> an attacker
> to insert different content to different keys, and get some info about
> targets that way, and it is non-convergent, not allowing for  
> reinserts. The
> latter doubles the length of the already way too long CHKs.
>
> I propose a new key type which is convergent, has URIs shorter than  
> existing
> CHKs, and any number of duplicate blocks: the Content Multiplication  
> Key (for
> want of a better name, alternatives are welcome).

Perhaps there is an easier solution?

How about extending the chk logic into an alternate-chk-key (ACK?);  
that simply adds 0.25 to the expected location (for routing and  
storage).

So when you insert the top block, put it in as a chk and an ack (no  
extra uri's neccesary). When you go to fetch it, if the chk does not  
work, try the ack variant of the same key.

--
Robert Hailey

[freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
> > -Original Message-
> > From: devl-bounces at freenetproject.org
> > [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne
> > Babenhauserheide
> > Sent: Thursday, April 23, 2009 9:14 AM
> > To: devl at freenetproject.org
> > Subject: Re: [freenet-dev] Non-convergent encryption kills
> > easy filesharing
> >
> > Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > > block [16:39:31]  duplicating the top block can be done with
> > > SSKs very easily [16:39:40]  but with CHKs it requires much
> > > longer URIs [16:39:43]  is that a problem?
> > > [16:40:04]  how much longer?
> > > [16:40:10]  CHK@,, ->
> > > CHK@,,, > > key>, [16:40:29]  i.e. at least twice as long
> >
> > I stopped reading at about the middle, so it might be that I
> > missed something, but why do you care if CHKs become longer?
> >
> > They are already so long that noone types them (I hope), and
> > I really don't care if the key I copy-paste has two times the length.
>
> If they are exchanged via Freetalk it will bloat the messages.
> Further, I was told to implement CHK messages in Freetalk because messages
> could not fit into SSK. So now Freetalk uses SSK message lists to publish
> CHK URI of the messages... If the CHK URI become insanely long then the
> message lists will be bloated very much and maybe only five messages or so
> will fit into a SSK message list. Sucks.

So why don't you take the approach of filesystems and add an option where the 
message list SSK can contain a link to a CHK which contains all message CHK 
URIs? 

(that's what ext2 does with inodes to support almost arbitrarily large files, 
though each inodes is only 4k in size) 

If the efficiency of CHKs should rise by more than a factor of 2 when using 
the large URI instead of non-republishable URIs, freetalk will even profit 
from this (ping time wise). 

But I don't know the implementation details of freetalk, so I don't know if 
this is a possible option for you. 

Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Progress page implemented

[bo-le]

Am Donnerstag, 23. April 2009 03:34:46 schrieb Ian Clarke:
> I propose "software" as an alternative to "node".

IMHO this is the wrong way. 'software' is to common...

Freenet is a network, running on top of  'internet' (currently only 
on 'internet', but a WLAN transport plugin allows 'internet free' smash 
networks/clowds/nodes), so the 'freenet software' *is* a node.

I suggest to use 'node' (helps to easily differ from other 'software'),
but explain it shortly on the firstpage the user see, all over occurrences 
of 'node' have an shortexplaintooltip and/or a link to ExplainTerms.html#node

MfG
saces

[freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

> -Original Message-
> From: devl-bounces at freenetproject.org 
> [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne 
> Babenhauserheide
> Sent: Thursday, April 23, 2009 9:14 AM
> To: devl at freenetproject.org
> Subject: Re: [freenet-dev] Non-convergent encryption kills 
> easy filesharing
> 
> Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > block [16:39:31]  duplicating the top block can be done with 
> > SSKs very easily [16:39:40]  but with CHKs it requires much 
> > longer URIs [16:39:43]  is that a problem?
> > [16:40:04]  how much longer?
> > [16:40:10]  CHK@,, -> 
> > CHK@,,, > key>, [16:40:29]  i.e. at least twice as long
> 
> I stopped reading at about the middle, so it might be that I 
> missed something, but why do you care if CHKs become longer? 
> 
> They are already so long that noone types them (I hope), and 
> I really don't care if the key I copy-paste has two times the length. 

If they are exchanged via Freetalk it will bloat the messages.
Further, I was told to implement CHK messages in Freetalk because messages
could not fit into SSK. So now Freetalk uses SSK message lists to publish
CHK URI of the messages... If the CHK URI become insanely long then the
message lists will be bloated very much and maybe only five messages or so
will fit into a SSK message list. Sucks.

> They are longer than my browser window is wide, so I won't 
> even notice it, if they become longer. 
> 
> And on freesites, we'll use links anyway, where the form of 
> the URL doesn't really matter. 
> 
> So if only the key length keeps you from doing the right 
> thing, just do the right thing anyway. 
> 
> Best wishes,
> Arne
> --
> -- Ein W?rfel System: http://1w6.org - einfach saubere 
> (Rollenspiel-) Regeln.
> -- Infinite Hands: http://infinite-hands.draketo.de - singing 
> a part of the history of free software.
> -- My stuff: http://draketo.de - stories, songs, poems, 
> programs and stuff :)
> 
> -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> block [16:39:31]  duplicating the top block can be done with SSKs
> very easily [16:39:40]  but with CHKs it requires much longer URIs
> [16:39:43]  is that a problem?
> [16:40:04]  how much longer?
> [16:40:10]  CHK@,, -> CHK@ key 1>
> [16:40:29]  i.e. at least twice as long

I stopped reading at about the middle, so it might be that I missed something, 
but why do you care if CHKs become longer? 

They are already so long that noone types them (I hope), and I really don't 
care if the key I copy-paste has two times the length. 

They are longer than my browser window is wide, so I won't even notice it, if 
they become longer. 

And on freesites, we'll use links anyway, where the form of the URL doesn't 
really matter. 

So if only the key length keeps you from doing the right thing, just do the 
right thing anyway. 

Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Easy top block duplication: Content MultiplicationKeys

[xor]

>
> The filename is ignored. This will make the Frost folk happy.
>
Now that DOES sound good. Especially if you consider that it is sometimes
difficult to restore the original filename, for example if someone uploaded
the file using Linux and the file name containes characters which are not
allowed on windows and you want to re-insert from windows.

[freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

Argh, no, this doesn't work, because the pubkey is known, and there is no way 
for the node to verify that the content is in fact valid. An attacker can not 
only cause collisions, he can preemptively block known content by inserting 
bogus data to where it would be posted.

So what does that leave?

On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
> 1) Duplicate the top block with SSKs: 
> SSK,3@,,/ would insert e.g. 
> SSK@,,/-N for a series of N's.

Make cryptokey the hash of the content, avoids different-content attacks.

> 2) CHKs with extra routing keys:
> CHK@

Replace crypto key with the hash of the content, and derive the crypto key for 
each block:
cryptokey_N = H ( H(data) + "N" )

3) Reinserting the top block when a download has finished, and also at some 
point after inserting the data. IMHO this may help, but it does not address 
the core problem. We need redundancy over *different areas of the keyspace*.

4) An FCP command to reinsert a file, given a known top-block key. If we 
cannot reconstruct as-is, we fail, but if we can, we reinsert that block 
(exactly as is, like a binary blob), and the rest of the CHK-based blocks 
under it.

Maybe a combination of 1) and 4) ?

On first insert:
User inserts the data as DSK,3@/filename.
The node creates a random pubkey, and hashes the content of the top block to 
get the cryptokey. It inserts each block, and returns 
DSK,3@,,/filename

The filename could be ignored if we want.

On reinsert:
The original URI must be specified, and have been downloaded. If it is kept on 
the download queue then we can simply click a button to reinsert.

For SSKs:
User inserts the data as SSK,3@,,/

We insert to SSK@,,/- for N in 0, 1, 2.

Since the URI is not content-derived, there is the possibility of the inserter 
will insert different content to each slot. AFAICS this cannot be avoided.

The major disadvantages are:
- When reinserting we MUST know the original URI.
- There is no way to heal the alternate top-blocks.

IMHO the latter is more serious than the former, but full convergence would be 
ideal. Option 2 has neither of these problems, but does have very long URIs. 
Mostly keys are copied and pasted, so maybe that isn't a big problem?

If people are happy with option 2 (very long CHKs), I can implement it 
reasonably quickly...
> 
> Neither of these schemes is acceptable IMHO. The former allows for an 
attacker 
> to insert different content to different keys, and get some info about 
> targets that way, and it is non-convergent, not allowing for reinserts. The 
> latter doubles the length of the already way too long CHKs.
> 
> I propose a new key type which is convergent, has URIs shorter than existing 
> CHKs, and any number of duplicate blocks: the Content Multiplication Key 
(for 
> want of a better name, alternatives are welcome).
> 
> DETAILS:
> 
> Insert the splitfile normally, except for the top block. The top block must 
> fit inside a 1KB SSK payload. 
> 
> Hash the content of the top block:
> 
> hash of content: H(data)
> 
> Create a private key:
> 
> privkey = MAKE_PRIVKEY ( H ( H(data) + "0" ) )
> 
> Create the base crypto key:
> 
> ckey_base = H ( H ( data + "1" ) )
> 
> Create a series of crypto keys:
> 
> ckey_N = H ( ckey_base + "N" )
> 
> Insert to a series of SSKs:
> 
> SSK@,,
> 
> Announce the key:
> 
> UMK,N@<H(data)>,/
> (Where N is the number of copies inserted)
> 
> The filename is ignored. This will make the Frost folk happy.
> 


-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/2f28e9a3/attachment.pgp>

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
> It has recently come to our attention that the problem with data persistence 
> is usually that the top block has fallen out or that the few nodes with the 
> block are never online at the same time as the person who wants to fetch it. 
> Hence we need to duplicate the top block. So far there have been two 
schemes:
> 1) Duplicate the top block with SSKs: 
> SSK,3@,,/ would insert e.g. 
> SSK@,,/-N for a series of N's.
> 2) CHKs with extra routing keys:
> CHK@
> 
> Neither of these schemes is acceptable IMHO. The former allows for an 
attacker 
> to insert different content to different keys, and get some info about 
> targets that way, and it is non-convergent, not allowing for reinserts. The 
> latter doubles the length of the already way too long CHKs.
> 
> I propose a new key type which is convergent, has URIs shorter than existing 
> CHKs, and any number of duplicate blocks: the Content Multiplication Key 
(for 
> want of a better name, alternatives are welcome).
> 
> DETAILS:
> 
> Insert the splitfile normally, except for the top block. The top block must 
> fit inside a 1KB SSK payload. 
> 
> Hash the content of the top block:
> 
> hash of content: H(data)
> 
> Create a private key:
> 
> privkey = MAKE_PRIVKEY ( H ( H(data) + "0" ) )
> 
> Create the base crypto key:
> 
> ckey_base = H ( H ( data + "1" ) )

Err, I mean H ( H ( data ) + "1" ) )
> 
> Create a series of crypto keys:
> 
> ckey_N = H ( ckey_base + "N" )
> 
> Insert to a series of SSKs:
> 
> SSK@,,
> 
> Announce the key:
> 
> UMK,N@<H(data)>,/
> (Where N is the number of copies inserted)
> 
> The filename is ignored. This will make the Frost folk happy.
> 


-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/17fecba9/attachment.pgp>

[freenet-dev] Our current web interface and its usability

[Matthew Toseland]

On Wednesday 22 April 2009 21:17:16 xor wrote:
> 
>   _  
> 
> From: devl-bounces at freenetproject.org
> [mailto:devl-bounces at freenetproject.org] On Behalf Of Ian Clarke
> Sent: Wednesday, April 22, 2009 7:58 PM
> To: Discussion of development issues
> Subject: Re: [freenet-dev] Our current web interface and its usability
> 
> 
> On Wed, Apr 22, 2009 at 4:05 AM, xor  wrote:
> 
> 
> We DO NOT need a new web interface. Our current web interface is easy to
> use, works well, is sufficient, and it is also easy to write plugins which
> use it - I've worked with it for WoT and Freetalk and it was fun.
> 
> 
> 
> I hope this is true, but I'm skeptical.  I'd recomend getting some
> non-techies to try Freenet, without guidance from you, and to point out
> anything in the web interface that doesn't make sense.  I'd like to think
> that they won't find anything to point to, but I doubt it :-)
>  
> 
> The persons I've showed it to sit at their PC very much but are not experts.
> I'd call them "power users", they are familiar with eMule etc. and do not
> like wasting much time on configuring complex stuff. They were impressed by
> the clear layout of the web interface and thought it was a good
> representation of a p2p program. 

This is also what esr said. But again he is hardly a non-geek!
>  
> Anyway I've tried my best to point out anything which is difficult to use in
> my bug reports.
>  
> 
> 
> 
> We're all steeped in a pretty good understanding of Freenet, and the
> terminology used, but newbies aren't.  A term that makes perfect sense to
> us, such as Matthew's use of the term "node" in the progress page, is likely
> to confuse newbies. 
>  
> 
> "Node" should really be replaced with "Client" *everywhere* because client
> is the common word.

Client is what connects to a node, no?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/cd01fb3c/attachment.pgp>

[freenet-dev] Progress page implemented

[Matthew Toseland]

On Wednesday 22 April 2009 18:48:12 Ian Clarke wrote:
> Looks pretty good.
> 
> Should we use the term "node"?  Seems like jargon to me.

Proposed improved wordings are very welcome.
> 
> Ian.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/ae581ad0/attachment.pgp>

[freenet-dev] Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

It has recently come to our attention that the problem with data persistence 
is usually that the top block has fallen out or that the few nodes with the 
block are never online at the same time as the person who wants to fetch it. 
Hence we need to duplicate the top block. So far there have been two schemes:
1) Duplicate the top block with SSKs: 
SSK,3@,,/ would insert e.g. 
SSK@,,/-N for a series of N's.
2) CHKs with extra routing keys:
CHK@

Neither of these schemes is acceptable IMHO. The former allows for an attacker 
to insert different content to different keys, and get some info about 
targets that way, and it is non-convergent, not allowing for reinserts. The 
latter doubles the length of the already way too long CHKs.

I propose a new key type which is convergent, has URIs shorter than existing 
CHKs, and any number of duplicate blocks: the Content Multiplication Key (for 
want of a better name, alternatives are welcome).

DETAILS:

Insert the splitfile normally, except for the top block. The top block must 
fit inside a 1KB SSK payload. 

Hash the content of the top block:

hash of content: H(data)

Create a private key:

privkey = MAKE_PRIVKEY ( H ( H(data) + "0" ) )

Create the base crypto key:

ckey_base = H ( H ( data + "1" ) )

Create a series of crypto keys:

ckey_N = H ( ckey_base + "N" )

Insert to a series of SSKs:

SSK@,,

Announce the key:

UMK,N@<H(data)>,/
(Where N is the number of copies inserted)

The filename is ignored. This will make the Frost folk happy.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090423/18abc534/attachment.pgp>

Re: [freenet-dev] Easy top block duplication: Content MultiplicationKeys

[xor]

 The filename is ignored. This will make the Frost folk happy.

Now that DOES sound good. Especially if you consider that it is sometimes
difficult to restore the original filename, for example if someone uploaded
the file using Linux and the file name containes characters which are not
allowed on windows and you want to re-insert from windows.

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
 block [16:39:31] toad_ duplicating the top block can be done with SSKs
 very easily [16:39:40] toad_ but with CHKs it requires much longer URIs
 [16:39:43] toad_ is that a problem?
 [16:40:04] p0s how much longer?
 [16:40:10] toad_ CHK@routing key,decrypt key,extra - CHK@routing
 key 1,routing key 2,routing key 3,decrypt key,extra
 [16:40:29] toad_ i.e. at least twice as long

I stopped reading at about the middle, so it might be that I missed something, 
but why do you care if CHKs become longer? 

They are already so long that noone types them (I hope), and I really don't 
care if the key I copy-paste has two times the length. 

They are longer than my browser window is wide, so I won't even notice it, if 
they become longer. 

And on freesites, we'll use links anyway, where the form of the URL doesn't 
really matter. 

So if only the key length keeps you from doing the right thing, just do the 
right thing anyway. 

Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

 

 -Original Message-
 From: devl-boun...@freenetproject.org 
 [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
 Babenhauserheide
 Sent: Thursday, April 23, 2009 9:14 AM
 To: devl@freenetproject.org
 Subject: Re: [freenet-dev] Non-convergent encryption kills 
 easy filesharing
 
 Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
  block [16:39:31] toad_ duplicating the top block can be done with 
  SSKs very easily [16:39:40] toad_ but with CHKs it requires much 
  longer URIs [16:39:43] toad_ is that a problem?
  [16:40:04] p0s how much longer?
  [16:40:10] toad_ CHK@routing key,decrypt key,extra - 
  CHK@routing key 1,routing key 2,routing key 3,decrypt 
  key,extra [16:40:29] toad_ i.e. at least twice as long
 
 I stopped reading at about the middle, so it might be that I 
 missed something, but why do you care if CHKs become longer? 
 
 They are already so long that noone types them (I hope), and 
 I really don't care if the key I copy-paste has two times the length. 

If they are exchanged via Freetalk it will bloat the messages.
Further, I was told to implement CHK messages in Freetalk because messages
could not fit into SSK. So now Freetalk uses SSK message lists to publish
CHK URI of the messages... If the CHK URI become insanely long then the
message lists will be bloated very much and maybe only five messages or so
will fit into a SSK message list. Sucks.

 They are longer than my browser window is wide, so I won't 
 even notice it, if they become longer. 
 
 And on freesites, we'll use links anyway, where the form of 
 the URL doesn't really matter. 
 
 So if only the key length keeps you from doing the right 
 thing, just do the right thing anyway. 
 
 Best wishes,
 Arne
 --
 -- Ein Würfel System: http://1w6.org - einfach saubere 
 (Rollenspiel-) Regeln.
 -- Infinite Hands: http://infinite-hands.draketo.de - singing 
 a part of the history of free software.
 -- My stuff: http://draketo.de - stories, songs, poems, 
 programs and stuff :)
 
 -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Daniel Cheng]

On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 After a long conversation with p0s, I am fairly sure that our decision at last
 year's summit to use non-convergent encryption for splitfiles (i.e. a
 different set of blocks each time) in order to largely solve our security
 problems will make filesharing on Freenet much less convenient.

 Other points:
 - The problem with persistence of data is frequently the top block. This can
 be solved by duplicating it. This is easy for SSKs, but for CHKs would
 require quite long URIs. This may be acceptable given the likely data
 persistence gains.
 - We must deal with the short segments problem.
 - Node references should be tolerant of losing newlines and gaining new ones.

 I had hoped that we could release a 0.8.1 with solutions to the data
 persistence problems and non-convergent encryption, but it looks like we will
 need to wait for 0.9 with tunnels and bloom filter sharing. :(

I believe the persistence would be improved with tunnels
  -- blocks can be insert from different paths and cache differently.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Progress page implemented

[bo-le]

Am Donnerstag, 23. April 2009 03:34:46 schrieb Ian Clarke:
 I propose software as an alternative to node.

IMHO this is the wrong way. 'software' is to common...

Freenet is a network, running on top of  'internet' (currently only 
on 'internet', but a WLAN transport plugin allows 'internet free' smash 
networks/clowds/nodes), so the 'freenet software' *is* a node.

I suggest to use 'node' (helps to easily differ from other 'software'),
but explain it shortly on the firstpage the user see, all over occurrences 
of 'node' have an shortexplaintooltip and/or a link to ExplainTerms.html#node

MfG
saces
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
  -Original Message-
  From: devl-boun...@freenetproject.org
  [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne
  Babenhauserheide
  Sent: Thursday, April 23, 2009 9:14 AM
  To: devl@freenetproject.org
  Subject: Re: [freenet-dev] Non-convergent encryption kills
  easy filesharing
 
  Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
   block [16:39:31] toad_ duplicating the top block can be done with
   SSKs very easily [16:39:40] toad_ but with CHKs it requires much
   longer URIs [16:39:43] toad_ is that a problem?
   [16:40:04] p0s how much longer?
   [16:40:10] toad_ CHK@routing key,decrypt key,extra -
   CHK@routing key 1,routing key 2,routing key 3,decrypt
   key,extra [16:40:29] toad_ i.e. at least twice as long
 
  I stopped reading at about the middle, so it might be that I
  missed something, but why do you care if CHKs become longer?
 
  They are already so long that noone types them (I hope), and
  I really don't care if the key I copy-paste has two times the length.

 If they are exchanged via Freetalk it will bloat the messages.
 Further, I was told to implement CHK messages in Freetalk because messages
 could not fit into SSK. So now Freetalk uses SSK message lists to publish
 CHK URI of the messages... If the CHK URI become insanely long then the
 message lists will be bloated very much and maybe only five messages or so
 will fit into a SSK message list. Sucks.

So why don't you take the approach of filesystems and add an option where the 
message list SSK can contain a link to a CHK which contains all message CHK 
URIs? 

(that's what ext2 does with inodes to support almost arbitrarily large files, 
though each inodes is only 4k in size) 

If the efficiency of CHKs should rise by more than a factor of 2 when using 
the large URI instead of non-republishable URIs, freetalk will even profit 
from this (ping time wise). 

But I don't know the implementation details of freetalk, so I don't know if 
this is a possible option for you. 

Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content MultiplicationKeys

[Matthew Toseland]

On Thursday 23 April 2009 07:17:36 xor wrote:
 
 
  The filename is ignored. This will make the Frost folk happy.
 
 Now that DOES sound good. Especially if you consider that it is sometimes
 difficult to restore the original filename, for example if someone uploaded
 the file using Linux and the file name containes characters which are not
 allowed on windows and you want to re-insert from windows

Uhm, what archaic version of Windows are you using that doesn't have unicode 
support?!


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

I would really appreciate input on option 2 i.e. how much of a problem are 
long CHKs?

On Thursday 23 April 2009 03:03:00 Matthew Toseland wrote:
 Argh, no, this doesn't work, because the pubkey is known, and there is no 
way 
 for the node to verify that the content is in fact valid. An attacker can 
not 
 only cause collisions, he can preemptively block known content by inserting 
 bogus data to where it would be posted.
 
 So what does that leave?
 
 On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
  1) Duplicate the top block with SSKs: 
  SSK,3@pubkey,cryptokey,extra/filename would insert e.g. 
  SSK@pubkey,cryptokey,extra/filename-N for a series of N's.
 
 Make cryptokey the hash of the content, avoids different-content attacks.
 
  2) CHKs with extra routing keys:
  CHK@routing key 1,routing key 2,routing key 3,crypto key,extra
 
 Replace crypto key with the hash of the content, and derive the crypto key 
for 
 each block:
 cryptokey_N = H ( H(data) + N )
 
 3) Reinserting the top block when a download has finished, and also at some 
 point after inserting the data. IMHO this may help, but it does not address 
 the core problem. We need redundancy over *different areas of the keyspace*.
 
 4) An FCP command to reinsert a file, given a known top-block key. If we 
 cannot reconstruct as-is, we fail, but if we can, we reinsert that block 
 (exactly as is, like a binary blob), and the rest of the CHK-based blocks 
 under it.
 
 Maybe a combination of 1) and 4) ?
 
 On first insert:
 User inserts the data as DSK,3@/filename.
 The node creates a random pubkey, and hashes the content of the top block to 
 get the cryptokey. It inserts each block, and returns 
 DSK,3@pubkey,cryptokey,extra/filename
 
 The filename could be ignored if we want.
 
 On reinsert:
 The original URI must be specified, and have been downloaded. If it is kept 
on 
 the download queue then we can simply click a button to reinsert.
 
 For SSKs:
 User inserts the data as SSK,3@privkey,cryptokey,extra/filename
 
 We insert to SSK@pubkey,cryptokey,extra/filename-N for N in 0, 1, 
2.
 
 Since the URI is not content-derived, there is the possibility of the 
inserter 
 will insert different content to each slot. AFAICS this cannot be avoided.
 
 The major disadvantages are:
 - When reinserting we MUST know the original URI.
 - There is no way to heal the alternate top-blocks.
 
 IMHO the latter is more serious than the former, but full convergence would 
be 
 ideal. Option 2 has neither of these problems, but does have very long URIs. 
 Mostly keys are copied and pasted, so maybe that isn't a big problem?
 
 If people are happy with option 2 (very long CHKs), I can implement it 
 reasonably quickly...
  
  Neither of these schemes is acceptable IMHO. The former allows for an 
 attacker 
  to insert different content to different keys, and get some info about 
  targets that way, and it is non-convergent, not allowing for reinserts. 
The 
  latter doubles the length of the already way too long CHKs.
  
  I propose a new key type which is convergent, has URIs shorter than 
existing 
  CHKs, and any number of duplicate blocks: the Content Multiplication Key 
 (for 
  want of a better name, alternatives are welcome).
  
  DETAILS:
  
  Insert the splitfile normally, except for the top block. The top block 
must 
  fit inside a 1KB SSK payload. 
  
  Hash the content of the top block:
  
  hash of content: H(data)
  
  Create a private key:
  
  privkey = MAKE_PRIVKEY ( H ( H(data) + 0 ) )
  
  Create the base crypto key:
  
  ckey_base = H ( H ( data + 1 ) )
  
  Create a series of crypto keys:
  
  ckey_N = H ( ckey_base + N )
  
  Insert to a series of SSKs:
  
  SSK@privkey,ckey_N,extra
  
  Announce the key:
  
  UMK,N@H(data),extra/filename
  (Where N is the number of copies inserted)
  
  The filename is ignored. This will make the Frost folk happy.
  
 
 
 




signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 08:25:15 xor wrote:
 
  -Original Message-
  From: devl-boun...@freenetproject.org 
  [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
  Babenhauserheide
  Sent: Thursday, April 23, 2009 9:14 AM
  To: devl@freenetproject.org
  Subject: Re: [freenet-dev] Non-convergent encryption kills 
  easy filesharing
  
  Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
   block [16:39:31] toad_ duplicating the top block can be done with 
   SSKs very easily [16:39:40] toad_ but with CHKs it requires much 
   longer URIs [16:39:43] toad_ is that a problem?
   [16:40:04] p0s how much longer?
   [16:40:10] toad_ CHK@routing key,decrypt key,extra - 
   CHK@routing key 1,routing key 2,routing key 3,decrypt 
   key,extra [16:40:29] toad_ i.e. at least twice as long
  
  I stopped reading at about the middle, so it might be that I 
  missed something, but why do you care if CHKs become longer? 
  
  They are already so long that noone types them (I hope), and 
  I really don't care if the key I copy-paste has two times the length. 
 
 If they are exchanged via Freetalk it will bloat the messages.
 Further, I was told to implement CHK messages in Freetalk because messages
 could not fit into SSK. So now Freetalk uses SSK message lists to publish
 CHK URI of the messages... If the CHK URI become insanely long then the
 message lists will be bloated very much and maybe only five messages or so
 will fit into a SSK message list. Sucks.

No, this would not be normal CHKs, sorry. I'd probably call them DHKs. They 
would only be used for the top block of big files.
 
  They are longer than my browser window is wide, so I won't 
  even notice it, if they become longer. 
  
  And on freesites, we'll use links anyway, where the form of 
  the URL doesn't really matter. 
  
  So if only the key length keeps you from doing the right 
  thing, just do the right thing anyway. 
  
  Best wishes,
  Arne


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 09:37:45 Daniel Cheng wrote:
 On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  After a long conversation with p0s, I am fairly sure that our decision at 
last
  year's summit to use non-convergent encryption for splitfiles (i.e. a
  different set of blocks each time) in order to largely solve our security
  problems will make filesharing on Freenet much less convenient.
 
  Other points:
  - The problem with persistence of data is frequently the top block. This 
can
  be solved by duplicating it. This is easy for SSKs, but for CHKs would
  require quite long URIs. This may be acceptable given the likely data
  persistence gains.
  - We must deal with the short segments problem.
  - Node references should be tolerant of losing newlines and gaining new 
ones.
 
  I had hoped that we could release a 0.8.1 with solutions to the data
  persistence problems and non-convergent encryption, but it looks like we 
will
  need to wait for 0.9 with tunnels and bloom filter sharing. :(
 
 I believe the persistence would be improved with tunnels
   -- blocks can be insert from different paths and cache differently.

Why? If specialisation works, and IMHO we have no reason to think it doesn't, 
then it will land up on the same nodes. The problem is simply that those 
nodes go offline.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: ContentMultiplicationKeys

[xor]

 

 -Original Message-
 From: devl-boun...@freenetproject.org 
 [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland
 Sent: Thursday, April 23, 2009 2:48 PM
 To: Discussion of development issues
 Subject: Re: [freenet-dev] Easy top block duplication: 
 ContentMultiplicationKeys
 
 On Thursday 23 April 2009 07:17:36 xor wrote:
  
  
   The filename is ignored. This will make the Frost folk happy.
  
  Now that DOES sound good. Especially if you consider that it is 
  sometimes difficult to restore the original filename, for 
 example if 
  someone uploaded the file using Linux and the file name containes 
  characters which are not allowed on windows and you want to 
 re-insert 
  from windows
 
 Uhm, what archaic version of Windows are you using that 
 doesn't have unicode support?!
 

Linux allows stuff in filenames which are control characters for windows.

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Matthew Toseland]

Anecdotal evidence suggests that right now at least one third of our content 
persistence problems boil down to this one bug: I added it 2 weeks ago and 
it still hasn't got past 0% (0/1). A new key type, DHKs (Duplicated Hash 
Keys), would solve the problem, but the new keys would be twice as long as 
current CHKs. Is this a problem? I would really appreciate input from users, 
particularly those who upload and download files:
- Is it a problem for the keys to be really long (twice as long as current 
CHKs)? CHKs are copied and pasted, so maybe not a problem?
- Is it true that a great many downloads get stuck at 0% for a long time, 
showing 0 blocks of 1 if you mouseover the percentage?

Example:
c...@4~2ftxtbe2so8nzzizneyrn5soaffk-hqsvjbhlc77a,97XjJekfSl8HxkJFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3

- (something like)

d...@97xjjekfsl8hxkjfyhj4cdo9n7s0exhe-ewmr8zuvxm,4~2FTXtBE2So8NZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9DrDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--8/chaosradio_142.mp3

GORY DETAILS:

Currently we use:
CHK@routing key,crypto key,extra

(Filenames afterwards are manifests, and therefore impact on the CHK)

The new key type would be:

DHK@data hash,routing key 1,routing key 2,routing key 
3,extra/ignore filename

(A filename is mandatory, and is always ignored, so does not impact on the 
rest of the key).

We might allow any number of routing keys from 2 upwards, for more redundancy 
at the cost of a longer URI, but IMHO 3 is a good default number.

You would get such a key when you insert a file as d...@.

Arguably nobody ever types CHKs even now, and copy and paste allows for fairly 
long keys. Thoughts?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

 

 -Original Message-
 From: arne_...@web.de [mailto:arne_...@web.de] 
 Sent: Thursday, April 23, 2009 11:21 AM
 To: devl@freenetproject.org
 Cc: xor
 Subject: Re: [freenet-dev] Non-convergent encryption kills 
 easy filesharing
 
 Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
   -Original Message-
   From: devl-boun...@freenetproject.org 
   [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
   Babenhauserheide
   Sent: Thursday, April 23, 2009 9:14 AM
   To: devl@freenetproject.org
   Subject: Re: [freenet-dev] Non-convergent encryption kills easy 
   filesharing
  
   Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
block [16:39:31] toad_ duplicating the top block can be done 
with SSKs very easily [16:39:40] toad_ but with CHKs 
 it requires 
much longer URIs [16:39:43] toad_ is that a problem?
[16:40:04] p0s how much longer?
[16:40:10] toad_ CHK@routing key,decrypt key,extra - 
CHK@routing key 1,routing key 2,routing key 3,decrypt
key,extra [16:40:29] toad_ i.e. at least twice as long
  
   I stopped reading at about the middle, so it might be 
 that I missed 
   something, but why do you care if CHKs become longer?
  
   They are already so long that noone types them (I hope), and I 
   really don't care if the key I copy-paste has two times 
 the length.
 
  If they are exchanged via Freetalk it will bloat the messages.
  Further, I was told to implement CHK messages in Freetalk because 
  messages could not fit into SSK. So now Freetalk uses SSK message 
  lists to publish CHK URI of the messages... If the CHK URI become 
  insanely long then the message lists will be bloated very much and 
  maybe only five messages or so will fit into a SSK message 
 list. Sucks.
 
 So why don't you take the approach of filesystems and add an 
 option where the message list SSK can contain a link to a CHK 
 which contains all message CHK URIs?

1. Too complex.
2. The problem is only shifted to another place. The problem with message
lists is that they must fit as many message references into a single SSK
block as possible to gurantee high network efficiency. You cannot tell me to
solve the space problem by INCREASING the used space by adding an extra CHK
block.
 
 
 (that's what ext2 does with inodes to support almost 
 arbitrarily large files, though each inodes is only 4k in size) 
 
 If the efficiency of CHKs should rise by more than a factor 
 of 2 when using the large URI instead of non-republishable 
 URIs, freetalk will even profit from this (ping time wise). 
 
 But I don't know the implementation details of freetalk, so I 
 don't know if this is a possible option for you. 
 
 Best wishes,
 Arne
 --
 -- Ein Würfel System: http://1w6.org - einfach saubere 
 (Rollenspiel-) Regeln.
 -- Infinite Hands: http://infinite-hands.draketo.de - singing 
 a part of the history of free software.
 -- My stuff: http://draketo.de - stories, songs, poems, 
 programs and stuff :)
 
 -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Solving I queued it 2 weeks ago and it's still at0% : are really long URIs a problem?

[xor]

 

 -Original Message-
 From: devl-boun...@freenetproject.org 
 [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland
 Sent: Thursday, April 23, 2009 3:17 PM
 To: supp...@freenetproject.org; Discussion of development issues
 Cc: Ian Clarke
 Subject: [freenet-dev] Solving I queued it 2 weeks ago and 
 it's still at0% : are really long URIs a problem?
 
 Anecdotal evidence suggests that right now at least one third 
 of our content persistence problems boil down to this one 
 bug: I added it 2 weeks ago and it still hasn't got past 0% 
 (0/1). A new key type, DHKs (Duplicated Hash Keys), would 
 solve the problem, but the new keys would be twice as long as 
 current CHKs. Is this a problem? I would really appreciate 
 input from users, particularly those who upload and download files:
 - Is it a problem for the keys to be really long (twice as 
 long as current CHKs)? CHKs are copied and pasted, so maybe 
 not a problem?
 - Is it true that a great many downloads get stuck at 0% for 
 a long time, showing 0 blocks of 1 if you mouseover the percentage?

I think you should first of all figure out whether it isn't a db4o
introduced bug or even a problem which has been there before db4o.
Relying on something brand new (the db4o branch) which has been out in the
real world only for some days to decide implementation of a new key type is
really insane IMHO. We don't know whether the db4o branch doesn't have
hundreds of other bugs. And in general the debugging of the node has been
mostly on hold for long time due to your work on db4o.

This would not be the first bug which makes downloads hang at 0% which we
had.

Further I do not understand why you don't want to solve the problem with
tunnels. You told me that tunnels fix three problems:

1. they increase security
2. toad_ well the other benefit is that it allows us to do bloom filter
sharing [17:07:58] 
3. tunnels would fix the top-block-not-reachable problem we are talking
about

So tunnels would fix 3 problems, this fixes 1 and tunnels will still be
necessary some day.

Why not just postpone the fix of the top-block-problem until we have
tunnels?

Why not make the node just automatically re-insert the top block upon
download? Thats easy to implement after all?

Why not debug first?

I don't understand why you would want to chose the annoying alternative of
implementing yet another key type if there are so many alternatives to it.


 
 Example:
 c...@4~2ftxtbe2so8nzzizneyrn5soaffk-hqsvjbhlc77a,97XjJekfSl8Hxk
 JFYhj4cdo9n7s0exhE-EWMr8zuVxM,AAIC--8/chaosradio_142.mp3
 
 - (something like)
 
 d...@97xjjekfsl8hxkjfyhj4cdo9n7s0exhe-ewmr8zuvxm,4~2FTXtBE2So8N
 ZZIzneYrn5SOaFFk-hQsvjBHLc77A,ughDyCjP0jeBuRRx33nULUb4Pl-6Dk9D
 rDrH1miXCj0,VIOAKDzD~YIzrD5NBbD3v5SxOiwYXg84qQYdbkJA3bo,AAIC--
 8/chaosradio_142.mp3
 
 GORY DETAILS:
 
 Currently we use:
 CHK@routing key,crypto key,extra
 
 (Filenames afterwards are manifests, and therefore impact on the CHK)
 
 The new key type would be:
 
 DHK@data hash,routing key 1,routing key 2,routing key 
 3,extra/ignore filename
 
 (A filename is mandatory, and is always ignored, so does not 
 impact on the rest of the key).
 
 We might allow any number of routing keys from 2 upwards, for 
 more redundancy at the cost of a longer URI, but IMHO 3 is a 
 good default number.
 
 You would get such a key when you insert a file as d...@.
 
 Arguably nobody ever types CHKs even now, and copy and paste 
 allows for fairly long keys. Thoughts?
 

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content MultiplicationKeys

[Daniel Cheng]

On Thu, Apr 23, 2009 at 8:48 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Thursday 23 April 2009 07:17:36 xor wrote:

 
  The filename is ignored. This will make the Frost folk happy.
 
 Now that DOES sound good. Especially if you consider that it is sometimes
 difficult to restore the original filename, for example if someone uploaded
 the file using Linux and the file name containes characters which are not
 allowed on windows and you want to re-insert from windows

 Uhm, what archaic version of Windows are you using that doesn't have unicode
 support?!


Try colon (:), this is not valid in windows file name.
Or just use FAT32.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Solving I queued it 2 weeks ago and it's still at0% : are really long URIs a problem?

[Matthew Toseland]

On Thursday 23 April 2009 14:36:37 xor wrote:
 
  -Original Message-
  From: devl-boun...@freenetproject.org 
  [mailto:devl-boun...@freenetproject.org] On Behalf Of Matthew Toseland
  Sent: Thursday, April 23, 2009 3:17 PM
  To: supp...@freenetproject.org; Discussion of development issues
  Cc: Ian Clarke
  Subject: [freenet-dev] Solving I queued it 2 weeks ago and 
  it's still at0% : are really long URIs a problem?
  
  Anecdotal evidence suggests that right now at least one third 
  of our content persistence problems boil down to this one 
  bug: I added it 2 weeks ago and it still hasn't got past 0% 
  (0/1). A new key type, DHKs (Duplicated Hash Keys), would 
  solve the problem, but the new keys would be twice as long as 
  current CHKs. Is this a problem? I would really appreciate 
  input from users, particularly those who upload and download files:
  - Is it a problem for the keys to be really long (twice as 
  long as current CHKs)? CHKs are copied and pasted, so maybe 
  not a problem?
  - Is it true that a great many downloads get stuck at 0% for 
  a long time, showing 0 blocks of 1 if you mouseover the percentage?
 
 I think you should first of all figure out whether it isn't a db4o
 introduced bug or even a problem which has been there before db4o.

AFAIK it is a general problem with Freenet, that predates db4o. To be 
specific:
- Most downloads of older data stall at 0% for a *long time*.
- When they get past 0%, they frequently progress fairly quickly.
- Splitfile blocks have redundancy, but the top block does not.
- Unpopular (old) content will have fallen out of caches.
- Unpopular content will only be in the stores of the small number of nodes 
(typically 3 but a bit more because of not taking into account low-uptime 
nodes).
- Low uptime and join-leave churn (newbies trying Freenet out and then 
leaving) mean that there is a good chance that all of these nodes are offline 
when you want to fetch the content. It is also possible that they have 
shifted location.

 Relying on something brand new (the db4o branch) which has been out in the
 real world only for some days to decide implementation of a new key type is
 really insane IMHO. We don't know whether the db4o branch doesn't have
 hundreds of other bugs. And in general the debugging of the node has been
 mostly on hold for long time due to your work on db4o.

I have done a vast amount of debugging on db4o while it was a branch. I accept 
that it likely has many bugs. However we have approximately 34 days to get a 
release out before we run out of money.
 
 This would not be the first bug which makes downloads hang at 0% which we
 had.

Sure, it might well be a bug, in which case we get a better architecture 
(everything except the top block is redundant already), we spend a few days 
implementing and debugging DHKs, and we still have to solve the bug.
 
 Further I do not understand why you don't want to solve the problem with
 tunnels. You told me that tunnels fix three problems:

Tunnels will not solve the problem if low uptime, newbies joining and leaving, 
and location swapping on darknet, are the root of the problem. Everything is 
redundant below the top block, and below the top block it works relatively 
well. So add redundancy for the top block. 

More redundancy for *all* blocks would be counterproductive, because block 
level redundancy is far less efficient than FEC redundancy, because one block 
can be reconstructed from other blocks and healed during a download. The top 
block is a special case because it can't be FEC'ed.
 
 1. they increase security

True.

 2. toad_ well the other benefit is that it allows us to do bloom filter
 sharing [17:07:58] 

Yes but the performance cost is considerable.

 3. tunnels would fix the top-block-not-reachable problem we are talking
 about

No, they would not, as I have explained.
 
 So tunnels would fix 3 problems, this fixes 1 and tunnels will still be
 necessary some day.
 
 Why not just postpone the fix of the top-block-problem until we have
 tunnels?

Tunnels are awesome, but they will be take quite some time to get right, and 
we really have not yet quantified the performance impact of bloom filters 
plus tunnels. It's a 0.9 matter. Right now we have funds for me for approx 34 
days.
 
 Why not make the node just automatically re-insert the top block upon
 download? Thats easy to implement after all?

It does not introduce any additional redundancy. We need redundancy across 
*different areas of the keyspace*.
 
 Why not debug first?

We will debug. But IMHO this is an architectural problem, as nextgens has 
pointed out.
 
 I don't understand why you would want to chose the annoying alternative of
 implementing yet another key type if there are so many alternatives to it.

Because none of the alternatives solves the problem.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list

Re: [freenet-dev] [freenet-cvs] r26829 - trunk/freenet/src/freenet/node/fcp

[Matthew Toseland]

On Wednesday 15 April 2009 07:43:14 j16s...@freenetproject.org wrote:
 Author: j16sdiz
 Date: 2009-04-15 06:43:12 + (Wed, 15 Apr 2009)
 New Revision: 26829
 
 Modified:
trunk/freenet/src/freenet/node/fcp/FCPClient.java
 Log:
 revert r26828: req.cacnel() in removeAll() not work as expected

Hmmm, what is the problem here? Clearly we are adding to toKill in order to 
mass-cancel?
 
 Modified: trunk/freenet/src/freenet/node/fcp/FCPClient.java
 ===
 --- trunk/freenet/src/freenet/node/fcp/FCPClient.java 2009-04-15 06:35:17 
UTC (rev 26828)
 +++ trunk/freenet/src/freenet/node/fcp/FCPClient.java 2009-04-15 06:43:12 
UTC (rev 26829)
 @@ -466,8 +466,6 @@
   if (persistenceType == ClientRequest.PERSIST_FOREVER)
   
 container.ext().store(clientRequestsByIdentifier, 2);
   }
 - for (ClientRequest req : toKill)
 - req.cancel();
   }
  
   public ClientGet getCompletedRequest(FreenetURI key, ObjectContainer 
container) {


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Matthew Toseland]

On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:
 On Wed, Apr 22, 2009 at 1:17 PM, xor x...@gmx.li wrote:
 
   Node should really be replaced with Client *everywhere* because
  client is the common word.
 
 Is it?  When I talk to non-techies about a client they think I'm referring
 to the person that employs a lawyer.  I think the least confusing term to
 use in this context may be software.
 
Very clumbersome. How would you translate Your node is downloading this page 
from Freenet ? The Freenet software running on your computer is downloading 
this page from the Freenet network ?
 
  I also think that some words should be replaced. I did that for a few
  things in my last few commits. Further, for example I suggested to replace
  Key with Download link on the downloads  uploads page.
 
 Yes, or perhaps just link.
 
 Ian.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:

 It has recently come to our attention that the problem with data  
 persistence
 is usually that the top block has fallen out or that the few nodes  
 with the
 block are never online at the same time as the person who wants to  
 fetch it.
 Hence we need to duplicate the top block. So far there have been two  
 schemes:
 1) Duplicate the top block with SSKs:
 SSK,3@pubkey,cryptokey,extra/filename would insert e.g.
 SSK@pubkey,cryptokey,extra/filename-N for a series of N's.
 2) CHKs with extra routing keys:
 CHK@routing key 1,routing key 2,routing key 3,crypto  
 key,extra

 Neither of these schemes is acceptable IMHO. The former allows for  
 an attacker
 to insert different content to different keys, and get some info about
 targets that way, and it is non-convergent, not allowing for  
 reinserts. The
 latter doubles the length of the already way too long CHKs.

 I propose a new key type which is convergent, has URIs shorter than  
 existing
 CHKs, and any number of duplicate blocks: the Content Multiplication  
 Key (for
 want of a better name, alternatives are welcome).

Perhaps there is an easier solution?

How about extending the chk logic into an alternate-chk-key (ACK?);  
that simply adds 0.25 to the expected location (for routing and  
storage).

So when you insert the top block, put it in as a chk and an ack (no  
extra uri's neccesary). When you go to fetch it, if the chk does not  
work, try the ack variant of the same key.

--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Florent Daigniere]

Robert Hailey wrote:
 On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:
 
 It has recently come to our attention that the problem with data  
 persistence
 is usually that the top block has fallen out or that the few nodes  
 with the
 block are never online at the same time as the person who wants to  
 fetch it.
 Hence we need to duplicate the top block. So far there have been two  
 schemes:
 1) Duplicate the top block with SSKs:
 SSK,3@pubkey,cryptokey,extra/filename would insert e.g.
 SSK@pubkey,cryptokey,extra/filename-N for a series of N's.
 2) CHKs with extra routing keys:
 CHK@routing key 1,routing key 2,routing key 3,crypto  
 key,extra

 Neither of these schemes is acceptable IMHO. The former allows for  
 an attacker
 to insert different content to different keys, and get some info about
 targets that way, and it is non-convergent, not allowing for  
 reinserts. The
 latter doubles the length of the already way too long CHKs.

 I propose a new key type which is convergent, has URIs shorter than  
 existing
 CHKs, and any number of duplicate blocks: the Content Multiplication  
 Key (for
 want of a better name, alternatives are welcome).
 
 Perhaps there is an easier solution?
 
 How about extending the chk logic into an alternate-chk-key (ACK?);  
 that simply adds 0.25 to the expected location (for routing and  
 storage).
 
 So when you insert the top block, put it in as a chk and an ack (no  
 extra uri's neccesary). When you go to fetch it, if the chk does not  
 work, try the ack variant of the same key.

At the moment each node on the path can verify that the data sent by 
previous hop corresponds to what it ought to; How would that work with 
your proposed solution?

NextGen$
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Current uservoice top 5 (20 node barrier)

[Matthew Toseland]

On Wednesday 22 April 2009 18:53:48 Arne Babenhauserheide wrote:
 Am Mittwoch 22 April 2009 15:53:39 schrieb Matthew Toseland:
  I don't understand why you want to run a jabber server. Surely announcing
  to your jabber contacts that you are interested in ref exchange would be
  sufficient, and would be client level?
 
 I don't mean announcing to your jabber contacts that you'd like to exchange 
 refs (though that would be a nice option, too). 
 
 I mean having all freenet refs as jabber contacts automatically, and having 
a 
 freenet jabber ID based on your ref. 
 
 That way communication with peers would become far easier (it can just rely 
on 
 jabbers own encryption - you're open to your peers anyway - and it can 
 automatically use all features people develop for jabber). 
 
 That way I'd just add my freenet jabber ID in my noderef with 127.0.0.1 as 
 server, and freenet would add all my refs as jabber contacts for that 
freenet 
 jabber ID. 

Encryption is irrelevant, any such traffic should be tunneled with regular 
freenet traffic and thus be invisible from the network. But it might be an 
interesting idea for a plugin.
 
 Best wishes, 
 Arne


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Current uservoice top 5 (20 node barrier)

[Matthew Toseland]

On Wednesday 22 April 2009 16:52:57 Robert Hailey wrote:
 
 On Apr 22, 2009, at 7:38 AM, Matthew Toseland wrote:
 
  On Wednesday 22 April 2009 02:09:21 Arne Babenhauserheide wrote:
  So we get to the question, what a freenet contact is: A friend or an
  aquaintance.
 
  If you look at myspace and similar sites, you'll see people with  
  hundreds of
  friends which in truth are aquaintances.
 
  Also the question arises, which number of friends will be efficient  
  for
  freenets algorithm: How many people have similar interest?
 
  In terms of routing, the main issues are:
  - There must be a small-world network. Clearly random automatically  
  selected
  participants will not form a small-world network, but acquaintances  
  probably
  do. I repeat, randomly selected people through any automated  
  mechanism WILL
  BREAK ROUTING!
 
 Are we even sure of that???
 
 I know that the whole routing algorithm is based on small world  
 theory. However, if we load up a sim of a large randomly connected  
 network, would freenet not operate on it? Perhaps it would sort out  
 effective and usable locations anyway (simply on graph theory)?

No. It would not work well. We demonstrated this pretty clearly with 
#freenet-refs !


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Solving I queued it 2 weeks ago and it's still at 0% : are really long URIs a problem?

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 15:16:40 schrieb Matthew Toseland:
 Arguably nobody ever types CHKs even now, and copy and paste allows for
 fairly long keys. Thoughts?

You know what I think. 

The length of the key doesn't matter to me, because freesites already hide 
them in links, and otherwise I just copy-paste them. 

I didn't ever watch my downloads long enough to say where exactly they stop. I 
just start them, and if they didn't finish in a week, I remove them again. I 
don't trust my memory enough to say much about the state. Many didn't even 
start, but I'm not sure in which state they were...

Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[VolodyA! V Anarhist]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Toseland wrote:
 I would really appreciate input on option 2 i.e. how much of a problem are 
 long CHKs?

If Long CHKs will become too much of a problem, and people won't mind their
content being spoofed, then people will start using KSK...

- Volodya

- --
http://freedom.libsyn.com/   Echo of Freedom, Radical Podcast
http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal
http://www.freedomporn.org/  Freedom Porn, anarchist and activist smut

 None of us are free until all of us are free.~ Mihail Bakunin
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknwvCUACgkQuWy2EFICg+00GgCgi0KdVgk3trar9NcfGYpNaOd2
7CAAn1mTjWDU5y6DIS3qZzHLaSLMmHdk
=7d0m
-END PGP SIGNATURE-
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Current uservoice top 5 (20 node barrier)

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 14:53:45 schrieb Arne Babenhauserheide:
 Am Mittwoch 22 April 2009 14:38:29 schrieb Matthew Toseland:
  I don't know. IMHO 150 is probably too much, have you spoken privately to
  all these people?

 I think all people I know privately, including school and university,
 account for maybe 100 to 120 people. Of them I'd trust about 40 as
 connections :)

To test the 150 people limit for the monkey space, my wife and I just did a 
test by counting all people we know personally and care about (we were walking 
to the supermarket, so we had some free time :) ). 

I got to over 200, and she got to over 300, so 150 is a bit too low as upper 
boundary, I'd say (the article didn't say 150, by the way, but 100 to 230 for 
95%). 

But of these 200 I'd trust only 50 enough that they'd keep the information 
private that I run freenet - she'd trust about 30 people enough (less tech 
savvy community :) ). 

So for pretty communicative people who mostly know tech geeks 150 to 200 
friends might be a good bet (for most tech geeks the number is likely to be 
lower, though). 

(I hope you don't mind my wording. Geek is no insult for me, and neither is 
Nerd)


Just wanted to give you the data :) 


I have a question, though: Would it help routing if people would exchange refs 
with other people who have similar interests? If yes: Can you guess how much 
it would help? 


Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Mike Bush]

Matthew Toseland wrote:
 On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:
   
 On Wed, Apr 22, 2009 at 1:17 PM, xor x...@gmx.li wrote:

 
  Node should really be replaced with Client *everywhere* because
 client is the common word.
   
 Is it?  When I talk to non-techies about a client they think I'm referring
 to the person that employs a lawyer.  I think the least confusing term to
 use in this context may be software.

 
 Very clumbersome. How would you translate Your node is downloading this page 
 from Freenet ? The Freenet software running on your computer is downloading 
 this page from the Freenet network ?
   

The Freenet software running on your computer is probably what I would use to 
describe what node means to non-techy users.
Couldn't it just use Your computer is downloading this page from Freenet, 
that's what people want to know, just that the stuff they are downloading will 
be on their computer soon.


 I also think that some words should be replaced. I did that for a few
 things in my last few commits. Further, for example I suggested to replace
 Key with Download link on the downloads  uploads page.
   
 Yes, or perhaps just link.

 Ian.
 
 

 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:


Robert Hailey wrote:



Perhaps there is an easier solution?

How about extending the chk logic into an alternate-chk-key (ACK?);
that simply adds 0.25 to the expected location (for routing and
storage).

So when you insert the top block, put it in as a chk and an ack (no
extra uri's neccesary). When you go to fetch it, if the chk does not
work, try the ack variant of the same key.


At the moment each node on the path can verify that the data sent by
previous hop corresponds to what it ought to; How would that work with
your proposed solution?

NextGen$


Sorta like this...

package freenet.keys;

public class ASKKey extends NodeCHK {
public double toNormalizedDouble() {
return (super.toNormalizedDouble()+0.25)%1.0;
}
}

The only difference is where any node would look for it. This would  
not be exposed to the client. My idea is that any chk could be  
converted to an alternate-location-finding-key just by type (which  
surely would mean a different fetch-command, e.g. fetchCHK/fetchACK...).


There would be no difference in handling, the only difference would be  
how the target-routing-location is identified from the key (the same  
as CHK plus a constant [mod 1.0]). After all, the mapping from the key  
to the small-world location is open to interpretation...


--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Robert Hailey]

On Apr 23, 2009, at 2:22 PM, Mike Bush wrote:

 Matthew Toseland wrote:
 On Thursday 23 April 2009 00:05:40 Ian Clarke wrote:

 On Wed, Apr 22, 2009 at 1:17 PM, xor x...@gmx.li wrote:


 Node should really be replaced with Client *everywhere* because
 client is the common word.

 Is it?  When I talk to non-techies about a client they think I'm  
 referring
 to the person that employs a lawyer.  I think the least confusing  
 term to
 use in this context may be software.


 Very clumbersome. How would you translate Your node is downloading  
 this page
 from Freenet ? The Freenet software running on your computer is  
 downloading
 this page from the Freenet network ?


 The Freenet software running on your computer is probably what I  
 would use to describe what node means to non-techy users.
 Couldn't it just use Your computer is downloading this page from  
 Freenet, that's what people want to know, just that the stuff they  
 are downloading will be on their computer soon.

Yea, but Matthew's language has a more technically-accurate flavor (as  
your node implies the distributed nature of freenet, whereas  
freenet is downloading makes it sound like a monolithic entity).

--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[VolodyA! V Anarhist]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Robert Hailey wrote:
 
 On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:
 
 Robert Hailey wrote:

 Perhaps there is an easier solution?

 How about extending the chk logic into an alternate-chk-key (ACK?);  
 that simply adds 0.25 to the expected location (for routing and  
 storage).

 So when you insert the top block, put it in as a chk and an ack (no  
 extra uri's neccesary). When you go to fetch it, if the chk does not  
 work, try the ack variant of the same key.

 At the moment each node on the path can verify that the data sent by
 previous hop corresponds to what it ought to; How would that work with
 your proposed solution?

 NextGen$
 
 Sorta like this...
 
 package freenet.keys;
 
 public class ASKKey extends NodeCHK {
 public double toNormalizedDouble() {
 return (super.toNormalizedDouble()+0.25)%1.0;
 }
 }
 
 The only difference is where any node would look for it. This would not
 be exposed to the client. My idea is that any chk could be converted to
 an alternate-location-finding-key just by type (which surely would mean
 a different fetch-command, e.g. fetchCHK/fetchACK...).
 
 There would be no difference in handling, the only difference would be
 how the target-routing-location is identified from the key (the same as
 CHK plus a constant [mod 1.0]). After all, the mapping from the key to
 the small-world location is open to interpretation...
 
 --
 Robert Hailey

But from the perspective of forwarding nodes top block is just another CHK
block, just like any other one. So you'd have to forward the information that it
is a special block through the network.

   - Volodya

- --
http://freedom.libsyn.com/   Echo of Freedom, Radical Podcast
http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal
http://www.freedomporn.org/  Freedom Porn, anarchist and activist smut

 None of us are free until all of us are free.~ Mihail Bakunin
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknwyxcACgkQuWy2EFICg+3KVwCfSwZywD9HH9boKzoGQCzSAohK
G74An0yhV48V9B+Cv5jhfWyMUJLqyh4z
=DKa2
-END PGP SIGNATURE-
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Evan Daniel]

On Thu, Apr 23, 2009 at 4:01 PM, Robert Hailey
rob...@freenetproject.org wrote:

 On Apr 23, 2009, at 12:34 PM, Florent Daigniere wrote:

 Robert Hailey wrote:

 Perhaps there is an easier solution?

 How about extending the chk logic into an alternate-chk-key (ACK?);

 that simply adds 0.25 to the expected location (for routing and

 storage).

 So when you insert the top block, put it in as a chk and an ack (no

 extra uri's neccesary). When you go to fetch it, if the chk does not

 work, try the ack variant of the same key.

 At the moment each node on the path can verify that the data sent by
 previous hop corresponds to what it ought to; How would that work with
 your proposed solution?

 NextGen$

 Sorta like this...
 package freenet.keys;
 public class ASKKey extends NodeCHK {
 public double toNormalizedDouble() {
 return (super.toNormalizedDouble()+0.25)%1.0;
 }
 }
 The only difference is where any node would look for it. This would not be
 exposed to the client. My idea is that any chk could be converted to an
 alternate-location-finding-key just by type (which surely would mean a
 different fetch-command, e.g. fetchCHK/fetchACK...).
 There would be no difference in handling, the only difference would be how
 the target-routing-location is identified from the key (the same as CHK plus
 a constant [mod 1.0]). After all, the mapping from the key to the
 small-world location is open to interpretation...
 --
 Robert Hailey

I suggested the obvious extension of this on IRC.  Instead of simple
searching at location + 0.25, you search at location + n/N, where n is
which copy of the block you're looking for, and N is the number of
copies inserted.

Toad didn't like this because it makes top blocks identifiable to
everyone on the routing path, and involves network-level changes.  The
other approaches can be implemented at a higher level as a translation
before handing a normal CHK request to the network.

Evan Daniel
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[guido]

Am Donnerstag 23 April 2009 14:48:43 schrieb Matthew Toseland:
 I would really appreciate input on option 2 i.e. how much of a problem are
 long CHKs?

If CHK key lengths as they are now are not bad enough to keep people from 
using them, then making them 50% longer won't be, either.

Besides, making the pathname of the file a mandatory part of the key is already 
having larger impact on average key lengths then this would.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 3:09 PM, VolodyA! V Anarhist wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Robert Hailey wrote:

 Sorta like this...

 package freenet.keys;

 public class ASKKey extends NodeCHK {
 public double toNormalizedDouble() {
 return (super.toNormalizedDouble()+0.25)%1.0;
 }
 }

 The only difference is where any node would look for it. This would  
 not
 be exposed to the client. My idea is that any chk could be  
 converted to
 an alternate-location-finding-key just by type (which surely would  
 mean
 a different fetch-command, e.g. fetchCHK/fetchACK...).

 There would be no difference in handling, the only difference would  
 be
 how the target-routing-location is identified from the key (the  
 same as
 CHK plus a constant [mod 1.0]). After all, the mapping from the key  
 to
 the small-world location is open to interpretation...

 --
 Robert Hailey

 But from the perspective of forwarding nodes top block is just  
 another CHK
 block, just like any other one. So you'd have to forward the  
 information that it
 is a special block through the network.

   - Volodya


True, I hadn't thought of that.

It is correctable, though, as via an option flag on the uri we can  
inverted the standard fetching procedure to be mostly alternates, and  
failover to the 'normal' routing location. Then the 'specialness' of  
the alternate request would fall back into the noise of standard chk  
fetching (eventually, after enough data is inserted, etc).

--
Robert Hailey


___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Easy top block duplication: Content Multiplication Keys

[Robert Hailey]

On Apr 23, 2009, at 3:11 PM, Evan Daniel wrote:


I suggested the obvious extension of this on IRC.  Instead of simple
searching at location + 0.25, you search at location + n/N, where n is
which copy of the block you're looking for, and N is the number of
copies inserted.

Toad didn't like this because it makes top blocks identifiable to
everyone on the routing path, and involves network-level changes.  The
other approaches can be implemented at a higher level as a translation
before handing a normal CHK request to the network.

Evan Daniel


The problem...
How to find a 'top block'... either
(1) we have to already know about it on-fetch (long chk uri)
(2) it needs to be derived from existing fetch data (like my idea)
(3) a new mechanism needs to be made (like ssk-top-blocks/matthew's- 
multiplier idea)


At this point, I'm greatly favoring Matthew's idea of multi-content- 
hash/ssk keys.


However, some alternative names to try on...

Reliable Fetch Key  RFK
Reliable Hash Key   RHK
Multiple Hash Key   MHK
Multi-link Key  MLK
Matthew's Multiplier KeyMMK

On Apr 22, 2009, at 7:48 PM, Matthew Toseland wrote:

UMK,N@H(data),extra/filename
(Where N is the number of copies inserted)


And maybe with N in the extra data so as to not introduce a new  
syntax... it is still fetch-time data after all!



MMK@H(data),extra:N/filename



--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Juiceman]

On Thu, Apr 23, 2009 at 8:48 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 I would really appreciate input on option 2 i.e. how much of a problem are
 long CHKs?

 On Thursday 23 April 2009 03:03:00 Matthew Toseland wrote:
 Argh, no, this doesn't work, because the pubkey is known, and there is no
 way
 for the node to verify that the content is in fact valid. An attacker can
 not
 only cause collisions, he can preemptively block known content by inserting
 bogus data to where it would be posted.

 So what does that leave?

 On Thursday 23 April 2009 01:48:46 Matthew Toseland wrote:
  1) Duplicate the top block with SSKs:
  SSK,3@pubkey,cryptokey,extra/filename would insert e.g.
  SSK@pubkey,cryptokey,extra/filename-N for a series of N's.

 Make cryptokey the hash of the content, avoids different-content attacks.

  2) CHKs with extra routing keys:
  CHK@routing key 1,routing key 2,routing key 3,crypto key,extra

 Replace crypto key with the hash of the content, and derive the crypto key
 for
 each block:
 cryptokey_N = H ( H(data) + N )

 3) Reinserting the top block when a download has finished, and also at some
 point after inserting the data. IMHO this may help, but it does not address
 the core problem. We need redundancy over *different areas of the keyspace*.

 4) An FCP command to reinsert a file, given a known top-block key. If we
 cannot reconstruct as-is, we fail, but if we can, we reinsert that block
 (exactly as is, like a binary blob), and the rest of the CHK-based blocks
 under it.

 Maybe a combination of 1) and 4) ?

 On first insert:
 User inserts the data as DSK,3@/filename.
 The node creates a random pubkey, and hashes the content of the top block to
 get the cryptokey. It inserts each block, and returns
 DSK,3@pubkey,cryptokey,extra/filename

 The filename could be ignored if we want.

 On reinsert:
 The original URI must be specified, and have been downloaded. If it is kept
 on
 the download queue then we can simply click a button to reinsert.

 For SSKs:
 User inserts the data as SSK,3@privkey,cryptokey,extra/filename

 We insert to SSK@pubkey,cryptokey,extra/filename-N for N in 0, 1,
 2.

 Since the URI is not content-derived, there is the possibility of the
 inserter
 will insert different content to each slot. AFAICS this cannot be avoided.

 The major disadvantages are:
 - When reinserting we MUST know the original URI.
 - There is no way to heal the alternate top-blocks.

 IMHO the latter is more serious than the former, but full convergence would
 be
 ideal. Option 2 has neither of these problems, but does have very long URIs.
 Mostly keys are copied and pasted, so maybe that isn't a big problem?

 If people are happy with option 2 (very long CHKs), I can implement it
 reasonably quickly...
 
  Neither of these schemes is acceptable IMHO. The former allows for an
 attacker
  to insert different content to different keys, and get some info about
  targets that way, and it is non-convergent, not allowing for reinserts.
 The
  latter doubles the length of the already way too long CHKs.
 
  I propose a new key type which is convergent, has URIs shorter than
 existing
  CHKs, and any number of duplicate blocks: the Content Multiplication Key
 (for
  want of a better name, alternatives are welcome).
 
  DETAILS:
 
  Insert the splitfile normally, except for the top block. The top block
 must
  fit inside a 1KB SSK payload.
 
  Hash the content of the top block:
 
  hash of content: H(data)
 
  Create a private key:
 
  privkey = MAKE_PRIVKEY ( H ( H(data) + 0 ) )
 
  Create the base crypto key:
 
  ckey_base = H ( H ( data + 1 ) )
 
  Create a series of crypto keys:
 
  ckey_N = H ( ckey_base + N )
 
  Insert to a series of SSKs:
 
  SSK@privkey,ckey_N,extra
 
  Announce the key:
 
  UMK,N@H(data),extra/filename
  (Where N is the number of copies inserted)
 
  The filename is ignored. This will make the Frost folk happy.
 






 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl


Extra long CHK's are no problem at all.  As many have pointed out, we
all just cut and paste or clink a link.  I say go for it.  Option 2
with option 3 would make me very happy.

-- 
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Ian Clarke]

On Thu, Apr 23, 2009 at 10:28 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 Is it?  When I talk to non-techies about a client they think I'm referring
 to the person that employs a lawyer.  I think the least confusing term to
 use in this context may be software.

 Very clumbersome. How would you translate Your node is downloading this page
 from Freenet ? The Freenet software running on your computer is downloading
 this page from the Freenet network ?

Freenet is downloading this page from the Freenet network

Speaking plain English isn't brain surgery (or it shouldn't be)!

Ian.


-- 
Ian Clarke
CEO, Uprizer Labs
Email: i...@uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Ian Clarke]

On Thu, Apr 23, 2009 at 3:05 PM, Robert Hailey
rob...@freenetproject.org wrote:
 Yea, but Matthew's language has a more technically-accurate flavor (as
 your node implies the distributed nature of freenet, whereas
 freenet is downloading makes it sound like a monolithic entity).

Technically accurate flavor is secondary to making newbies feel
comfortable.  Newbies are made comfortable by using plain English, and
avoiding terms of art.

Here is a good resource on examples of jargon versus plain English:

  http://www.plainenglish.co.uk/examples/before_and_after.html

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: i...@uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-cvs] r26829 - trunk/freenet/src/freenet/node/fcp

[Daniel Cheng]

2009/4/23 Matthew Toseland t...@amphibian.dyndns.org:
 On Wednesday 15 April 2009 07:43:14 j16s...@freenetproject.org wrote:
 Author: j16sdiz
 Date: 2009-04-15 06:43:12 + (Wed, 15 Apr 2009)
 New Revision: 26829

 Modified:
    trunk/freenet/src/freenet/node/fcp/FCPClient.java
 Log:
 revert r26828: req.cacnel() in removeAll() not work as expected

 Hmmm, what is the problem here? Clearly we are adding to toKill in order to
 mass-cancel?

IIRC, it's getting NPEs, maybe related to activations.
I am just not capable to debug this, so i reverted my changes.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] [freenet-cvs] r27271 - in trunk/freenet/src/freenet/clients/http: . staticfiles staticfiles/js

[Daniel Cheng]

2009/4/24  sas...@freenetproject.org:
 Author: sashee
 Date: 2009-04-23 20:06:00 + (Thu, 23 Apr 2009)
 New Revision: 27271

 Added:
   trunk/freenet/src/freenet/clients/http/staticfiles/js/
   trunk/freenet/src/freenet/clients/http/staticfiles/js/progresspage.js
 Modified:
   trunk/freenet/src/freenet/clients/http/FProxyToadlet.java
   trunk/freenet/src/freenet/clients/http/ToadletContainer.java
   trunk/freenet/src/freenet/clients/http/ToadletContextImpl.java
 Log:
 The progress page is now refreshed with AJAX, if enabled in the configuration 
 and in the browser.

 Modified: trunk/freenet/src/freenet/clients/http/FProxyToadlet.java
 ===
 --- trunk/freenet/src/freenet/clients/http/FProxyToadlet.java   2009-04-23 
 20:04:56 UTC (rev 27270)
 +++ trunk/freenet/src/freenet/clients/http/FProxyToadlet.java   2009-04-23 
 20:06:00 UTC (rev 27271)
 @@ -513,12 +513,26 @@
                                break;
                        } else {
                                // Still in progress
 +                               boolean 
 isJsEnabled=ctx.getContainer().isFProxyJavascriptEnabled();
                                HTMLNode pageNode = 
 ctx.getPageMaker().getPageNode(l10n(fetchingPageTitle), ctx);
 +                               String location = getLink(key, 
 requestedMimeType, maxSize, httprequest.getParam(force, null), 
 httprequest.isParameterSet(forcedownload));
 +                               HTMLNode 
 headNode=ctx.getPageMaker().getHeadNode(pageNode);
 +                               if(isJsEnabled){
 +                                       //If the user has enabled javascript, 
 we add a noscript http refresh(if he has disabled it in the browser)
 +                                       //And the script file
 +                                       
 headNode.addChild(noscript).addChild(meta, http-equiv, 
 Refresh).addAttribute(content, 2;URL= + location);
 +                                       HTMLNode 
 scriptNode=headNode.addChild(script,//abc);
 +                                       scriptNode.addAttribute(type, 
 text/javascript);
 +                                       scriptNode.addAttribute(src, 
 /static/js/progresspage.js);
 +                               }else{
 +                                       //If he disabled it, we just put the 
 http refresh meta, without the noscript
 +                                       headNode.addChild(meta, 
 http-equiv, Refresh).addAttribute(content, 2;URL= + location);
 +                               }
                                HTMLNode contentNode = 
 ctx.getPageMaker().getContentNode(pageNode);
 -
                                HTMLNode infobox = contentNode.addChild(div, 
 class, infobox infobox-information);
                                infobox.addChild(div, class, 
 infobox-header, l10n(fetchingPageBox));
                                HTMLNode infoboxContent = 
 infobox.addChild(div, class, infobox-content);
 +                               infoboxContent.addAttribute(id, 
 infoContent);
                                infoboxContent.addChild(#, 
 l10n(filenameLabel)+  );
                                infoboxContent.addChild(a, href, 
 /+key.toString(false, false), key.getPreferredFilename());
                                if(fr.mimeType != null) 
 infoboxContent.addChild(br, l10n(contentTypeLabel)+ +fr.mimeType);
 @@ -586,9 +600,8 @@

                                ul.addChild(li).addChild(p).addChild(a, 
 new String[] { href, title }, new String[] { /, 
 L10n.getString(Toadlet.homepage) }, l10n(abortToHomepage));

 -                               String location = getLink(key, 
 requestedMimeType, maxSize, httprequest.getParam(force, null), 
 httprequest.isParameterSet(forcedownload));
                                MultiValueTableString, String retHeaders = 
 new MultiValueTableString, String();
 -                               retHeaders.put(Refresh, 2; url=+location);
 +                               //retHeaders.put(Refresh, 2; 
 url=+location);
                                writeHTMLReply(ctx, 200, OK, retHeaders, 
 pageNode.generate());
                                fr.close();
                                fetch.close();

 Modified: trunk/freenet/src/freenet/clients/http/ToadletContainer.java
 ===
 --- trunk/freenet/src/freenet/clients/http/ToadletContainer.java        
 2009-04-23 20:04:56 UTC (rev 27270)
 +++ trunk/freenet/src/freenet/clients/http/ToadletContainer.java        
 2009-04-23 20:06:00 UTC (rev 27271)
 @@ -64,4 +64,6 @@
        public boolean publicGatewayMode();

        public boolean enableActivelinks();
 +
 +       public boolean isFProxyJavascriptEnabled();
  }

 Modified: trunk/freenet/src/freenet/clients/http/ToadletContextImpl.java
 ===
 --- 

Re: [freenet-dev] CMKs don't work, but there are o ther options was Re: Easy top block duplication : Content Multiplication Keys

[Matthew Toseland]

On Thursday 23 April 2009 20:06:13 VolodyA! V Anarhist wrote:
 Matthew Toseland wrote:
  I would really appreciate input on option 2 i.e. how much of a problem are
  long CHKs?
 
 If Long CHKs will become too much of a problem, and people won't mind their
 content being spoofed, then people will start using KSK...

And not only will they be spoofed, they will be unreliable too.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[Matthew Toseland]

On Thursday 23 April 2009 21:14:01 guido wrote:
 Am Donnerstag 23 April 2009 14:48:43 schrieb Matthew Toseland:
  I would really appreciate input on option 2 i.e. how much of a problem are
  long CHKs?
 
 If CHK key lengths as they are now are not bad enough to keep people from 
 using them, then making them 50% longer won't be, either.

Twice as long.
 
 Besides, making the pathname of the file a mandatory part of the key is 
already 
 having larger impact on average key lengths then this would.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Our current web interface and its usability

[Matthew Toseland]

On Friday 24 April 2009 00:44:59 Ian Clarke wrote:
 On Thu, Apr 23, 2009 at 10:28 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  Is it?  When I talk to non-techies about a client they think I'm 
referring
  to the person that employs a lawyer.  I think the least confusing term to
  use in this context may be software.
 
  Very clumbersome. How would you translate Your node is downloading this 
page
  from Freenet ? The Freenet software running on your computer is 
downloading
  this page from the Freenet network ?
 
 Freenet is downloading this page from the Freenet network

Ewww.
 
 Speaking plain English isn't brain surgery (or it shouldn't be)!
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] CMKs don't work, but there are other options was Re: Easy top block duplication: Content Multiplication Keys

[bbackde]

Long CHK keys are ok for me, most important is that they are static
and all inserts produces the same key.
Will the CHK keys have a fix length?

On Fri, Apr 24, 2009 at 03:06, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Thursday 23 April 2009 21:14:01 guido wrote:
 Am Donnerstag 23 April 2009 14:48:43 schrieb Matthew Toseland:
  I would really appreciate input on option 2 i.e. how much of a problem are
  long CHKs?

 If CHK key lengths as they are now are not bad enough to keep people from
 using them, then making them 50% longer won't be, either.

 Twice as long.

 Besides, making the pathname of the file a mandatory part of the key is
 already
 having larger impact on average key lengths then this would.
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl




-- 
__
GnuPG key:   (0x48DBFA8A)
Keyserver:   pgpkeys.pca.dfn.de
Fingerprint:
477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A
__
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl