[freenet-dev] Non-convergent encryption kills easy filesharing

[Daniel Cheng]

On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
 wrote:
> After a long conversation with p0s, I am fairly sure that our decision at last
> year's summit to use non-convergent encryption for splitfiles (i.e. a
> different set of blocks each time) in order to largely solve our security
> problems will make filesharing on Freenet much less convenient.
>
> Other points:
> - The problem with persistence of data is frequently the top block. This can
> be solved by duplicating it. This is easy for SSKs, but for CHKs would
> require quite long URIs. This may be acceptable given the likely data
> persistence gains.
> - We must deal with the short segments problem.
> - Node references should be tolerant of losing newlines and gaining new ones.
>
> I had hoped that we could release a 0.8.1 with solutions to the data
> persistence problems and non-convergent encryption, but it looks like we will
> need to wait for 0.9 with tunnels and bloom filter sharing. :(

I believe the persistence would be improved with tunnels
  -- blocks can be insert from different paths and cache differently.

[freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

> -Original Message-
> From: arne_bab at web.de [mailto:arne_bab at web.de] 
> Sent: Thursday, April 23, 2009 11:21 AM
> To: devl at freenetproject.org
> Cc: xor
> Subject: Re: [freenet-dev] Non-convergent encryption kills 
> easy filesharing
> 
> Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
> > > -Original Message-
> > > From: devl-bounces at freenetproject.org 
> > > [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne 
> > > Babenhauserheide
> > > Sent: Thursday, April 23, 2009 9:14 AM
> > > To: devl at freenetproject.org
> > > Subject: Re: [freenet-dev] Non-convergent encryption kills easy 
> > > filesharing
> > >
> > > Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > > > block [16:39:31]  duplicating the top block can be done 
> > > > with SSKs very easily [16:39:40]  but with CHKs 
> it requires 
> > > > much longer URIs [16:39:43]  is that a problem?
> > > > [16:40:04]  how much longer?
> > > > [16:40:10]  CHK@,, -> 
> > > > CHK@,,, > > > key>, [16:40:29]  i.e. at least twice as long
> > >
> > > I stopped reading at about the middle, so it might be 
> that I missed 
> > > something, but why do you care if CHKs become longer?
> > >
> > > They are already so long that noone types them (I hope), and I 
> > > really don't care if the key I copy-paste has two times 
> the length.
> >
> > If they are exchanged via Freetalk it will bloat the messages.
> > Further, I was told to implement CHK messages in Freetalk because 
> > messages could not fit into SSK. So now Freetalk uses SSK message 
> > lists to publish CHK URI of the messages... If the CHK URI become 
> > insanely long then the message lists will be bloated very much and 
> > maybe only five messages or so will fit into a SSK message 
> list. Sucks.
> 
> So why don't you take the approach of filesystems and add an 
> option where the message list SSK can contain a link to a CHK 
> which contains all message CHK URIs?

1. Too complex.
2. The problem is only shifted to another place. The problem with message
lists is that they must fit as many message references into a single SSK
block as possible to gurantee high network efficiency. You cannot tell me to
solve the space problem by INCREASING the used space by adding an extra CHK
block.

> 
> (that's what ext2 does with inodes to support almost 
> arbitrarily large files, though each inodes is only 4k in size) 
> 
> If the efficiency of CHKs should rise by more than a factor 
> of 2 when using the large URI instead of non-republishable 
> URIs, freetalk will even profit from this (ping time wise). 
> 
> But I don't know the implementation details of freetalk, so I 
> don't know if this is a possible option for you. 
> 
> Best wishes,
> Arne
> --
> -- Ein W?rfel System: http://1w6.org - einfach saubere 
> (Rollenspiel-) Regeln.
> -- Infinite Hands: http://infinite-hands.draketo.de - singing 
> a part of the history of free software.
> -- My stuff: http://draketo.de - stories, songs, poems, 
> programs and stuff :)
> 
> -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 09:37:45 Daniel Cheng wrote:
> On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
>  wrote:
> > After a long conversation with p0s, I am fairly sure that our decision at 
last
> > year's summit to use non-convergent encryption for splitfiles (i.e. a
> > different set of blocks each time) in order to largely solve our security
> > problems will make filesharing on Freenet much less convenient.
> >
> > Other points:
> > - The problem with persistence of data is frequently the top block. This 
can
> > be solved by duplicating it. This is easy for SSKs, but for CHKs would
> > require quite long URIs. This may be acceptable given the likely data
> > persistence gains.
> > - We must deal with the short segments problem.
> > - Node references should be tolerant of losing newlines and gaining new 
ones.
> >
> > I had hoped that we could release a 0.8.1 with solutions to the data
> > persistence problems and non-convergent encryption, but it looks like we 
will
> > need to wait for 0.9 with tunnels and bloom filter sharing. :(
> 
> I believe the persistence would be improved with tunnels
>   -- blocks can be insert from different paths and cache differently.

Why? If specialisation works, and IMHO we have no reason to think it doesn't, 
then it will land up on the same nodes. The problem is simply that those 
nodes go offline.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
> > -Original Message-
> > From: devl-bounces at freenetproject.org
> > [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne
> > Babenhauserheide
> > Sent: Thursday, April 23, 2009 9:14 AM
> > To: devl at freenetproject.org
> > Subject: Re: [freenet-dev] Non-convergent encryption kills
> > easy filesharing
> >
> > Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > > block [16:39:31]  duplicating the top block can be done with
> > > SSKs very easily [16:39:40]  but with CHKs it requires much
> > > longer URIs [16:39:43]  is that a problem?
> > > [16:40:04]  how much longer?
> > > [16:40:10]  CHK@,, ->
> > > CHK@,,, > > key>, [16:40:29]  i.e. at least twice as long
> >
> > I stopped reading at about the middle, so it might be that I
> > missed something, but why do you care if CHKs become longer?
> >
> > They are already so long that noone types them (I hope), and
> > I really don't care if the key I copy-paste has two times the length.
>
> If they are exchanged via Freetalk it will bloat the messages.
> Further, I was told to implement CHK messages in Freetalk because messages
> could not fit into SSK. So now Freetalk uses SSK message lists to publish
> CHK URI of the messages... If the CHK URI become insanely long then the
> message lists will be bloated very much and maybe only five messages or so
> will fit into a SSK message list. Sucks.

So why don't you take the approach of filesystems and add an option where the 
message list SSK can contain a link to a CHK which contains all message CHK 
URIs? 

(that's what ext2 does with inodes to support almost arbitrarily large files, 
though each inodes is only 4k in size) 

If the efficiency of CHKs should rise by more than a factor of 2 when using 
the large URI instead of non-republishable URIs, freetalk will even profit 
from this (ping time wise). 

But I don't know the implementation details of freetalk, so I don't know if 
this is a possible option for you. 

Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

[freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

> -Original Message-
> From: devl-bounces at freenetproject.org 
> [mailto:devl-bounces at freenetproject.org] On Behalf Of Arne 
> Babenhauserheide
> Sent: Thursday, April 23, 2009 9:14 AM
> To: devl at freenetproject.org
> Subject: Re: [freenet-dev] Non-convergent encryption kills 
> easy filesharing
> 
> Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> > block [16:39:31]  duplicating the top block can be done with 
> > SSKs very easily [16:39:40]  but with CHKs it requires much 
> > longer URIs [16:39:43]  is that a problem?
> > [16:40:04]  how much longer?
> > [16:40:10]  CHK@,, -> 
> > CHK@,,, > key>, [16:40:29]  i.e. at least twice as long
> 
> I stopped reading at about the middle, so it might be that I 
> missed something, but why do you care if CHKs become longer? 
> 
> They are already so long that noone types them (I hope), and 
> I really don't care if the key I copy-paste has two times the length. 

If they are exchanged via Freetalk it will bloat the messages.
Further, I was told to implement CHK messages in Freetalk because messages
could not fit into SSK. So now Freetalk uses SSK message lists to publish
CHK URI of the messages... If the CHK URI become insanely long then the
message lists will be bloated very much and maybe only five messages or so
will fit into a SSK message list. Sucks.

> They are longer than my browser window is wide, so I won't 
> even notice it, if they become longer. 
> 
> And on freesites, we'll use links anyway, where the form of 
> the URL doesn't really matter. 
> 
> So if only the key length keeps you from doing the right 
> thing, just do the right thing anyway. 
> 
> Best wishes,
> Arne
> --
> -- Ein W?rfel System: http://1w6.org - einfach saubere 
> (Rollenspiel-) Regeln.
> -- Infinite Hands: http://infinite-hands.draketo.de - singing 
> a part of the history of free software.
> -- My stuff: http://draketo.de - stories, songs, poems, 
> programs and stuff :)
> 
> -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
> ___
> Devl mailing list
> Devl at freenetproject.org
> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
> block [16:39:31]  duplicating the top block can be done with SSKs
> very easily [16:39:40]  but with CHKs it requires much longer URIs
> [16:39:43]  is that a problem?
> [16:40:04]  how much longer?
> [16:40:10]  CHK@,, -> CHK@ key 1>
> [16:40:29]  i.e. at least twice as long

I stopped reading at about the middle, so it might be that I missed something, 
but why do you care if CHKs become longer? 

They are already so long that noone types them (I hope), and I really don't 
care if the key I copy-paste has two times the length. 

They are longer than my browser window is wide, so I won't even notice it, if 
they become longer. 

And on freesites, we'll use links anyway, where the form of the URL doesn't 
really matter. 

So if only the key length keeps you from doing the right thing, just do the 
right thing anyway. 

Best wishes, 
Arne
-- 
-- Ein W?rfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
 block [16:39:31] toad_ duplicating the top block can be done with SSKs
 very easily [16:39:40] toad_ but with CHKs it requires much longer URIs
 [16:39:43] toad_ is that a problem?
 [16:40:04] p0s how much longer?
 [16:40:10] toad_ CHK@routing key,decrypt key,extra - CHK@routing
 key 1,routing key 2,routing key 3,decrypt key,extra
 [16:40:29] toad_ i.e. at least twice as long

I stopped reading at about the middle, so it might be that I missed something, 
but why do you care if CHKs become longer? 

They are already so long that noone types them (I hope), and I really don't 
care if the key I copy-paste has two times the length. 

They are longer than my browser window is wide, so I won't even notice it, if 
they become longer. 

And on freesites, we'll use links anyway, where the form of the URL doesn't 
really matter. 

So if only the key length keeps you from doing the right thing, just do the 
right thing anyway. 

Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

 

 -Original Message-
 From: devl-boun...@freenetproject.org 
 [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
 Babenhauserheide
 Sent: Thursday, April 23, 2009 9:14 AM
 To: devl@freenetproject.org
 Subject: Re: [freenet-dev] Non-convergent encryption kills 
 easy filesharing
 
 Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
  block [16:39:31] toad_ duplicating the top block can be done with 
  SSKs very easily [16:39:40] toad_ but with CHKs it requires much 
  longer URIs [16:39:43] toad_ is that a problem?
  [16:40:04] p0s how much longer?
  [16:40:10] toad_ CHK@routing key,decrypt key,extra - 
  CHK@routing key 1,routing key 2,routing key 3,decrypt 
  key,extra [16:40:29] toad_ i.e. at least twice as long
 
 I stopped reading at about the middle, so it might be that I 
 missed something, but why do you care if CHKs become longer? 
 
 They are already so long that noone types them (I hope), and 
 I really don't care if the key I copy-paste has two times the length. 

If they are exchanged via Freetalk it will bloat the messages.
Further, I was told to implement CHK messages in Freetalk because messages
could not fit into SSK. So now Freetalk uses SSK message lists to publish
CHK URI of the messages... If the CHK URI become insanely long then the
message lists will be bloated very much and maybe only five messages or so
will fit into a SSK message list. Sucks.

 They are longer than my browser window is wide, so I won't 
 even notice it, if they become longer. 
 
 And on freesites, we'll use links anyway, where the form of 
 the URL doesn't really matter. 
 
 So if only the key length keeps you from doing the right 
 thing, just do the right thing anyway. 
 
 Best wishes,
 Arne
 --
 -- Ein Würfel System: http://1w6.org - einfach saubere 
 (Rollenspiel-) Regeln.
 -- Infinite Hands: http://infinite-hands.draketo.de - singing 
 a part of the history of free software.
 -- My stuff: http://draketo.de - stories, songs, poems, 
 programs and stuff :)
 
 -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
 ___
 Devl mailing list
 Devl@freenetproject.org
 http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Daniel Cheng]

On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 After a long conversation with p0s, I am fairly sure that our decision at last
 year's summit to use non-convergent encryption for splitfiles (i.e. a
 different set of blocks each time) in order to largely solve our security
 problems will make filesharing on Freenet much less convenient.

 Other points:
 - The problem with persistence of data is frequently the top block. This can
 be solved by duplicating it. This is easy for SSKs, but for CHKs would
 require quite long URIs. This may be acceptable given the likely data
 persistence gains.
 - We must deal with the short segments problem.
 - Node references should be tolerant of losing newlines and gaining new ones.

 I had hoped that we could release a 0.8.1 with solutions to the data
 persistence problems and non-convergent encryption, but it looks like we will
 need to wait for 0.9 with tunnels and bloom filter sharing. :(

I believe the persistence would be improved with tunnels
  -- blocks can be insert from different paths and cache differently.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Arne Babenhauserheide]

Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
  -Original Message-
  From: devl-boun...@freenetproject.org
  [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne
  Babenhauserheide
  Sent: Thursday, April 23, 2009 9:14 AM
  To: devl@freenetproject.org
  Subject: Re: [freenet-dev] Non-convergent encryption kills
  easy filesharing
 
  Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
   block [16:39:31] toad_ duplicating the top block can be done with
   SSKs very easily [16:39:40] toad_ but with CHKs it requires much
   longer URIs [16:39:43] toad_ is that a problem?
   [16:40:04] p0s how much longer?
   [16:40:10] toad_ CHK@routing key,decrypt key,extra -
   CHK@routing key 1,routing key 2,routing key 3,decrypt
   key,extra [16:40:29] toad_ i.e. at least twice as long
 
  I stopped reading at about the middle, so it might be that I
  missed something, but why do you care if CHKs become longer?
 
  They are already so long that noone types them (I hope), and
  I really don't care if the key I copy-paste has two times the length.

 If they are exchanged via Freetalk it will bloat the messages.
 Further, I was told to implement CHK messages in Freetalk because messages
 could not fit into SSK. So now Freetalk uses SSK message lists to publish
 CHK URI of the messages... If the CHK URI become insanely long then the
 message lists will be bloated very much and maybe only five messages or so
 will fit into a SSK message list. Sucks.

So why don't you take the approach of filesystems and add an option where the 
message list SSK can contain a link to a CHK which contains all message CHK 
URIs? 

(that's what ext2 does with inodes to support almost arbitrarily large files, 
though each inodes is only 4k in size) 

If the efficiency of CHKs should rise by more than a factor of 2 when using 
the large URI instead of non-republishable URIs, freetalk will even profit 
from this (ping time wise). 

But I don't know the implementation details of freetalk, so I don't know if 
this is a possible option for you. 

Best wishes, 
Arne
-- 
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 08:25:15 xor wrote:
 
  -Original Message-
  From: devl-boun...@freenetproject.org 
  [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
  Babenhauserheide
  Sent: Thursday, April 23, 2009 9:14 AM
  To: devl@freenetproject.org
  Subject: Re: [freenet-dev] Non-convergent encryption kills 
  easy filesharing
  
  Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
   block [16:39:31] toad_ duplicating the top block can be done with 
   SSKs very easily [16:39:40] toad_ but with CHKs it requires much 
   longer URIs [16:39:43] toad_ is that a problem?
   [16:40:04] p0s how much longer?
   [16:40:10] toad_ CHK@routing key,decrypt key,extra - 
   CHK@routing key 1,routing key 2,routing key 3,decrypt 
   key,extra [16:40:29] toad_ i.e. at least twice as long
  
  I stopped reading at about the middle, so it might be that I 
  missed something, but why do you care if CHKs become longer? 
  
  They are already so long that noone types them (I hope), and 
  I really don't care if the key I copy-paste has two times the length. 
 
 If they are exchanged via Freetalk it will bloat the messages.
 Further, I was told to implement CHK messages in Freetalk because messages
 could not fit into SSK. So now Freetalk uses SSK message lists to publish
 CHK URI of the messages... If the CHK URI become insanely long then the
 message lists will be bloated very much and maybe only five messages or so
 will fit into a SSK message list. Sucks.

No, this would not be normal CHKs, sorry. I'd probably call them DHKs. They 
would only be used for the top block of big files.
 
  They are longer than my browser window is wide, so I won't 
  even notice it, if they become longer. 
  
  And on freesites, we'll use links anyway, where the form of 
  the URL doesn't really matter. 
  
  So if only the key length keeps you from doing the right 
  thing, just do the right thing anyway. 
  
  Best wishes,
  Arne


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

On Thursday 23 April 2009 09:37:45 Daniel Cheng wrote:
 On Thu, Apr 23, 2009 at 12:26 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  After a long conversation with p0s, I am fairly sure that our decision at 
last
  year's summit to use non-convergent encryption for splitfiles (i.e. a
  different set of blocks each time) in order to largely solve our security
  problems will make filesharing on Freenet much less convenient.
 
  Other points:
  - The problem with persistence of data is frequently the top block. This 
can
  be solved by duplicating it. This is easy for SSKs, but for CHKs would
  require quite long URIs. This may be acceptable given the likely data
  persistence gains.
  - We must deal with the short segments problem.
  - Node references should be tolerant of losing newlines and gaining new 
ones.
 
  I had hoped that we could release a 0.8.1 with solutions to the data
  persistence problems and non-convergent encryption, but it looks like we 
will
  need to wait for 0.9 with tunnels and bloom filter sharing. :(
 
 I believe the persistence would be improved with tunnels
   -- blocks can be insert from different paths and cache differently.

Why? If specialisation works, and IMHO we have no reason to think it doesn't, 
then it will land up on the same nodes. The problem is simply that those 
nodes go offline.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Non-convergent encryption kills easy filesharing

[xor]

 

 -Original Message-
 From: arne_...@web.de [mailto:arne_...@web.de] 
 Sent: Thursday, April 23, 2009 11:21 AM
 To: devl@freenetproject.org
 Cc: xor
 Subject: Re: [freenet-dev] Non-convergent encryption kills 
 easy filesharing
 
 Am Donnerstag 23 April 2009 09:25:15 schrieb xor:
   -Original Message-
   From: devl-boun...@freenetproject.org 
   [mailto:devl-boun...@freenetproject.org] On Behalf Of Arne 
   Babenhauserheide
   Sent: Thursday, April 23, 2009 9:14 AM
   To: devl@freenetproject.org
   Subject: Re: [freenet-dev] Non-convergent encryption kills easy 
   filesharing
  
   Am Mittwoch 22 April 2009 18:26:05 schrieb Matthew Toseland:
block [16:39:31] toad_ duplicating the top block can be done 
with SSKs very easily [16:39:40] toad_ but with CHKs 
 it requires 
much longer URIs [16:39:43] toad_ is that a problem?
[16:40:04] p0s how much longer?
[16:40:10] toad_ CHK@routing key,decrypt key,extra - 
CHK@routing key 1,routing key 2,routing key 3,decrypt
key,extra [16:40:29] toad_ i.e. at least twice as long
  
   I stopped reading at about the middle, so it might be 
 that I missed 
   something, but why do you care if CHKs become longer?
  
   They are already so long that noone types them (I hope), and I 
   really don't care if the key I copy-paste has two times 
 the length.
 
  If they are exchanged via Freetalk it will bloat the messages.
  Further, I was told to implement CHK messages in Freetalk because 
  messages could not fit into SSK. So now Freetalk uses SSK message 
  lists to publish CHK URI of the messages... If the CHK URI become 
  insanely long then the message lists will be bloated very much and 
  maybe only five messages or so will fit into a SSK message 
 list. Sucks.
 
 So why don't you take the approach of filesystems and add an 
 option where the message list SSK can contain a link to a CHK 
 which contains all message CHK URIs?

1. Too complex.
2. The problem is only shifted to another place. The problem with message
lists is that they must fit as many message references into a single SSK
block as possible to gurantee high network efficiency. You cannot tell me to
solve the space problem by INCREASING the used space by adding an extra CHK
block.
 
 
 (that's what ext2 does with inodes to support almost 
 arbitrarily large files, though each inodes is only 4k in size) 
 
 If the efficiency of CHKs should rise by more than a factor 
 of 2 when using the large URI instead of non-republishable 
 URIs, freetalk will even profit from this (ping time wise). 
 
 But I don't know the implementation details of freetalk, so I 
 don't know if this is a possible option for you. 
 
 Best wishes,
 Arne
 --
 -- Ein Würfel System: http://1w6.org - einfach saubere 
 (Rollenspiel-) Regeln.
 -- Infinite Hands: http://infinite-hands.draketo.de - singing 
 a part of the history of free software.
 -- My stuff: http://draketo.de - stories, songs, poems, 
 programs and stuff :)
 
 -- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

After a long conversation with p0s, I am fairly sure that our decision at last 
year's summit to use non-convergent encryption for splitfiles (i.e. a 
different set of blocks each time) in order to largely solve our security 
problems will make filesharing on Freenet much less convenient.

Other points:
- The problem with persistence of data is frequently the top block. This can 
be solved by duplicating it. This is easy for SSKs, but for CHKs would 
require quite long URIs. This may be acceptable given the likely data 
persistence gains.
- We must deal with the short segments problem.
- Node references should be tolerant of losing newlines and gaining new ones.

I had hoped that we could release a 0.8.1 with solutions to the data 
persistence problems and non-convergent encryption, but it looks like we will 
need to wait for 0.9 with tunnels and bloom filter sharing. :(

[16:22:39]  toad_: hi. yesterday i told you about the bad user experience 
of one of my friends ("the node is slow in general") and that he had updated 
to trunk then... i visited him today and he said that it isn't sluggish 
anymore and the pages which did not work with 1208 work now with trunk
[16:22:55]  toad_: so its maybe time to consider releasing 1209
[16:23:13]  yeah that's my thinking
[16:23:16]  but we need more testing
[16:23:27]  any specific testing i can help with?
[16:24:14]  dunno
[16:26:02]  well i'll just update to trunk and see if it announces etc
[16:26:19]  yeah, bootstrap tests are always a good idea before 
releasing
[16:26:23]  but it still feels like downloads stall at 0%
[16:26:25] * toad_ needs to wade through the 270 commits on cvs@
[16:26:41]  downloads frequently stall at 0%, the reason is that the 
top block can't be found
[16:26:44]  afaik
[16:26:47]  one has started this morning at few blocks, below 30 or so... 
after not starting for several days... now its at 38% !
[16:28:08]  i just wonder why it does not find the top block for almost a 
week at 3 downloads at once
[16:28:49]  because the non-top blocks have tons of redundancy, whereas 
with the top block it's a matter of whether the 3 or 4 nodes that have it in 
their stores are online at the time, not backed off, etc
[16:29:13]  the solution is to duplicate the top block, this is hard 
for CHKs but very easy for SSKs
[16:30:58]  the other thing to deal with is segment size
[16:31:16]  unfortunately SSKs are not the large files which usually get 
screwed by missing top blocks...
[16:31:17]  put those together we should improve data reliability quite 
a bit
[16:31:22]  p0s: well, they should be !
[16:31:36]  all my testing is with CHK
[16:31:54]  they will be if we make it clear that inserting as CHK is 
deprecated and in any case will yield a different key each time because of 
non-convergent encryption (a security feature)
[16:31:56]  SSKs work well IMHO
[16:32:06]  CHK is deprecated? wtf?
[16:32:20]  it isn't yet, but imho it will be
[16:32:22]  the point of CHK is to allow anyone to re-insert if the 
original inserter is gone
[16:32:31]  in favour of SSK,3@
[16:32:34]  non-convergent encryption? why?
[16:32:36]  p0s: well we are abolishing that
[16:32:50]  because predictable keys are *REALLY* bad for security
[16:33:20]  for opennet attacks or also if you insert/fetch as darknet 
only?
[16:33:39]  either way, predictable keys make mobile attacker tracing 
of an inserter much easier
[16:33:59]  on opennet getting connections to target nodes is of course 
much easier than on darknet
[16:34:06]  but it will screw data-availability even more because people 
cannot re-insert!
[16:34:50]  for SSKs it does not matter because freetalk messages and 
freesites will probably be 99% of the SSKs and those will not be re-inserted 
anyway
[16:34:53]  they can reinsert they just have to announce the new keys!
[16:34:56]  but for filesharing this SUCKS
[16:35:04]  it will screw the avaiability all over
[16:35:19]  do you think that anyone cares to announce new keys? i really 
doubt it
[16:35:21]  no, like you said, the main problem is you can never fetch 
the top block
[16:35:35]  yes and that can be solved by re-inserting
[16:35:50]  but no chance that people will transfer the new keys over and 
over again if they re-insert that would be a major annoyance
[16:36:07]  full PITA
[16:36:11]  no it can't be solved by reinserting
[16:36:26]  why?
[16:36:32]  the current situation is after 2 weeks stuff is no longer 
in cache, it's only in store
[16:36:40]  so it takes weeks and months to fetch the top block
[16:36:45]  but once we have the top block, it's not so bad
[16:36:52]  apart from the over-short last segment
[16:37:01]  but the top block will also be re-inserted, won't it?
[16:37:14]  yes but only if the file is reinserted
[16:37:24]  i am talking about manual re-inserts here
[16:37:28]  manual re-inserts should work
[16:37:33]  you really want a filesharing system where you have to 
reinsert everything every 2 weeks?
[16:37:37]  i am against making manual 

[freenet-dev] Non-convergent encryption kills easy filesharing

[Matthew Toseland]

After a long conversation with p0s, I am fairly sure that our decision at last 
year's summit to use non-convergent encryption for splitfiles (i.e. a 
different set of blocks each time) in order to largely solve our security 
problems will make filesharing on Freenet much less convenient.

Other points:
- The problem with persistence of data is frequently the top block. This can 
be solved by duplicating it. This is easy for SSKs, but for CHKs would 
require quite long URIs. This may be acceptable given the likely data 
persistence gains.
- We must deal with the short segments problem.
- Node references should be tolerant of losing newlines and gaining new ones.

I had hoped that we could release a 0.8.1 with solutions to the data 
persistence problems and non-convergent encryption, but it looks like we will 
need to wait for 0.9 with tunnels and bloom filter sharing. :(

[16:22:39] p0s toad_: hi. yesterday i told you about the bad user experience 
of one of my friends (the node is slow in general) and that he had updated 
to trunk then... i visited him today and he said that it isn't sluggish 
anymore and the pages which did not work with 1208 work now with trunk
[16:22:55] p0s toad_: so its maybe time to consider releasing 1209
[16:23:13] toad_ yeah that's my thinking
[16:23:16] toad_ but we need more testing
[16:23:27] p0s any specific testing i can help with?
[16:24:14] toad_ dunno
[16:26:02] p0s well i'll just update to trunk and see if it announces etc
[16:26:19] toad_ yeah, bootstrap tests are always a good idea before 
releasing
[16:26:23] p0s but it still feels like downloads stall at 0%
[16:26:25] * toad_ needs to wade through the 270 commits on cvs@
[16:26:41] toad_ downloads frequently stall at 0%, the reason is that the 
top block can't be found
[16:26:44] toad_ afaik
[16:26:47] p0s one has started this morning at few blocks, below 30 or so... 
after not starting for several days... now its at 38% !
[16:28:08] p0s i just wonder why it does not find the top block for almost a 
week at 3 downloads at once
[16:28:49] toad_ because the non-top blocks have tons of redundancy, whereas 
with the top block it's a matter of whether the 3 or 4 nodes that have it in 
their stores are online at the time, not backed off, etc
[16:29:13] toad_ the solution is to duplicate the top block, this is hard 
for CHKs but very easy for SSKs
[16:30:58] toad_ the other thing to deal with is segment size
[16:31:16] p0s unfortunately SSKs are not the large files which usually get 
screwed by missing top blocks...
[16:31:17] toad_ put those together we should improve data reliability quite 
a bit
[16:31:22] toad_ p0s: well, they should be !
[16:31:36] p0s all my testing is with CHK
[16:31:54] toad_ they will be if we make it clear that inserting as CHK is 
deprecated and in any case will yield a different key each time because of 
non-convergent encryption (a security feature)
[16:31:56] p0s SSKs work well IMHO
[16:32:06] p0s CHK is deprecated? wtf?
[16:32:20] toad_ it isn't yet, but imho it will be
[16:32:22] p0s the point of CHK is to allow anyone to re-insert if the 
original inserter is gone
[16:32:31] toad_ in favour of SSK,3@
[16:32:34] p0s non-convergent encryption? why?
[16:32:36] toad_ p0s: well we are abolishing that
[16:32:50] toad_ because predictable keys are *REALLY* bad for security
[16:33:20] p0s for opennet attacks or also if you insert/fetch as darknet 
only?
[16:33:39] toad_ either way, predictable keys make mobile attacker tracing 
of an inserter much easier
[16:33:59] toad_ on opennet getting connections to target nodes is of course 
much easier than on darknet
[16:34:06] p0s but it will screw data-availability even more because people 
cannot re-insert!
[16:34:50] p0s for SSKs it does not matter because freetalk messages and 
freesites will probably be 99% of the SSKs and those will not be re-inserted 
anyway
[16:34:53] toad_ they can reinsert they just have to announce the new keys!
[16:34:56] p0s but for filesharing this SUCKS
[16:35:04] p0s it will screw the avaiability all over
[16:35:19] p0s do you think that anyone cares to announce new keys? i really 
doubt it
[16:35:21] toad_ no, like you said, the main problem is you can never fetch 
the top block
[16:35:35] p0s yes and that can be solved by re-inserting
[16:35:50] p0s but no chance that people will transfer the new keys over and 
over again if they re-insert that would be a major annoyance
[16:36:07] p0s full PITA
[16:36:11] toad_ no it can't be solved by reinserting
[16:36:26] p0s why?
[16:36:32] toad_ the current situation is after 2 weeks stuff is no longer 
in cache, it's only in store
[16:36:40] toad_ so it takes weeks and months to fetch the top block
[16:36:45] toad_ but once we have the top block, it's not so bad
[16:36:52] toad_ apart from the over-short last segment
[16:37:01] p0s but the top block will also be re-inserted, won't it?
[16:37:14] toad_ yes but only if the file is reinserted
[16:37:24] p0s i am talking about manual