date:20090512

I will post this on the website tomorrow if there are no objections. If anyone 
can suggest any improvements, please do so; sometimes what I write isn't 
readable by human beings!

7th May, 2009 - Another big donation!

Google's Open Source team has donated US$18,000 to the Freenet Project to 
support the ongoing development of the Freenet software (thanks again 
Google!).

Their last donation funded the db4o project, which has now been merged into 
Freenet, greatly improving performance for large download queues while 
reducing memory usage, amongst other benefits.

We are currently working on Freenet 0.8, which will be released later this 
year, and will include additional performance improvements, usability work, 
and security improvements, as well as the usual debugging. Features are not 
yet finalized but we expect it to include Freetalk (a new anonymous web 
forums tool), a new Vista-compatible installer for Windows (that part will be 
out in a few days), and hopefully Bloom filter sharing, a new feature 
enabling nodes to know what is in their peers' datastores, greatly improving 
performance, combined with some related security improvements.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/f477a9a5/attachment.pgp>

[freenet-dev] Recent progress on Interdex

src/plugins/Interdex/index/IndexBloomFilter.java :
We already have bloom filter implementations in 
src/freenet/support/*BloomFilter.java. You should use these internally, no 
need to reimplement.

Is it a good idea to use MD5? I guess you're using it the same way that 
XMLLibrarian does, but it may be more of a problem for your application?

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

2009-05-12 Thread Florent Daigniere

Robert Hailey wrote:
> 
> On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:
> 
>> ...
>> So the question is, how practical is it for a mobile attacker to  
>> identify a
>> block in its neighbours' stores (from the bloom filter), get  
>> connected to the
>> neighbour's neighbours, download their bloom filters, and trace an  
>> insert or
>> request back to source one hop at a time?
> 
> At first thought, I would imagine if bloom filter sharing increases  
> the fetch effectiveness by x20, so to it would make such an attack 20  
> times easier. A powerful attacker might track ~everyone's~ bloom  
> filter
> 
> What would the security implications be if we were to have the node  
> with the store check the bloom filter rather than sharing it (~one-hop- 
> probe~)? Would that buy us anything, being able to mete out bloom  
> filter checks? Then such an attacker could not check to see if we  
> contained ~80% of a split file because he would have to ask us to  
> check our bloom filter for so many keys. It would also be an order of  
> magnitude easier to implement than "sharing".
> 

Huh.

What are you suggesting here? That we re-invent the current protocol?

The whole point of sharing bloom filters is to get rid of the 
network/node latency. We already have local bloom filters to avoid 
unnecessary datastore lookups.

NextGen$

[freenet-dev] Request for proofreading: Announcing donation from Google

2009-05-12 Thread Ian Clarke

On Tue, May 12, 2009 at 2:36 PM, Matthew Toseland
 wrote:
> 7th May, 2009 - Another big donation!

How about: "Google makes a second donation to Freenet!"

Everything else looks good.

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: ian at uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674

[freenet-dev] What to do on SSK collisions: f86448d51c2e3248e1dfec513eefde50902aac30

On Tue, May 12, 2009 at 7:49 AM, Matthew Toseland
 wrote:
> On Tuesday 12 May 2009 00:45:54 Matthew Toseland wrote:
>> commit f86448d51c2e3248e1dfec513eefde50902aac30
>> Author: Daniel Cheng (???) 
>> Date:   Fri May 8 21:04:28 2009 +0800
>>
>> FreenetStore: Simplify code, remove "overwrite" parameter
>>
>> This parameter is always "false".
>> (Except when doing BDB->SaltedHash migration, which does not have to
>> overwrite)
>>
>>
>> The reason I introduced the overwrite parameter was that when we send an SSK
>> insert and get a DataFound with different data, we should not only propagate
>> the data downstream, but also replace our local copy of it. However
>> apparently I never implemented this. Is it a good idea?
>>
> Looks like we do store the collided data for a local insert
> (NodeClientCore.java:XXX we have collided), but not for a remote one. Except
> that a bug prevents the former from working. So we need to fix getBlock().
> Ok...
>

Let's see if these two commit make sense: (this is on my fork, not
committed to the main staging yet)

http://github.com/j16sdiz/fred/commit/8e2ef42c286450813dbfa575bcd3f54dc8cb4c83
http://github.com/j16sdiz/fred/commit/7e6040ce3359486557bdd832c526e473a4f95577

Regards,
Daniel

[freenet-dev] XMLLibrarian changes

On Tuesday 12 May 2009 14:16:54 Matthew Toseland wrote:
> commit c6f84c96d60a1e53126824e8c2fa74c1f1d4b658
> Author: platy 
> Date:   Sun May 3 13:52:16 2009 +0100
> 
> Search progress now works with or without javascript enabled, non-js get 
a 
> link to view results when search completes
> 
> 
> Great! But...
> 
> Why all the while(blah) { Thread.sleep() } 's ? Wouldn't wait/notify be 
> better?
> 
> What is the purpose of the hidden Find! button?
> 
> commit 3a9754b92ea88bfe99019af8810f96d537ff3a79
> Author: platy 
> Date:   Tue May 5 03:43:39 2009 +0100
> 
> Removed use of JS & Iframes entirely
> 
> Why? Isn't javascript useful to refresh only part of the page, avoid the 
> clunky feeling of reloading frequently? Clearly refresh is better for 
> non-javascript... It does work pretty well now though, thanks!
> 
> We should include the searched-for term in the title of the page.

I have made some changes:
- Refactored handleInner() a bit.
- Fixed a bug causing infinite looping continually fetching and re-fetching 
the same index (for me it happens when i search for antarctica).
- Fixed the Searching for blah\nnull bug.

The plugin will be deployed soon.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/66826bd8/attachment.pgp>

[freenet-dev] XMLLibrarian changes

commit c6f84c96d60a1e53126824e8c2fa74c1f1d4b658
Author: platy 
Date:   Sun May 3 13:52:16 2009 +0100

Search progress now works with or without javascript enabled, non-js get a 
link to view results when search completes


Great! But...

Why all the while(blah) { Thread.sleep() } 's ? Wouldn't wait/notify be 
better?

What is the purpose of the hidden Find! button?

commit 3a9754b92ea88bfe99019af8810f96d537ff3a79
Author: platy 
Date:   Tue May 5 03:43:39 2009 +0100

Removed use of JS & Iframes entirely

Why? Isn't javascript useful to refresh only part of the page, avoid the 
clunky feeling of reloading frequently? Clearly refresh is better for 
non-javascript... It does work pretty well now though, thanks!

We should include the searched-for term in the title of the page.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/c998525f/attachment.pgp>

[freenet-dev] Bloom filter sharing and opennet behaviour/LRU

It is probably a good idea for reasons of bandwidth, and security to only 
transfer bloom filters to an opennet peer after a certain amount of uptime, 
over some period. But even if we don't, it will take time to transfer 
everything. Will this result in two tiers of opennet peers, and us keeping 
nodes that aren't very good for routing, and not keeping nodes that could be? 
I expect the proportion of requests satisfied locally by bloom filters to be 
relatively low...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/fb188974/attachment.pgp>

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

an attacker, but only if there
are very many blocks to be found. One difficulty is where to start: if we
assume we are tracing after the event, we didn't intercept the original
insert, so it's unlikely our peers were on the chain, and hence unlikely they
have the data - but it is 20x more likely than we were, so it may be a good
starting point nonetheless, especially if we run more than one node. We could
in fact combine both attacks: we run a bunch of nodes, we are waiting for a
file, we pick up on an hourly update that one of our peers was involved in
it, we connect to our peer's peers and try both to intercept more blocks and
to get adjacent nodes' bloom filters (or in the current code, to just probe
for stuff, but that is slow). OTOH if we weren't running at the time, this
introduces more delays, and the longer after the event we start the less
likely we will succeed. In conclusion, it is probably a bit easier for an
attacker...

How to improve security against such attacks? Increasing the uptime required
before we will transfer bloom filters is an obvious measure. Reducing the
average datastore size might help. :| Making it harder for a mobile attacker
to connect to targeted peers is obviously a good thing for beating all these
attacks. Starting to cache further away from the originator might help a lot.
Probabilistic caching at any distance might help a bit. More average
bandwidth will help somewhat. Also we should look at backtracking behaviour -
clearly we do need to treat routing to the second choice node differently
than routing to the first choice node (decide whether to decrement based on a
different criterion), or requests will go to a huge number of hops in some
cases. But this isn't too great from a security pov, as you may get a low
enough HTL to cache on a node that is adjacent to the originator. A much
bigger network would increase the number of hops involved (only log^2
though), and the number of nodes to probe; on the current opennet of a few
thousand nodes, a moderately powerful attacker could conceivably connect to
everyone and get everyone's bloom filters. And of course tunnels would
probably solve the problem, but at a rather massive performance cost - which
may be a price worth paying for inserts, although not generally for requests.

Caching behaviour could be different for inserts. E.g. cache min 2 avg 4 hops
away for requests, min 4 avg 6 hops away for inserts? Or even decrease the
decrement probability for inserts, so e.g. min 4 avg 8 hops away for inserts?
The wider the circle, the more nodes it contains, and the more uncertain the
distance to the centre (due to different decrement probabilities), the more
work an attacker has to do to find the centre. Of course, the fact of not
caching might give away some information, especially for large cache-at hop
numbers compared to the radius of the network...

IMHO we should probably go ahead with the current plans, with a slightly
greater caching distance (lower cache-at-HTL threshold) for inserts than for
requests, and with some sort of uptime threshold for transferring bloom
filters. And implement tunnels later on, before 0.8 if we have time
(depending on how long it takes to build bloom filters and debug db4o), after
it if we don't.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL:
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/93d40bf1/attachment.pgp>

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing


On May 12, 2009, at 11:57 AM, Robert Hailey wrote:

>
> On May 12, 2009, at 11:45 AM, Florent Daigniere wrote:
>
>> Huh.
>>
>> What are you suggesting here? That we re-invent the current protocol?
>
> Sort of... but a request which does not route; one which only checks
> the datastore for the presence of a key. I believe that this idea was
> rejected earlier since the request "does not propagate the requested
> data," but appears to be much better security-wise than bloom filter
> sharing. And we could convert a fraction of these probes into full
> requests if we wanted to.

In case it is not clear, let me correct my sloppy semantics.
"A request which only checks a peer's bloom filter for the presence of  
a key"

>> The whole point of sharing bloom filters is to get rid of the
>> network/node latency. We already have local bloom filters to avoid
>> unnecessary datastore lookups.
>>
>> NextGen$
>
> If the bloom filters are recovering already-failed requests, then
> surely latency is not the issue being addressed.
>
> I thought that the point of having bloom filters was to increase the
> effectiveness of fetches (a more reliable fallback).

"point of having bloom filter sharing was to..."

> --
> Robert Hailey

-- next part --
An HTML attachment was scrubbed...
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/8466e5e3/attachment.html>

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

On May 12, 2009, at 11:45 AM, Florent Daigniere wrote:

> Robert Hailey wrote:
>>
>> On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:
>>
>>> ...
>>> So the question is, how practical is it for a mobile attacker to
>>> identify a
>>> block in its neighbours' stores (from the bloom filter), get
>>> connected to the
>>> neighbour's neighbours, download their bloom filters, and trace an
>>> insert or
>>> request back to source one hop at a time?
>>
>> At first thought, I would imagine if bloom filter sharing increases
>> the fetch effectiveness by x20, so to it would make such an attack 20
>> times easier. A powerful attacker might track ~everyone's~ bloom
>> filter
>>
>> What would the security implications be if we were to have the node
>> with the store check the bloom filter rather than sharing it (~one- 
>> hop-
>> probe~)? Would that buy us anything, being able to mete out bloom
>> filter checks? Then such an attacker could not check to see if we
>> contained ~80% of a split file because he would have to ask us to
>> check our bloom filter for so many keys. It would also be an order of
>> magnitude easier to implement than "sharing".
>>
>
> Huh.
>
> What are you suggesting here? That we re-invent the current protocol?

Sort of... but a request which does not route; one which only checks  
the datastore for the presence of a key. I believe that this idea was  
rejected earlier since the request "does not propagate the requested  
data," but appears to be much better security-wise than bloom filter  
sharing. And we could convert a fraction of these probes into full  
requests if we wanted to.

> The whole point of sharing bloom filters is to get rid of the
> network/node latency. We already have local bloom filters to avoid
> unnecessary datastore lookups.
>
> NextGen$

If the bloom filters are recovering already-failed requests, then  
surely latency is not the issue being addressed.

I thought that the point of having bloom filters was to increase the  
effectiveness of fetches (a more reliable fallback).

--
Robert Hailey

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:

> ...
> So the question is, how practical is it for a mobile attacker to  
> identify a
> block in its neighbours' stores (from the bloom filter), get  
> connected to the
> neighbour's neighbours, download their bloom filters, and trace an  
> insert or
> request back to source one hop at a time?

At first thought, I would imagine if bloom filter sharing increases  
the fetch effectiveness by x20, so to it would make such an attack 20  
times easier. A powerful attacker might track ~everyone's~ bloom  
filter

What would the security implications be if we were to have the node  
with the store check the bloom filter rather than sharing it (~one-hop- 
probe~)? Would that buy us anything, being able to mete out bloom  
filter checks? Then such an attacker could not check to see if we  
contained ~80% of a split file because he would have to ask us to  
check our bloom filter for so many keys. It would also be an order of  
magnitude easier to implement than "sharing".

--
Robert Hailey

[freenet-dev] Infocalypse feedback (Mercurial over Freenet)

2009-05-12 Thread Arne Babenhauserheide

On Monday, 11. May 2009 22:24:48 Matthew Toseland wrote:
> > The mirror of freenet staging worked, too.
> >
> > The initial pull took about an hour, and it succeeded, though I got the
> > log output "GetFailed".
> > Most recent revision was 13919
> > user:Daniel Cheng (???) 
> > date:Mon May 11 10:48:31 2009 +0800
> > summary: Remove unused variable (leftovers of r27142)
>
> Woah, nice, dogfood at last! :)

Infocalypse is being available via infocalypse, too :) 

And subsequent pulls will be much faster (only the bundled (->compressed) 
diffs to pull :) ). 

Best wishes, 
Arne

--- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- 
   - singing a part of the history of free software -
  http://infinite-hands.draketo.de
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/84265339/attachment.pgp>

[freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

2009-05-12 Thread gh...@hushmail.com

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Mon, 11 May 2009 17:07:14 -0600 Matthew Toseland
 wrote:
>We have removed the memory limit option in git ...

Ignoring the memory issue/non-issue, here is my install testing
today:

Installed version of Java:
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)

1. Freenet installer as Administrator
2. Asks for Java 1.5 or greater, installing.
3. Taking default install options
4. Only sees IE as I use  portable FF.
5. Switch to FF after configuration.
6. Did not see is ask for a node name.(probably due to using wizard
vs manual install)
7. Install took the BW changes, but memory adjustments ignored.(Was
suggested possible access rights issue to wrapper.conf)
8. Up and running:

Peer statistics

* Connected: 3
* Busy: 2
* Never connected: 5
* Seed nodes: 4
Took less than 2 minutes to bring up one of the indexes (The
Ultimate Freenet Index).

Error in wrapper: (known non-issue I believe)
STATUS | wrapper  | 2009/05/12   | Freenet background service
installed.
STATUS | wrapper  | 2009/05/12   | --> Wrapper Started as Service
STATUS | wrapper  | 2009/05/12   | Java Service Wrapper Community
Edition 3.3.3
STATUS | wrapper  | 2009/05/12   |   Copyright (C) 1999-2009 Tanuki
Software, Ltd.  All Rights Reserved.
STATUS | wrapper  | 2009/05/12   |
http://wrapper.tanukisoftware.org
STATUS | wrapper  | 2009/05/12   |
STATUS | wrapper  | 2009/05/12   | Launching a JVM...
INFO   | jvm 1| 2009/05/12   | WrapperManager: Initializing...
INFO   | jvm 1| 2009/05/12   | WrapperManager: WARNING - The
Wrapper jar file currently in use is version "3.3.1"
INFO   | jvm 1| 2009/05/12   | WrapperManager:   while
the version of the Wrapper which launched this JVM is
INFO   | jvm 1| 2009/05/12   | WrapperManager:
"3.3.3".
INFO   | jvm 1| 2009/05/12   | WrapperManager:   The
Wrapper may appear to work correctly but some features may
INFO   | jvm 1| 2009/05/12   | WrapperManager:   not
function correctly.  This configuration has not been tested
INFO   | jvm 1| 2009/05/12   | WrapperManager:   and is
not supported.


9.Install was quick easy and painless. Shut node down via browser,
then started it via the destop shortcut. All is well.
Input Rate: 101 KiB/sec (of 256 KiB)
Output Rate: 6.39 KiB/sec (of 64.0 KiB)
Total Input: 1.56 MiB (16.7 KiB/sec)
Total Output: 739 KiB (7.70 KiB/sec)
Payload Output: 383 KiB (3.98 KiB/sec)(51%)

10. Uninstalled, shortcut still on destop. Nope, F5(refresh) and
it's gone.

2nd Install, with java pre-installed (32/64 bit)

1. Freenet installer as Administrator
2. Java detected, taking default install options
3. Cut/paste url into FF to continue at this point.
4. Receive warning about using same brower for Internet/Freenet.
5. Same issues as 6/7 above.
6. Up and running. One small oddity, I accessed the same index page
as above, but this time I had to go to it three times to get the
latest, most current version. Will this cause confusion with new
users that they sometimes can get old content from the main page?
7. Not installer related, but upon uninstalling w/ option to do
survey, the survey errored out with: "Something bad happened. Don't
worry, though. The Spreadsheets Team has been notified and we'll
get right on it." This was from IE as it is still set as the
default browser. (but not for long)
-BEGIN PGP SIGNATURE-
Charset: UTF8
Note: This signature can be verified at https://www.hushtools.com/verify
Version: Hush 3.0

wkYEARECAAYFAkoJkysACgkQDAg0OvA3V4CWmQCgu1Gk4lu6RjglBVRobtuIVtBxZOcA
n09Kc0eWzu7eVxsWk7/mehdrHura
=HK/U
-END PGP SIGNATURE-

--
Put your future on the fast track with a pharmacy career. Click now!
 
http://tagline.hushmail.com/fc/BLSrjkqZ0UM9sRv6ore3dQnafUlaqZLbMCvOfOvJdzlKKwHZwUzqqwb5Pmo/

[freenet-dev] LoggerHook.registerClass in non-plugin code: good idea or not? 006d85dbcd31ffce94279667f5e9ab62d054c9cf

On Tue, May 12, 2009 at 7:10 AM, Matthew Toseland
 wrote:
> On Wednesday 06 May 2009 01:39:54 Daniel Cheng wrote:
>> On Wed, May 6, 2009 at 6:21 AM, Matthew Toseland
>>  wrote:
>> > sdiz commit 006d85dbcd31ffce94279667f5e9ab62d054c9cf
>> >
>> > diff --git a/src/freenet/client/ArchiveHandlerImpl.java
>> > b/src/freenet/client/ArchiveHandlerImpl.java
>> > index 56a9b81..e8046ca 100644
>> > --- a/src/freenet/client/ArchiveHandlerImpl.java
>> > +++ b/src/freenet/client/ArchiveHandlerImpl.java
>> > @@ -24,13 +24,7 @@ class ArchiveHandlerImpl implements ArchiveHandler {
>> > ? ? ? ?private static volatile boolean logMINOR;
>> >
>> > ? ? ? ?static {
>> > - ? ? ? ? ? ? ? Logger.registerLogThresholdCallback(new
> LogThresholdCallback()
>> > {
>> > -
>> > - ? ? ? ? ? ? ? ? ? ? ? @Override
>> > - ? ? ? ? ? ? ? ? ? ? ? public void shouldUpdate() {
>> > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? logMINOR = Logger.shouldLog(Logger.MINOR,
>> > this);
>> > - ? ? ? ? ? ? ? ? ? ? ? }
>> > - ? ? ? ? ? ? ? });
>> > + ? ? ? ? ? ? ? Logger.registerClass(ArchiveHandlerImpl.class);
>> > ? ? ? ?}
>> >
>> > ? ? ? ?private final FreenetURI key;
>> >
>> > Etc.
>> >
>> > Logger.registerClass uses JNI and weak references, it is a great thing. It
>>
>> That's not JNI, ... JNI is native C code.
>> This is Reflection (
>> http://java.sun.com/docs/books/tutorial/reflect/index.html )
>
> Yes, I meant reflection. :)
>>
>> > could simplify much code as above, but shouldUpdate() is slow because of
> JNI.
>> > Given we very rarely change logging settings, is that a problem? How long
>> > would it take to run the JNI hooks on every class? If it's less than 100ms
> or
>> > so, we should use this everywhere.
>
> Do you have an opinion on this part?

The reflection part (minor the "shouldLog()" function) take <2ms on my machine
for each class.

[freenet-dev] What to do on SSK collisions: f86448d51c2e3248e1dfec513eefde50902aac30

On Tuesday 12 May 2009 00:45:54 Matthew Toseland wrote:
> commit f86448d51c2e3248e1dfec513eefde50902aac30
> Author: Daniel Cheng (???) 
> Date:   Fri May 8 21:04:28 2009 +0800
> 
> FreenetStore: Simplify code, remove "overwrite" parameter
> 
> This parameter is always "false".
> (Except when doing BDB->SaltedHash migration, which does not have to 
> overwrite)
> 
> 
> The reason I introduced the overwrite parameter was that when we send an SSK 
> insert and get a DataFound with different data, we should not only propagate 
> the data downstream, but also replace our local copy of it. However 
> apparently I never implemented this. Is it a good idea?
> 
Looks like we do store the collided data for a local insert 
(NodeClientCore.java:XXX we have collided), but not for a remote one. Except 
that a bug prevents the former from working. So we need to fix getBlock(). 
Ok...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/6c3b0962/attachment.pgp>

[freenet-dev] What to do on SSK collisions: f86448d51c2e3248e1dfec513eefde50902aac30

commit f86448d51c2e3248e1dfec513eefde50902aac30
Author: Daniel Cheng (???) 
Date:   Fri May 8 21:04:28 2009 +0800

FreenetStore: Simplify code, remove "overwrite" parameter

This parameter is always "false".
(Except when doing BDB->SaltedHash migration, which does not have to 
overwrite)


The reason I introduced the overwrite parameter was that when we send an SSK 
insert and get a DataFound with different data, we should not only propagate 
the data downstream, but also replace our local copy of it. However 
apparently I never implemented this. Is it a good idea?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/bf1b45d3/attachment.pgp>

[freenet-dev] LoggerHook.registerClass in non-plugin code: good idea or not? 006d85dbcd31ffce94279667f5e9ab62d054c9cf

On Wednesday 06 May 2009 01:39:54 Daniel Cheng wrote:
> On Wed, May 6, 2009 at 6:21 AM, Matthew Toseland
>  wrote:
> > sdiz commit 006d85dbcd31ffce94279667f5e9ab62d054c9cf
> >
> > diff --git a/src/freenet/client/ArchiveHandlerImpl.java
> > b/src/freenet/client/ArchiveHandlerImpl.java
> > index 56a9b81..e8046ca 100644
> > --- a/src/freenet/client/ArchiveHandlerImpl.java
> > +++ b/src/freenet/client/ArchiveHandlerImpl.java
> > @@ -24,13 +24,7 @@ class ArchiveHandlerImpl implements ArchiveHandler {
> > ? ? ? ?private static volatile boolean logMINOR;
> >
> > ? ? ? ?static {
> > - ? ? ? ? ? ? ? Logger.registerLogThresholdCallback(new 
LogThresholdCallback()
> > {
> > -
> > - ? ? ? ? ? ? ? ? ? ? ? @Override
> > - ? ? ? ? ? ? ? ? ? ? ? public void shouldUpdate() {
> > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? logMINOR = Logger.shouldLog(Logger.MINOR,
> > this);
> > - ? ? ? ? ? ? ? ? ? ? ? }
> > - ? ? ? ? ? ? ? });
> > + ? ? ? ? ? ? ? Logger.registerClass(ArchiveHandlerImpl.class);
> > ? ? ? ?}
> >
> > ? ? ? ?private final FreenetURI key;
> >
> > Etc.
> >
> > Logger.registerClass uses JNI and weak references, it is a great thing. It
> 
> That's not JNI, ... JNI is native C code.
> This is Reflection (
> http://java.sun.com/docs/books/tutorial/reflect/index.html )

Yes, I meant reflection. :)
> 
> > could simplify much code as above, but shouldUpdate() is slow because of 
JNI.
> > Given we very rarely change logging settings, is that a problem? How long
> > would it take to run the JNI hooks on every class? If it's less than 100ms 
or
> > so, we should use this everywhere.

Do you have an opinion on this part?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/118e19df/attachment.pgp>

[freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

On Friday 08 May 2009 19:15:46 ghoul at hushmail.com wrote:
> 
> On Fri, 08 May 2009 05:51:58 -0600 Zero3 
> wrote:
> >Hey
> >
> >You seem to have tested the *old* Windows installer, which
> >unfortunately
> >doesn't work on Vista. If you can, please test the new one from
> >the link
> >from toad's mail :).
> >
> >- Zero3
> 
> 
> The good:
> Under Vista64 the installer does produce a functioning node.

Great!
> 
> The bad:
> Wrapper.conf seems to not get an update from wizard mode. Maximum
> memory was still default of 192 and not the 512 I had set. If I do
> a new install and configure it manually rather than using the
> wizard, after a node reboot I can see the changes. A reboot after
> the wizard install does not.

Permissions problem, surely?
> 
> The unfortunate:
> The installer does not see existing 64 bit java installs under the
> Vista OS. The option to install java only installs the 32 bit java.
> This works and allows the install to complete but now you have both
> 32/64 bit jre's.

:|

Better than nothing.
> 
> The messy workaround:
> After the node is installed and working, shut it down.
> Uninstall the 32 bit java.
> Start-run-cmd, then cd to %windir%\syswow64.
> There create three symbolic links to the 64 bit java in
> %windir%\system32.
> mklink java.exe ..\system32\java.exe
> mklink javaw.exe ..\system32\javaw.exe
> mklinik javaws.exe ..\system32\javaws.exe
> Restart the node.
> (I believe these links will break if java is ever updated and need
> to be recreated manually at that point in time.)

Nice, a desktop version of Windows with support for symlinks.
> 
> 
> Thoughts:
> The installer cannot be spoofed by the symbolic links alone, only
> after a completed install is done can the freenet software be ran
> under a 64 bit java.
> 
> Could the installer be revised to detect a 64 bit OS? Also
> adjusting for 64 bit java detection and it's possible download
> accordingly?
> 
> The options to make changes to the wrapper don't happen when using
> the wizard. Can this be fixed or should it be removed and then
> after the node is up remind the user to adjust memory usage if
> he/she wants and then to reboot the node.

We have removed the memory limit option in git ...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/90cc5861/attachment.pgp>

[freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

On Sunday 10 May 2009 00:54:48 Luke771 wrote:
> Zero3 wrote:
> > ghoul at hushmail.com skrev:
> >   
> >> -BEGIN PGP SIGNED MESSAGE-
> >> Hash: SHA1
> >>
> >>
> >>
> >> On Fri, 08 May 2009 05:51:58 -0600 Zero3 
> >> wrote:
> >> 
> >>> Hey
> >>>
> >>> You seem to have tested the *old* Windows installer, which
> >>> unfortunately
> >>> doesn't work on Vista. If you can, please test the new one from
> >>> the link
> >>>   
> >> >from toad's mail :).
> >> 
> >>> - Zero3
> >>>   
> >> The good:
> >> Under Vista64 the installer does produce a functioning node.
> >> 
> >
> > Great! What about the problems you mentioned in your previous mail? (I 
> > just replied to those in another message)
> >
> >   
> >> The bad:
> >> Wrapper.conf seems to not get an update from wizard mode. Maximum
> >> memory was still default of 192 and not the 512 I had set. If I do
> >> a new install and configure it manually rather than using the
> >> wizard, after a node reboot I can see the changes. A reboot after
> >> the wizard install does not.
> >> 
> >
> > We already have a couple of bugs reported in that area. You probably hit 
> > one of them:
> > https://bugs.freenetproject.org/view.php?id=2500
> > https://bugs.freenetproject.org/view.php?id=2573
> >
> >   
> >> The unfortunate:
> >> The installer does not see existing 64 bit java installs under the
> >> Vista OS. The option to install java only installs the 32 bit java.
> >> This works and allows the install to complete but now you have both
> >> 32/64 bit jre's.
> >> 
> >
> > OK - see previous mail.
> >
> >
> >   
> >> The messy workaround:
> >> After the node is installed and working, shut it down.
> >> Uninstall the 32 bit java.
> >> Start-run-cmd, then cd to %windir%\syswow64.
> >> There create three symbolic links to the 64 bit java in
> >> %windir%\system32.
> >> mklink java.exe ..\system32\java.exe
> >> mklink javaw.exe ..\system32\javaw.exe
> >> mklinik javaws.exe ..\system32\javaws.exe
> >> Restart the node.
> >> (I believe these links will break if java is ever updated and need
> >> to be recreated manually at that point in time.)
> >>
> >>
> >> Thoughts:
> >> The installer cannot be spoofed by the symbolic links alone, only
> >> after a completed install is done can the freenet software be ran
> >> under a 64 bit java.
> >>
> >> Could the installer be revised to detect a 64 bit OS? Also
> >> adjusting for 64 bit java detection and it's possible download
> >> accordingly?
> >>
> >> The options to make changes to the wrapper don't happen when using
> >> the wizard. Can this be fixed or should it be removed and then
> >> after the node is up remind the user to adjust memory usage if
> >> he/she wants and then to reboot the node.
> >> 
> >
> > As previously mentioned: Does anyone know how to handle this 32-bit vs. 
> > 64-bit stuff? What to do? (I have no idea, and nothing to test 64-bit 
> > stuff on).
> >   
> OK, I _know_ I'm gonna say something stupid but I have to, so here it goes:
> 
> Joe Cynical suggests: the average user probably wouldnt be bothered by 
> having both a 32-bit and a 64-bit JRE's, especially if told that he's 
> supposed to.
> Make "simple install" just install JRE automatically and "expert mode" 
> let the user manually give the path to his JRE.

IMHO it should work in any case. But it does, so there is no great urgency to 
fix it.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/a9f8bb7b/attachment.pgp>

[freenet-dev] Windows installer category on the bug tracker

On Friday 08 May 2009 19:50:45 Zero3 wrote:
> Hey
> 
> Now that bugs regarding my installer are being submitted to the bug 
> tracker, it would be really nice with a "Windows Installer" category 
> with me being assigned/monitored/something to all bugs by default, so I 
> get notified when something is posted - especially in these months I'm 
> in the military and don't have enough time to keep up-to-date with every 
> single bug/mail/commit/etc. :).

Done. "wininstaller", will auto-assign bugs to you.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/ee42d5ee/attachment.pgp>

[freenet-dev] "file queue"

Who created this bugtracker category? What is its purpose, how is it different 
from fred-client-layer?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20090512/66b61416/attachment.pgp>

[freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

On Friday 08 May 2009 15:12:33 ghoul at hushmail.com wrote:
> Both installers were from toad's links yesterday. One earlier in
> the day and the second around four hours later (with a slightly
> different file size). Here is the MD5 generated on both files.
> a8f88c158cc3927cdc786b67150b86f9 and
> 4fb8e9ff969d7c2b0575aa370874cf89.
> 
> I'm more than happy to give it another go or two or three. Once
> I've had a bit more coffee...

New version, should include update.cmd:
8854028 2009-05-11 23:54 FreenetInstaller-1209.exe
md5: 23e1b4ef70b1bd35ba2fc32fbc54637f
sha256: 00461dd83d9b0370d9f59fabe00168966ef5e6c5ad2d6b588a4d5f3036b304aa
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL:

[freenet-dev] New Windows update script, please commit

On Friday 08 May 2009 19:06:18 Zero3 wrote:
> Matthew Toseland skrev:
> > On Friday 08 May 2009 12:40:32 Zero3 wrote:
> >> Matthew Toseland skrev:
> >>> On Tuesday 20 January 2009 03:18:07 Juiceman wrote:
> > On Fri, Jan 16, 2009 at 5:46 AM, Zero3  wrote:
> >> Juiceman skrev:
> >>> Here is a patch for a new version of the Windows update script.
> >>> Please commit since that svn directory is restricted.  I've tested 
it
> >>> extensively on my XP machine.
> >>> This should handle the new installer start/stop.exe while still 
being
> >>> compatible with current installs.  It will need to be made to handle
> >>> the new node service ID the new installer creates.
> >>> It also still needs some Vista functionality.
>  Here is a further revision of the update script.  Mostly error
>  handling, beautification.  Please commit.
> >>> I have applied this for the new wininstaller. Is it 100% Vista 
compatible? 
> > In 
> >>> particular, will it RunAs itself in order to self-escalate on Vista?
> >> Note that the update.cmd script doesn't need to (and wouldn't even be 
> >> able to) escalate itself. It should have write access to the files 
> >> already (inherited from the user running the script), and the start/stop 
> >> scripts handle escalation themselves.
> > 
> > So it should just work? Unless the user runs it from a user other than the 
one 
> > they installed as? I thought we created a user for Freenet, are the files 
> > writable by both the installing user and the freenet user?
> 
> I haven't had a chance to test it myself then. Juiceman knows best what 
> works and what doesn't (yet). (We discussed what things that needed to 
> change to support both types of installations a while ago.)
> 
> You need admin privs to install Freenet. As any admin by default will 
> have readwrite access to program files, any admin (including the 
> installing one) should be able to use the update.cmd script. Normal 
> users will not have access to do that (which they shouldn't...) The 
> custom freenet user also has full access to the files to allow Freenet 
> to update itself (but the update.cmd script is run as the executing user!).
> 
> Btw., did you remember to add the update.cmd to 
> src_freenetinstaller/FreenetInstaller_Include_Files.inc? It isn't 
> included in the .exe you linked to.

I have now. :| New prototype wininstaller released.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL:

Re: [freenet-dev] What to do on SSK collisions: f86448d51c2e3248e1dfec513eefde50902aac30

On Tue, May 12, 2009 at 7:49 AM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Tuesday 12 May 2009 00:45:54 Matthew Toseland wrote:
 commit f86448d51c2e3248e1dfec513eefde50902aac30
 Author: Daniel Cheng (鄭郁邦) j16s...@freenetproject.org
 Date:   Fri May 8 21:04:28 2009 +0800

 FreenetStore: Simplify code, remove overwrite parameter

 This parameter is always false.
 (Except when doing BDB-SaltedHash migration, which does not have to
 overwrite)


 The reason I introduced the overwrite parameter was that when we send an SSK
 insert and get a DataFound with different data, we should not only propagate
 the data downstream, but also replace our local copy of it. However
 apparently I never implemented this. Is it a good idea?

 Looks like we do store the collided data for a local insert
 (NodeClientCore.java:XXX we have collided), but not for a remote one. Except
 that a bug prevents the former from working. So we need to fix getBlock().
 Ok...


Let's see if these two commit make sense: (this is on my fork, not
committed to the main staging yet)

http://github.com/j16sdiz/fred/commit/8e2ef42c286450813dbfa575bcd3f54dc8cb4c83
http://github.com/j16sdiz/fred/commit/7e6040ce3359486557bdd832c526e473a4f95577

Regards,
Daniel
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

If an average node has a store size of 20GB (10GB cache) and adds 0.24 keys 
per second, it will add a full cache worth of keys in 713 hours. Salted hash 
means that it does not wipe the store in that period (not being LRU); a 
salted hash store would normally have a half-life for un-wiped keys, but 
quadratic probing makes this a bit easier.

In any case, it takes a *long time*: 29 days. We can maybe halve the figure if 
we assume almost all new keys are CHKs. But that's still around 14 days. And 
it's more on a larger store.

So the question is, how practical is it for a mobile attacker to identify a 
block in its neighbours' stores (from the bloom filter), get connected to the 
neighbour's neighbours, download their bloom filters, and trace an insert or 
request back to source one hop at a time?

We will likely have requirements for uptime before we start transferring the 
bloom filter to a node. Assuming the attacker is not already connected, this 
could add a significant delay: we need to transfer the filter, limited by the 
node connected to's upstream and what it is prepared to use of it, and it 
shouldn't offer the bloom filter until we have had some amount of uptime, 
maybe 1 week? This adds a delay of approx 1 week per hop. In some cases we 
will have so few hops that this won't solve the problem; in other cases a 
node will have a small cache (or a huge one) so the trail for one key will go 
cold, although the attacker could try connecting to all its peers 
speculatively, or he could look for the other keys.

And assuming we don't cache at max HTL, it will only get you so far. But if 
you did follow the right breadcrumb trail, it should be possible to look for 
the other keys in nearby nodes, walk around the circle surrounding the 
originator, and thus maybe figure out which node was the originator, or at 
least narrow it down to a small group. With the current plan, we don't store 
a block until we are at least 2 hops away from the originator, with an 
average of 4 hops due to the probabilistic decrement of the HTL from maximum 
(with probability 50%). Whether to decrement is determined once on each node, 
although it is different with backtracking, so it is possible that it might 
be cached on a node immediately adjacent if there was lots of backtracking. 
If there is no backtracking, half of the originator's peers will have dropped 
the HTL, resulting in their peers caching the data; so it is surely possible 
to trace it, but it will probably require probing a largish number of nodes, 
probably hundreds, some dependant on the results of other nodes and some not; 
bandwidth is important to determine capacity here, but by this point the 
attacker should have a whole bunch of keys on the node. If he traced right 
back to the second hop (due to backtracking), he just needs the blooms for 
the nodes connected to the target. Since he doesn't know which is the target, 
that means a 2 hop radius - but there is likely to be overlap, so it may not 
be that many nodes. If he traced back to the third hop (the minimum without 
backtracking), he has a similar problem to determine the second hop, since we 
decremented on the second and on the third hops. Once he has found the second 
hop he can find other second hops and hence the common first hop by a similar 
means, since 50% of the second hops decremented. And to identify the 
originator, he just needs to find lots of first hops. The more connections he 
can have open at once, the easier this is going to be, this is limited by his 
bandwidth; in terms of timing, it is probably possible to avoid this taking 
the equivalent of many more hops, given reasonable bandwidth (it's a fair bet 
that the attacker has way more bandwidth than the average node e.g. because 
he hired a colo).

Is this worse than the current situation? Assume an attacker is trying to 
trace the originator of a large file, after the fact i.e. they didn't 
intercept the insert. They can ask each peer for blocks from the file. Timing 
will tell them which blocks were already cached locally. Some will answer 
even though they didn't cache it locally; this complicates matters by 
propagating those blocks a few hops. So connect to the nodes which had cached 
blocks locally already, and request those blocks from their peers, and maybe 
a few blocks we haven't tried yet. So the principle is very similar. Once we 
get close to the target, things are a bit easier than with the current plans, 
because all nodes cached the data, rather than having to guess at the second 
hops which didn't, and because the originator cached ALL of the data. But 
further out, Bloom filters do have some advantages to an attacker: Once we 
have fetched the bloom filters, we can identify all of the blocks that a peer 
had, without having to wait for load-limited probe requests, but on the other 
hand we have to wait for the transfer to finish and/or the uptime threshold. 
Transferring bloom filters in pieces may help

[freenet-dev] Bloom filter sharing and opennet behaviour/LRU

It is probably a good idea for reasons of bandwidth, and security to only 
transfer bloom filters to an opennet peer after a certain amount of uptime, 
over some period. But even if we don't, it will take time to transfer 
everything. Will this result in two tiers of opennet peers, and us keeping 
nodes that aren't very good for routing, and not keeping nodes that could be? 
I expect the proportion of requests satisfied locally by bloom filters to be 
relatively low...


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] XMLLibrarian changes

commit c6f84c96d60a1e53126824e8c2fa74c1f1d4b658
Author: platy mpb...@gmail.com
Date:   Sun May 3 13:52:16 2009 +0100

Search progress now works with or without javascript enabled, non-js get a 
link to view results when search completes


Great! But...

Why all the while(blah) { Thread.sleep() } 's ? Wouldn't wait/notify be 
better?

What is the purpose of the hidden Find! button?

commit 3a9754b92ea88bfe99019af8810f96d537ff3a79
Author: platy mpb...@gmail.com
Date:   Tue May 5 03:43:39 2009 +0100

Removed use of JS  Iframes entirely

Why? Isn't javascript useful to refresh only part of the page, avoid the 
clunky feeling of reloading frequently? Clearly refresh is better for 
non-javascript... It does work pretty well now though, thanks!

We should include the searched-for term in the title of the page.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] XMLLibrarian changes

On Tuesday 12 May 2009 14:16:54 Matthew Toseland wrote:
 commit c6f84c96d60a1e53126824e8c2fa74c1f1d4b658
 Author: platy mpb...@gmail.com
 Date:   Sun May 3 13:52:16 2009 +0100
 
 Search progress now works with or without javascript enabled, non-js get 
a 
 link to view results when search completes
 
 
 Great! But...
 
 Why all the while(blah) { Thread.sleep() } 's ? Wouldn't wait/notify be 
 better?
 
 What is the purpose of the hidden Find! button?
 
 commit 3a9754b92ea88bfe99019af8810f96d537ff3a79
 Author: platy mpb...@gmail.com
 Date:   Tue May 5 03:43:39 2009 +0100
 
 Removed use of JS  Iframes entirely
 
 Why? Isn't javascript useful to refresh only part of the page, avoid the 
 clunky feeling of reloading frequently? Clearly refresh is better for 
 non-javascript... It does work pretty well now though, thanks!
 
 We should include the searched-for term in the title of the page.

I have made some changes:
- Refactored handleInner() a bit.
- Fixed a bug causing infinite looping continually fetching and re-fetching 
the same index (for me it happens when i search for antarctica).
- Fixed the Searching for blah\nnull bug.

The plugin will be deployed soon.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

2009-05-12 Thread ghoul

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Mon, 11 May 2009 17:07:14 -0600 Matthew Toseland
t...@amphibian.dyndns.org wrote:
We have removed the memory limit option in git ...

Ignoring the memory issue/non-issue, here is my install testing
today:

Installed version of Java:
java version 1.6.0_13
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)

1. Freenet installer as Administrator
2. Asks for Java 1.5 or greater, installing.
3. Taking default install options
4. Only sees IE as I use  portable FF.
5. Switch to FF after configuration.
6. Did not see is ask for a node name.(probably due to using wizard
vs manual install)
7. Install took the BW changes, but memory adjustments ignored.(Was
suggested possible access rights issue to wrapper.conf)
8. Up and running:

Peer statistics

* Connected: 3
* Busy: 2
* Never connected: 5
* Seed nodes: 4
Took less than 2 minutes to bring up one of the indexes (The
Ultimate Freenet Index).

Error in wrapper: (known non-issue I believe)
STATUS | wrapper  | 2009/05/12   | Freenet background service
installed.
STATUS | wrapper  | 2009/05/12   | -- Wrapper Started as Service
STATUS | wrapper  | 2009/05/12   | Java Service Wrapper Community
Edition 3.3.3
STATUS | wrapper  | 2009/05/12   |   Copyright (C) 1999-2009 Tanuki
Software, Ltd.  All Rights Reserved.
STATUS | wrapper  | 2009/05/12   |
http://wrapper.tanukisoftware.org
STATUS | wrapper  | 2009/05/12   |
STATUS | wrapper  | 2009/05/12   | Launching a JVM...
INFO   | jvm 1| 2009/05/12   | WrapperManager: Initializing...
INFO   | jvm 1| 2009/05/12   | WrapperManager: WARNING - The
Wrapper jar file currently in use is version 3.3.1
INFO   | jvm 1| 2009/05/12   | WrapperManager:   while
the version of the Wrapper which launched this JVM is
INFO   | jvm 1| 2009/05/12   | WrapperManager:
3.3.3.
INFO   | jvm 1| 2009/05/12   | WrapperManager:   The
Wrapper may appear to work correctly but some features may
INFO   | jvm 1| 2009/05/12   | WrapperManager:   not
function correctly.  This configuration has not been tested
INFO   | jvm 1| 2009/05/12   | WrapperManager:   and is
not supported.


9.Install was quick easy and painless. Shut node down via browser,
then started it via the destop shortcut. All is well.
Input Rate: 101 KiB/sec (of 256 KiB)
Output Rate: 6.39 KiB/sec (of 64.0 KiB)
Total Input: 1.56 MiB (16.7 KiB/sec)
Total Output: 739 KiB (7.70 KiB/sec)
Payload Output: 383 KiB (3.98 KiB/sec)(51%)

10. Uninstalled, shortcut still on destop. Nope, F5(refresh) and
it's gone.

2nd Install, with java pre-installed (32/64 bit)

1. Freenet installer as Administrator
2. Java detected, taking default install options
3. Cut/paste url into FF to continue at this point.
4. Receive warning about using same brower for Internet/Freenet.
5. Same issues as 6/7 above.
6. Up and running. One small oddity, I accessed the same index page
as above, but this time I had to go to it three times to get the
latest, most current version. Will this cause confusion with new
users that they sometimes can get old content from the main page?
7. Not installer related, but upon uninstalling w/ option to do
survey, the survey errored out with: Something bad happened. Don't
worry, though. The Spreadsheets Team has been notified and we'll
get right on it. This was from IE as it is still set as the
default browser. (but not for long)
-BEGIN PGP SIGNATURE-
Charset: UTF8
Note: This signature can be verified at https://www.hushtools.com/verify
Version: Hush 3.0

wkYEARECAAYFAkoJkysACgkQDAg0OvA3V4CWmQCgu1Gk4lu6RjglBVRobtuIVtBxZOcA
n09Kc0eWzu7eVxsWk7/mehdrHura
=HK/U
-END PGP SIGNATURE-

--
Put your future on the fast track with a pharmacy career. Click now!
 
http://tagline.hushmail.com/fc/BLSrjkqZ0UM9sRv6ore3dQnafUlaqZLbMCvOfOvJdzlKKwHZwUzqqwb5Pmo/

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing



On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:

 ...
 So the question is, how practical is it for a mobile attacker to  
 identify a
 block in its neighbours' stores (from the bloom filter), get  
 connected to the
 neighbour's neighbours, download their bloom filters, and trace an  
 insert or
 request back to source one hop at a time?

At first thought, I would imagine if bloom filter sharing increases  
the fetch effectiveness by x20, so to it would make such an attack 20  
times easier. A powerful attacker might track ~everyone's~ bloom  
filter

What would the security implications be if we were to have the node  
with the store check the bloom filter rather than sharing it (~one-hop- 
probe~)? Would that buy us anything, being able to mete out bloom  
filter checks? Then such an attacker could not check to see if we  
contained ~80% of a split file because he would have to ask us to  
check our bloom filter for so many keys. It would also be an order of  
magnitude easier to implement than sharing.

--
Robert Hailey


___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

2009-05-12 Thread Florent Daigniere

Robert Hailey wrote:
 
 On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:
 
 ...
 So the question is, how practical is it for a mobile attacker to  
 identify a
 block in its neighbours' stores (from the bloom filter), get  
 connected to the
 neighbour's neighbours, download their bloom filters, and trace an  
 insert or
 request back to source one hop at a time?
 
 At first thought, I would imagine if bloom filter sharing increases  
 the fetch effectiveness by x20, so to it would make such an attack 20  
 times easier. A powerful attacker might track ~everyone's~ bloom  
 filter
 
 What would the security implications be if we were to have the node  
 with the store check the bloom filter rather than sharing it (~one-hop- 
 probe~)? Would that buy us anything, being able to mete out bloom  
 filter checks? Then such an attacker could not check to see if we  
 contained ~80% of a split file because he would have to ask us to  
 check our bloom filter for so many keys. It would also be an order of  
 magnitude easier to implement than sharing.
 

Huh.

What are you suggesting here? That we re-invent the current protocol?

The whole point of sharing bloom filters is to get rid of the 
network/node latency. We already have local bloom filters to avoid 
unnecessary datastore lookups.

NextGen$
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing


On May 12, 2009, at 11:45 AM, Florent Daigniere wrote:

 Robert Hailey wrote:

 On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:

 ...
 So the question is, how practical is it for a mobile attacker to
 identify a
 block in its neighbours' stores (from the bloom filter), get
 connected to the
 neighbour's neighbours, download their bloom filters, and trace an
 insert or
 request back to source one hop at a time?

 At first thought, I would imagine if bloom filter sharing increases
 the fetch effectiveness by x20, so to it would make such an attack 20
 times easier. A powerful attacker might track ~everyone's~ bloom
 filter

 What would the security implications be if we were to have the node
 with the store check the bloom filter rather than sharing it (~one- 
 hop-
 probe~)? Would that buy us anything, being able to mete out bloom
 filter checks? Then such an attacker could not check to see if we
 contained ~80% of a split file because he would have to ask us to
 check our bloom filter for so many keys. It would also be an order of
 magnitude easier to implement than sharing.


 Huh.

 What are you suggesting here? That we re-invent the current protocol?

Sort of... but a request which does not route; one which only checks  
the datastore for the presence of a key. I believe that this idea was  
rejected earlier since the request does not propagate the requested  
data, but appears to be much better security-wise than bloom filter  
sharing. And we could convert a fraction of these probes into full  
requests if we wanted to.

 The whole point of sharing bloom filters is to get rid of the
 network/node latency. We already have local bloom filters to avoid
 unnecessary datastore lookups.

 NextGen$

If the bloom filters are recovering already-failed requests, then  
surely latency is not the issue being addressed.

I thought that the point of having bloom filters was to increase the  
effectiveness of fetches (a more reliable fallback).

--
Robert Hailey

___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing



On May 12, 2009, at 11:57 AM, Robert Hailey wrote:



On May 12, 2009, at 11:45 AM, Florent Daigniere wrote:


Huh.

What are you suggesting here? That we re-invent the current protocol?


Sort of... but a request which does not route; one which only checks
the datastore for the presence of a key. I believe that this idea was
rejected earlier since the request does not propagate the requested
data, but appears to be much better security-wise than bloom filter
sharing. And we could convert a fraction of these probes into full
requests if we wanted to.


In case it is not clear, let me correct my sloppy semantics.
A request which only checks a peer's bloom filter for the presence of  
a key



The whole point of sharing bloom filters is to get rid of the
network/node latency. We already have local bloom filters to avoid
unnecessary datastore lookups.

NextGen$


If the bloom filters are recovering already-failed requests, then
surely latency is not the issue being addressed.

I thought that the point of having bloom filters was to increase the
effectiveness of fetches (a more reliable fallback).


point of having bloom filter sharing was to...


--
Robert Hailey


___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

[freenet-dev] Request for proofreading: Announcing donation from Google

I will post this on the website tomorrow if there are no objections. If anyone 
can suggest any improvements, please do so; sometimes what I write isn't 
readable by human beings!

7th May, 2009 - Another big donation!

Google's Open Source team has donated US$18,000 to the Freenet Project to 
support the ongoing development of the Freenet software (thanks again 
Google!).

Their last donation funded the db4o project, which has now been merged into 
Freenet, greatly improving performance for large download queues while 
reducing memory usage, amongst other benefits.

We are currently working on Freenet 0.8, which will be released later this 
year, and will include additional performance improvements, usability work, 
and security improvements, as well as the usual debugging. Features are not 
yet finalized but we expect it to include Freetalk (a new anonymous web 
forums tool), a new Vista-compatible installer for Windows (that part will be 
out in a few days), and hopefully Bloom filter sharing, a new feature 
enabling nodes to know what is in their peers' datastores, greatly improving 
performance, combined with some related security improvements.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Please help us test the new wininstaller (Vista users especially welcome)

2009-05-12 Thread Zero3

gh...@hushmail.com skrev:
 4. Only sees IE as I use  portable FF.

Makes sense. I doubt the portable FF adds any kind of indication that it 
is available? (Registry key or the like, which we could look for in the 
browser search done in the launcher)

 
 2nd Install, with java pre-installed (32/64 bit)
 
 1. Freenet installer as Administrator
 2. Java detected, taking default install options

Did the installer correctly detect your 64-bit Java installation? I 
thought it didn't do that last time?

 6. Up and running. One small oddity, I accessed the same index page
 as above, but this time I had to go to it three times to get the
 latest, most current version. Will this cause confusion with new
 users that they sometimes can get old content from the main page?

Was it a very old version, or the second latest? Your node probably 
found the second latest version first, and transparently replaced it 
with the latest one once it stumbled upon it. This is quite normal. It 
shouldn't happen for long though, as all nodes should update their 
copies as soon as they hear about the new one.

 7. Not installer related, but upon uninstalling w/ option to do
 survey, the survey errored out with: Something bad happened. Don't
 worry, though. The Spreadsheets Team has been notified and we'll
 get right on it. This was from IE as it is still set as the
 default browser. (but not for long)

Sounds like an error from Google (who hosts the survey spreadsheet). 
Should fix itself once they figure out what's wrong.

- Zero3
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Recent progress on Interdex

2009-05-12 Thread Ximin Luo

Matthew Toseland wrote:
 Is it a good idea to use MD5? I guess you're using it the same way that 
 XMLLibrarian does, but it may be more of a problem for your application?

you mean collisions? with md5 the expected rate of collisions is around 1 in
2^64, so that would give an average word length of 4 assuming 2^16 possible
letters, or word length of 8 with 2^8 possible letters... it seems ok, but
maybe sha1 is safer, then.

in any case, the actual plain keyword would be stored with each index so
collisions could be detected.

 From your docs...
 An IndexNode corresponds to an SSK subspace, it contains a filter (e.g. bloom 
 filter) for quickly ruling out an index based on the sought keywords, and a 
 bunch of entries, which can each either be redirects to other indexes, or can 
 be pointers to files.

what's a subspace - the entire first directory of an SSK, or any
subdirectory at any level? (i'm not familiar with the terminology here, sorry)
an IndexTree is the former; an IndexNode is the latter.

yes, entries can be redirects to other indexes, or point to files that contain
the actual index data = {(keyword, freenetURI that matches it, other relevant
information)*}. is this what you meant by files? they *don't* point to
non-index files of content.

 What does it mean to inflate or deflate the index?

inflate = REQ the relevant data from freenet, and use it to build internal data
structure

deflate = INS the internal data structure into freenet. for the SSKSerialiser,
SSK/USKs can't be partially updated without updating the whole subspace, (or so
i thought), which is why token-deflate throws UnsupportedOperationException.

(one way of storing it which would allow token-deflate would be having each
indexnode as a CHK, then you'd only have to INS an updated node and all its
parents up to the root, but i chose not to do this as CHKs have a higher limit
for being turned into a splitfile. was this the right decision?)

X
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Bloom filters and store probing, after the fact splitfile originator tracing

On Tuesday 12 May 2009 17:57:04 Robert Hailey wrote:
 
 On May 12, 2009, at 11:45 AM, Florent Daigniere wrote:
 
  Robert Hailey wrote:
 
  On May 12, 2009, at 6:47 AM, Matthew Toseland wrote:
 
  ...
  So the question is, how practical is it for a mobile attacker to
  identify a
  block in its neighbours' stores (from the bloom filter), get
  connected to the
  neighbour's neighbours, download their bloom filters, and trace an
  insert or
  request back to source one hop at a time?
 
  At first thought, I would imagine if bloom filter sharing increases
  the fetch effectiveness by x20, so to it would make such an attack 20
  times easier. A powerful attacker might track ~everyone's~ bloom
  filter
 
  What would the security implications be if we were to have the node
  with the store check the bloom filter rather than sharing it (~one- 
  hop-
  probe~)? Would that buy us anything, being able to mete out bloom
  filter checks? Then such an attacker could not check to see if we
  contained ~80% of a split file because he would have to ask us to
  check our bloom filter for so many keys. It would also be an order of
  magnitude easier to implement than sharing.
 
 
  Huh.
 
  What are you suggesting here? That we re-invent the current protocol?
 
 Sort of... but a request which does not route; one which only checks  
 the datastore for the presence of a key. I believe that this idea was  
 rejected earlier since the request does not propagate the requested  
 data, but appears to be much better security-wise than bloom filter  
 sharing. And we could convert a fraction of these probes into full  
 requests if we wanted to.
 
  The whole point of sharing bloom filters is to get rid of the
  network/node latency. We already have local bloom filters to avoid
  unnecessary datastore lookups.
 
  NextGen$
 
 If the bloom filters are recovering already-failed requests, then  
 surely latency is not the issue being addressed.
 
 I thought that the point of having bloom filters was to increase the  
 effectiveness of fetches (a more reliable fallback).

So your proposal is that we broadcast all our requests to all our peers, and 
then what, wait for all their responses before routing outwards?


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Request for proofreading: Announcing donation from Google

2009-05-12 Thread Ian Clarke

On Tue, May 12, 2009 at 2:36 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 7th May, 2009 - Another big donation!

How about: Google makes a second donation to Freenet!

Everything else looks good.

Ian.

-- 
Ian Clarke
CEO, Uprizer Labs
Email: i...@uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Recent progress on Interdex

On Tuesday 12 May 2009 21:26:53 Ximin Luo wrote:
 Matthew Toseland wrote:
  Is it a good idea to use MD5? I guess you're using it the same way that 
  XMLLibrarian does, but it may be more of a problem for your application?
 
 you mean collisions? with md5 the expected rate of collisions is around 1 in
 2^64, so that would give an average word length of 4 assuming 2^16 possible
 letters, or word length of 8 with 2^8 possible letters... it seems ok, but
 maybe sha1 is safer, then.

What about deliberate collisions? Could they cause you to use more memory or 
anything like that? md5 is broken...
 
 in any case, the actual plain keyword would be stored with each index so
 collisions could be detected.
 
  From your docs...
  An IndexNode corresponds to an SSK subspace, it contains a filter (e.g. 
bloom 
  filter) for quickly ruling out an index based on the sought keywords, and 
a 
  bunch of entries, which can each either be redirects to other indexes, or 
can 
  be pointers to files.
 
 what's a subspace - the entire first directory of an SSK, 

Yes.

 or any 
 subdirectory at any level? (i'm not familiar with the terminology here, 
sorry)
 an IndexTree is the former; an IndexNode is the latter.
 
 yes, entries can be redirects to other indexes, or point to files that 
contain
 the actual index data = {(keyword, freenetURI that matches it, other 
relevant
 information)*}. is this what you meant by files? they *don't* point to
 non-index files of content.

Hmmm, ok.
 
  What does it mean to inflate or deflate the index?
 
 inflate = REQ the relevant data from freenet, and use it to build internal 
data
 structure
 
 deflate = INS the internal data structure into freenet. for the 
SSKSerialiser,
 SSK/USKs can't be partially updated without updating the whole subspace, (or 
so
 i thought), which is why token-deflate throws UnsupportedOperationException.

Ok.
 
 (one way of storing it which would allow token-deflate would be having each
 indexnode as a CHK, then you'd only have to INS an updated node and all its
 parents up to the root, but i chose not to do this as CHKs have a higher 
limit
 for being turned into a splitfile. was this the right decision?)

Or you could store them as separate SSKs, but I wouldn't recommend it. SSKs 
can have any name after the slash. But inserting it all at once adds 
redundancy etc, it's generally a good idea.


signature.asc
Description: This is a digitally signed message part.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] What to do on SSK collisions: f86448d51c2e3248e1dfec513eefde50902aac30

2009/5/13 Matthew Toseland t...@amphibian.dyndns.org:
 On Tuesday 12 May 2009 09:33:11 you wrote:
 On Tue, May 12, 2009 at 7:49 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Tuesday 12 May 2009 00:45:54 Matthew Toseland wrote:
  commit f86448d51c2e3248e1dfec513eefde50902aac30
  Author: Daniel Cheng (鄭郁邦) j16s...@freenetproject.org
  Date:   Fri May 8 21:04:28 2009 +0800
 
  FreenetStore: Simplify code, remove overwrite parameter
 
  This parameter is always false.
  (Except when doing BDB-SaltedHash migration, which does not have to
  overwrite)
 
 
  The reason I introduced the overwrite parameter was that when we send an
 SSK
  insert and get a DataFound with different data, we should not only
 propagate
  the data downstream, but also replace our local copy of it. However
  apparently I never implemented this. Is it a good idea?
 
  Looks like we do store the collided data for a local insert
  (NodeClientCore.java:XXX we have collided), but not for a remote one.
 Except
  that a bug prevents the former from working. So we need to fix getBlock().
  Ok...
 

 Let's see if these two commit make sense: (this is on my fork, not
 committed to the main staging yet)


 http://github.com/j16sdiz/fred/commit/8e2ef42c286450813dbfa575bcd3f54dc8cb4c83

 http://github.com/j16sdiz/fred/commit/7e6040ce3359486557bdd832c526e473a4f95577

 Regards,
 Daniel

 Does this deal with the case where the request is remote, i.e. came from
 outside via a SSKInsertHandler?


I think the SSKInsertSender code have handle the remote request case:
  
http://github.com/freenet/fred-staging/commit/6a341ed359a9ef6800a9830685c97072e9845912#diff-3

If it do not, I have no idea where should I fix it.
___
Devl mailing list
Devl@freenetproject.org
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl

Re: [freenet-dev] Recent progress on Interdex