Send Devl mailing list submissions to
devl at freenetproject.org
To subscribe or unsubscribe via the World Wide Web, visit
http://www.uprizer.com/mailman/listinfo/devl
or, via email, send a message with subject or body 'help' to
devl-request at freenetproject.org
You can reach the person managing the list at
devl-admin at freenetproject.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Devl digest..."
Today's Topics:
1. Re: Proposal: algorithm for forgetting documents in datastore (Scott G.
Miller)
2. Re: Proposal: algorithm for forgetting documents in datastore (Scott G.
Miller)
3. Protozilla (Adam Langley)
4. Seems quite relevant (Scott G. Miller)
5. Re: Protozilla (Scott G. Miller)
6. Re: Seems quite relevant (Sebastian Spaeth)
7. Re: Proposal: algorithm for forgetting documents in datastore (Theodore
Hong)
8. Re: Proposal: algorithm for forgetting documents in datastore (Theodore
Hong)
9. Re: Protozilla (Adam Langley)
--__--__--
Message: 1
Date: Sun, 28 Jan 2001 12:05:33 -0500
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: "Scott G. Miller" <scgmi...@indiana.edu>
Reply-To: devl at freenetproject.org
--hYooF8G/hrfVAmum
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, Jan 28, 2001 at 02:28:26AM -0500, Tavin Cole wrote:
> On Sat, Jan 27, 2001 at 05:14:55PM -0500, Benjamin Coates wrote:
> > >It seems totally reasonble for a node to reject caching documents grea=
ter
> > >in size than a certain fraction of the total datastore size (1% or less
> > >I should think). That prevents your worst-case example, but saves us =
from
> > >making relative value judgments using file size as the metric..
> >=20
> > 1% is pretty tiny, that's 1MB on the default datastore size, and also l=
eads to=20
> > the bizarre situation where all the insert nodes pass on the request an=
d=20
> > return success, even though none of them store it. Maybe nodes should =
return=20
> > failure instead of passing through an insert they can't cache...
>=20
> OTOH, it's probably not good if a datastore can only hold a handful of ke=
ys
> b/c they are all a considerable fraction of its max size. 1% may not be
> right, tho.
1% is definitely too small. I don't want any assumptions made unless they
include datastores that are on average 100-200mb in size (a size I
consider optimal in a large freenet consisting of 'normal', i.e., not
dedicated nodes).
I think its probably reasonable to store 5mb files in a datastore, as
well. This all comes down to the file splitting issue, where there are a
number of questions;
Max file size
Minimum file size (under which we pad)
Continuous or discrete file sizes (ie only powers of 2, or any size).
Number of parts into which a file can be split but still be retrieved
Redundancy? (I'm with Ian, this is not really a good idea in Freenet)
--hYooF8G/hrfVAmum
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6dFFdr9IW4v3mHtQRAmA7AJ4tv/y7q/b416bkDj+X8rVPeZ2D7QCfQ1CT
QJU7dqh3xrJoPuA9EJvaiX8=
=hiq7
-----END PGP SIGNATURE-----
--hYooF8G/hrfVAmum--
--__--__--
Message: 2
Date: Sun, 28 Jan 2001 12:08:04 -0500
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: "Scott G. Miller" <scgmi...@indiana.edu>
Reply-To: devl at freenetproject.org
--g7w8+K/95kPelPD2
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, Jan 28, 2001 at 04:38:49PM +0000, Theodore Hong wrote:
> Tavin Cole <tavin at mailandnews.com> wrote:
> > It's clear to me now that the only sane mechanism for deciding what fil=
es
> > to keep in the document store is one that consistently preserves the
> > files that receive the greatest number of requests. So let's define P =
as
> > the measure of a file's request history. Simplistically,
> >
> > P =3D total no. of requests for this file / time since file was stored
> >
> > In practice, though, I think we'd want to limit the past time interval =
to
> > what is "recent". Also, the units in the denominator should not be too
> > granular, to keep P from blowing out of proportion for brand new files.
> > Something in the neighborhood of hours or minutes, probably.
>=20
> Limiting the past time interval would require picking an arbitrary cutoff.
> The best way is to apply a decay factor to old requests, something like:
>=20
> P =3D #requests in the past hour
> + (#requests in the previous 2 hrs)/2
> + (#requests in the previous 4 hrs)/4
> + (#requests in the previous 8 hrs)/8
> + ...
Every two hours you halve P. When a request comes in you add a constant.
--g7w8+K/95kPelPD2
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6dFH0r9IW4v3mHtQRApN8AJ9d0fkUJWcC/1R8/8iAs/OUvDs2ZACfe/lY
05LzAtj+xs5aDi8OEuxGRIc=
=cAj/
-----END PGP SIGNATURE-----
--g7w8+K/95kPelPD2--
--__--__--
Message: 3
Date: Sun, 28 Jan 2001 17:47:40 +0000
From: Adam Langley <a...@linuxpower.org>
To: devl at freenetproject.org
Subject: [freenet-devl] Protozilla
Reply-To: devl at freenetproject.org
--0F1p//8PRICkK4MW
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
http://protozilla.mozdev.org/white-paper.html
Interresting bit:
Peer-to-peer (P2P) URLs: Seamless access to P2P protocols through=20
the familiar browser interface should make them much more accessible
to the end-user. Not only will the the user be able to access the
protocols by typing a P2P URL in the URL bar, but the clickable P2P
URLs may also be embedded in e-mail messages and other documents=20
and exchanged just like HTTP or FTP URLs. Protozilla enables=20
clients for P2P protocols to be implemented trivially in Mozilla.
AGL
--=20
Join in the new game that's sweeping the country. It's called "Bureaucracy=
". Everybody stands in a circle. The first person to do anything loses.
--0F1p//8PRICkK4MW
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iEYEARECAAYFAjp0WzsACgkQzaVS3yy2PWCR7gCgpzd0ptByLFGnHjfOcM7oM46Z
uqYAnjJCO1LoJW903Nm53Ct3JCvK06//
=lJsk
-----END PGP SIGNATURE-----
--0F1p//8PRICkK4MW--
--__--__--
Message: 4
Date: Sun, 28 Jan 2001 12:39:44 -0500
To: devl at freenetproject.org
From: "Scott G. Miller" <scgmi...@indiana.edu>
Subject: [freenet-devl] Seems quite relevant
Reply-To: devl at freenetproject.org
--45Z9DzgjV8m4Oswq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
http://protozilla.mozdev.org/white-paper.html
Looks like the last to bullet points aim directly at us.
--45Z9DzgjV8m4Oswq
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6dFlgr9IW4v3mHtQRAl2OAJ4lWyAAC/Q1SXwIRQhajBwOYz4uGACdGpRG
6uAY/S2ZhcxZXxsuIuT7tQI=
=AWY+
-----END PGP SIGNATURE-----
--45Z9DzgjV8m4Oswq--
--__--__--
Message: 5
Date: Sun, 28 Jan 2001 12:58:03 -0500
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Protozilla
From: "Scott G. Miller" <scgmi...@indiana.edu>
Reply-To: devl at freenetproject.org
--T4sUOijqQbZv57TR
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Damn you! FIRST POST!
On Sun, Jan 28, 2001 at 05:47:40PM +0000, Adam Langley wrote:
> http://protozilla.mozdev.org/white-paper.html
>=20
> Interresting bit:
> Peer-to-peer (P2P) URLs: Seamless access to P2P protocols through=20
> the familiar browser interface should make them much more accessible
> to the end-user. Not only will the the user be able to access the
> protocols by typing a P2P URL in the URL bar, but the clickable P2P
> URLs may also be embedded in e-mail messages and other documents=20
> and exchanged just like HTTP or FTP URLs. Protozilla enables=20
> clients for P2P protocols to be implemented trivially in Mozilla.
>=20
> AGL
>=20
> --=20
> Join in the new game that's sweeping the country. It's called "Bureaucra=
cy". Everybody stands in a circle. The first person to do anything loses.
--=20
cog gil sorter myrtle
--T4sUOijqQbZv57TR
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6dF2rr9IW4v3mHtQRAmozAJ0XW6jarQImWKMNTXOq4vVvks7WRACfYrlm
PhujwlDvBErgkmyrMIO1o0A=
=k61W
-----END PGP SIGNATURE-----
--T4sUOijqQbZv57TR--
--__--__--
Message: 6
Date: Sun, 28 Jan 2001 19:16:12 +0100
From: Sebastian Spaeth <sebast...@sspaeth.de>
Organization: University of =?iso-8859-1?Q?Link=F6ping?=
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Seems quite relevant
Reply-To: devl at freenetproject.org
"Scott G. Miller" wrote:
>
> http://protozilla.mozdev.org/white-paper.html
>
> Looks like the last to bullet points aim directly at us.
It's already the official browser plugin for Mozilla (speaking for Win
at least) once we decide to include it...
Sebastian
--__--__--
Message: 7
From: Theodore Hong <t...@doc.ic.ac.uk>
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
To: devl at freenetproject.org
Date: Sun, 28 Jan 2001 18:45:00 +0000 (GMT)
Reply-To: devl at freenetproject.org
Oskar Sandberg <md98-osa at nada.kth.se> wrote:
> On Sun, Jan 28, 2001 at 02:16:21PM +0000, Theodore Hong wrote:
> > "Mark J. Roberts" <mjr at statesmean.com> wrote:
> < >
> > > I tend to favor Ian's model because I'm a proponent of segmentation (even
> > > slightly redundant segmentation), and Ian's model encourages such
> > > segmentation. However, I certainly wouldn't want to see it pushed to
> > > extremes, where data must be inserted as tiny segments to survive. But I
> > > think we should discourage large (>1M) files. Looks like a compromise is
> > > necessary.
> >
> > Historically, this tension between large and small resulted in the Great
> > Compromise of 1787 in the United States. The rights of small states were
> > preserved by equal representation by state in the Senate, while those of
> > large states were preserved by proportional representation by size in the
> > House. Perhaps we should adopt something similar.
>
> Because the whole American democracy thing worked out sooo well....
As a matter of fact, it did (the Great Compromise part, anyway). If you
notice, there are no arguments between large and small American states
anymore. It is a complete non-issue, whereas it once threatened to
undermine the entire concept of "united" states. See "qualified majority
voting" in the EU and tell me that getting states not to argue wasn't a
great triumph...
theo
--__--__--
Message: 8
From: Theodore Hong <t...@doc.ic.ac.uk>
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
To: devl at freenetproject.org
Date: Sun, 28 Jan 2001 18:47:56 +0000 (GMT)
Reply-To: devl at freenetproject.org
Theodore Hong <twh1 at doc.ic.ac.uk> wrote:
> Tavin Cole <tavin at mailandnews.com> wrote:
> > It's clear to me now that the only sane mechanism for deciding what files
> > to keep in the document store is one that consistently preserves the
> > files that receive the greatest number of requests. So let's define P as
> > the measure of a file's request history. Simplistically,
> >
> > P = total no. of requests for this file / time since file was stored
> >
> > In practice, though, I think we'd want to limit the past time interval to
> > what is "recent". Also, the units in the denominator should not be too
> > granular, to keep P from blowing out of proportion for brand new files.
> > Something in the neighborhood of hours or minutes, probably.
>
> Limiting the past time interval would require picking an arbitrary cutoff.
> The best way is to apply a decay factor to old requests, something like:
>
> P = #requests in the past hour
> + (#requests in the previous 2 hrs)/2
> + (#requests in the previous 4 hrs)/4
> + (#requests in the previous 8 hrs)/8
> + ...
obviously I mean "frequency of requests", not #requests...
> (I'm sure there's a nicer way to express this as a continuous function
> using an integral, but I can't think of it at the moment. Anyway, you
> get the idea.)
>
> > When a new file is added to the store, as a result of either a request or
> > an insert, it could be allocated 1 request in the above equation. When
> > the store is full, you basically want to discard the entries with lowest
> > P.
> >
> > This is trivial if all files are the same size. The question is, if a
> > file is going to displace >1 smaller files, do we compare its P to the
> > summed P of the smaller files? Do we compare its P to some function of
> > the P of the smaller files, where f(P1..n) < P1 + .. + Pn ? Or do we
> > ignore the size difference and just look at the highest P of the smaller
> > files?
>
> I think the most straightforward way is simply to compare the one file's P
> to the sum of all the P's it would displace. This maximizes the number of
> successful requests. For example, one 100k file getting 11 requests/day
> should be ranked higher than ten 10k files getting 1 request/day each.
>
> The difficulty is deciding how to assess P for a new file. If it's too
> low, new files may never make it in. If it's too high, we just have the
> current situation (new files always have highest priority).
>
> theo
--__--__--
Message: 9
Date: Sun, 28 Jan 2001 19:08:34 +0000
From: Adam Langley <a...@linuxpower.org>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Protozilla
Reply-To: devl at freenetproject.org
--G4iJoqBmSsgzjUCe
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sun, Jan 28, 2001 at 12:58:03PM -0500, Scott G. Miller wrote:
> Damn you! FIRST POST!
Yea, ok. Your post hadn't come thru when I sent mine. You win ;)
AGL
--=20
Join in the new game that's sweeping the country. It's called "Bureaucracy=
". Everybody stands in a circle. The first person to do anything loses.
--G4iJoqBmSsgzjUCe
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iEYEARECAAYFAjp0bjIACgkQzaVS3yy2PWAaHQCfZDU8g2WzqQk4GNoQvRZz/ysW
aXkAoICaykDujwNpPiVDRgAwSOGUt1AI
=bgr9
-----END PGP SIGNATURE-----
--G4iJoqBmSsgzjUCe--
--__--__--
_______________________________________________
Devl mailing list
Devl at freenetproject.org
http://www.uprizer.com/mailman/listinfo/devl
End of Devl Digest
