Send Devl mailing list submissions to
devl at freenetproject.org
To subscribe or unsubscribe via the World Wide Web, visit
http://www.uprizer.com/mailman/listinfo/devl
or, via email, send a message with subject or body 'help' to
devl-request at freenetproject.org
You can reach the person managing the list at
devl-admin at freenetproject.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Devl digest..."
Today's Topics:
1. Re: Problems! -> 0.3.7 any day now (Mark J. Roberts)
2. Re: Proposal: algorithm for forgetting documents in datastore (toad)
3. Re: Proposal: algorithm for forgetting documents in datastore (Oskar
Sandberg)
4. Re: Proposal: algorithm for forgetting documents in
datastore (Mark J. Roberts)
5. Re: FProxy URLS should use %40, not '@' (Brandon)
6. Re: Proposal: algorithm for forgetting documents in datastore (Oskar
Sandberg)
7. Re: Proposal: algorithm for forgetting documents in
datastore (Mark J. Roberts)
8. libfreenet page (Steven Hazel)
9. Re: Proposal: algorithm for forgetting documents in datastore (Scott G.
Miller)
10. Re: Proposal: algorithm for forgetting documents in datastore (Scott G.
Miller)
11. Re: Proposal: algorithm for forgetting documents in datastore (Oskar
Sandberg)
12. RE: Proposal: algorithm for forgetting documents in datastore (Benjamin
Coates)
13. Re: Proposal: algorithm for forgetting documents in
datastore (Mark J. Roberts)
14. Re: Proposal: algorithm for forgetting documents in datastore (Oskar
Sandberg)
15. Re: Proposal: algorithm for forgetting documents in datastore (Mr.Bad)
--__--__--
Message: 1
Date: Sat, 27 Jan 2001 14:05:13 -0600 (CST)
From: "Mark J. Roberts" <m...@statesmean.com>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Problems! -> 0.3.7 any day now
Reply-To: devl at freenetproject.org
On Sat, 27 Jan 2001, Timm Murray wrote:
> Mark J. Roberts wrote on 1/22/01 11:06 am:
>
> >On Mon, 22 Jan 2001, Oskar
> >Sandberg wrote:
> <>
> >> Can't our plugin intervene
> >and change the URL then?
> >
> >I think they may already do
> >just that. Maybe I'll test this
> >today.
>
> If you've downloaded that love-android movie, you already
> have. For me, it downloads the file and THEN Win Media Player
> plays it. Does not stream.
So fucking what? Your anecdote does not negate the fact that some helper
apps ARE passed a URL. Netscape has a special syntax for this, where %s
means download first, pass the filename of the temp file; and %h means
pass the URL without downloading.
The right question is "how often is this URL-passing scheme used anymore?"
--
Mark Roberts
mjr at statesmean.com
--__--__--
Message: 2
Date: Sat, 27 Jan 2001 20:38:56 +0000
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: toad <matt...@toseland.f9.co.uk>
Reply-To: devl at freenetproject.org
On Sat, Jan 27, 2001 at 02:45:17PM -0500, Chris Anderson wrote:
>
> On Sat, 27 Jan 2001, Mark J. Roberts wrote:
>
> > On Sat, 27 Jan 2001, Oskar Sandberg wrote:
> >
> > > Perhaps a compromise that sets VAD = 1/f(size) where f is some
> > > less than linear function is a possible compromise.
> >
> > I tend to favor Ian's model because I'm a proponent of
> > segmentation (even slightly redundant segmentation), and Ian's
> > model encourages such segmentation. However, I certainly
> > wouldn't want to see it pushed to extremes, where data must be
> > inserted as tiny segments to survive. But I think we should
> > discourage large (>1M) files. Looks like a compromise is
> > necessary.
>
> Segmentation does make it a lot easier to maximize caching without
> giving up much variety.
>
> The danger of VAD(size=1Kb) > VAD(size=1Mb) is that cache
> utilization will drop over time if the number of DataStore entries
> is constant.
>
> Say dataStoreSize = 1000 and diskCache = 100Mb, if the
> segmentation size is 256Kb, there would be 400 * 256Kb disk blocks
> in your DataStore.
>
Why do we have a fixed dataStoreSize? Will this be removed (so that we only set
diskCache) when we use a filesystem-in-a-file in 0.4?
> If average small file size is 4Kb bytes (a newspaper article, but
> most redirects less the 500 bytes), a single 256Kb disk block
> would hold 64 of them. You would need at most 16 disk blocks of
> small files to fill up the entire dataStore. To maximize both
> dataStore and diskCache, if you reserve 360 dataStore entries for
> large blocks, you are still left with 640 small file entries for
> the remaining 10Mb.
>
>
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://www.uprizer.com/mailman/listinfo/devl
--__--__--
Message: 3
Date: Sat, 27 Jan 2001 21:59:57 +0100
From: Oskar Sandberg <md98-...@nada.kth.se>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, Jan 27, 2001 at 08:38:56PM +0000, toad wrote:
> Why do we have a fixed dataStoreSize?
Because DataStoreSize is the size of the routing table. Not all entries in
the table contain data.
> Will this be removed (so that we only set
> diskCache) when we use a filesystem-in-a-file in 0.4?
No, of course not.
--
'DeCSS would be fine. Where is it?'
'Here,' Montag touched his head.
'Ah,' Granger smiled and nodded.
Oskar Sandberg
md98-osa at nada.kth.se
--__--__--
Message: 4
Date: Sat, 27 Jan 2001 15:04:15 -0600 (CST)
From: "Mark J. Roberts" <m...@statesmean.com>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, 27 Jan 2001, Oskar Sandberg wrote:
> On Sat, Jan 27, 2001 at 08:38:56PM +0000, toad wrote:
> > Why do we have a fixed dataStoreSize?
>
> Because DataStoreSize is the size of the routing table. Not all entries in
> the table contain data.
>
> > Will this be removed (so that we only set
> > diskCache) when we use a filesystem-in-a-file in 0.4?
>
> No, of course not.
Solely for reasons of efficiency and memory use?
--
Mark Roberts
mjr at statesmean.com
--__--__--
Message: 5
Date: Sat, 27 Jan 2001 15:37:52 -0600 (CST)
From: Brandon <bl...@uts.cc.utexas.edu>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] FProxy URLS should use %40, not '@'
Reply-To: devl at freenetproject.org
> >That excludes apt-get, xmms, Real Audio, and Windows Media Player from
> >the list of being in any way relevant.
>
> Apt-get will NOT need a patch. In fact, doing so would just be useless
> bloat, since its as simple as the user setting a config line, as opposed
> to some web developer off somewhere hoping that the user
> has a freenet: plugin or runs FProxy on port 8081. Apt is irrelevent
> for this discussion.
Um, yeah, that's what I said.
--__--__--
Message: 6
Date: Sat, 27 Jan 2001 22:41:49 +0100
From: Oskar Sandberg <md98-...@nada.kth.se>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, Jan 27, 2001 at 03:04:15PM -0600, Mark J. Roberts wrote:
> On Sat, 27 Jan 2001, Oskar Sandberg wrote:
>
> > On Sat, Jan 27, 2001 at 08:38:56PM +0000, toad wrote:
> > > Why do we have a fixed dataStoreSize?
> >
> > Because DataStoreSize is the size of the routing table. Not all entries in
> > the table contain data.
> >
> > > Will this be removed (so that we only set
> > > diskCache) when we use a filesystem-in-a-file in 0.4?
> >
> > No, of course not.
>
> Solely for reasons of efficiency and memory use?
Because DataStoreSize is the size of the routing table. Not all entries in
the table contain data.
>
>
> --
> Mark Roberts
> mjr at statesmean.com
>
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://www.uprizer.com/mailman/listinfo/devl
--
'DeCSS would be fine. Where is it?'
'Here,' Montag touched his head.
'Ah,' Granger smiled and nodded.
Oskar Sandberg
md98-osa at nada.kth.se
--__--__--
Message: 7
Date: Sat, 27 Jan 2001 15:45:32 -0600 (CST)
From: "Mark J. Roberts" <m...@statesmean.com>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, 27 Jan 2001, Oskar Sandberg wrote:
> > Solely for reasons of efficiency and memory use?
>
> Because DataStoreSize is the size of the routing table. Not all entries in
> the table contain data.
I understand that--perhaps I was unclear. Is there any reason why bigger
routing tables are bad besides efficiency problems?
--
Mark Roberts
mjr at statesmean.com
--__--__--
Message: 8
To: Freenet Development List <devl at freenetproject.org>
From: Steven Hazel <s...@thalassocracy.org>
Date: 27 Jan 2001 15:54:23 -0600
Subject: [freenet-devl] libfreenet page
Reply-To: devl at freenetproject.org
There's now a skeletal libfreenet page up at
http://thalassocracy.org/libfreenet/ for your linking pleasure.
--__--__--
Message: 9
Date: Sat, 27 Jan 2001 16:55:19 -0500
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: "Scott G. Miller" <scgmi...@indiana.edu>
Reply-To: devl at freenetproject.org
--cvVnyQ+4j833TQvp
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
> > giving up much variety.
> >=20
> > The danger of VAD(size=3D1Kb) > VAD(size=3D1Mb) is that cache
> > utilization will drop over time if the number of DataStore entries
> > is constant.
> >=20
> > Say dataStoreSize =3D 1000 and diskCache =3D 100Mb, if the
> > segmentation size is 256Kb, there would be 400 * 256Kb disk blocks
> > in your DataStore.
> >=20
> Why do we have a fixed dataStoreSize? Will this be removed (so that we on=
ly set
> diskCache) when we use a filesystem-in-a-file in 0.4?
It can be. =20
--cvVnyQ+4j833TQvp
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6c0PHr9IW4v3mHtQRAjKCAJ4tQV4wwqZCUaSnpilipCo9l+r2KgCgihrU
RFIV5Ex0+tWXdbKf74A9lGA=
=/gBT
-----END PGP SIGNATURE-----
--cvVnyQ+4j833TQvp--
--__--__--
Message: 10
Date: Sat, 27 Jan 2001 16:55:40 -0500
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: "Scott G. Miller" <scgmi...@indiana.edu>
Reply-To: devl at freenetproject.org
--0vzXIDBeUiKkjNJl
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Sat, Jan 27, 2001 at 09:59:57PM +0100, Oskar Sandberg wrote:
> On Sat, Jan 27, 2001 at 08:38:56PM +0000, toad wrote:
> > Why do we have a fixed dataStoreSize?=20
>=20
> Because DataStoreSize is the size of the routing table. Not all entries in
> the table contain data.
>=20
> > Will this be removed (so that we only set
> > diskCache) when we use a filesystem-in-a-file in 0.4?
>=20
> No, of course not.
Ah. I thought this was the number of files allowed in the datastore. My
bad.
--0vzXIDBeUiKkjNJl
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE6c0Pcr9IW4v3mHtQRAsX+AJ9o+c6hx/eLLbEf8dk2932R3jaBTQCeLVzw
9WhIYzvhjAWLMJ6a/pS2Lu8=
=h+g5
-----END PGP SIGNATURE-----
--0vzXIDBeUiKkjNJl--
--__--__--
Message: 11
Date: Sat, 27 Jan 2001 23:02:17 +0100
From: Oskar Sandberg <md98-...@nada.kth.se>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, Jan 27, 2001 at 03:45:32PM -0600, Mark J. Roberts wrote:
> On Sat, 27 Jan 2001, Oskar Sandberg wrote:
>
> > > Solely for reasons of efficiency and memory use?
> >
> > Because DataStoreSize is the size of the routing table. Not all entries in
> > the table contain data.
>
> I understand that--perhaps I was unclear. Is there any reason why bigger
> routing tables are bad besides efficiency problems?
Um, the reason we have a dataStoreSize var is that bigger routing tables
are good, we don't want to get rid of the entries from the routing table
just because we don't have space to store the data.
--
'DeCSS would be fine. Where is it?'
'Here,' Montag touched his head.
'Ah,' Granger smiled and nodded.
Oskar Sandberg
md98-osa at nada.kth.se
--__--__--
Message: 12
Date: Sat, 27 Jan 2001 17:14:55 -0500
From: Benjamin Coates <coa...@mailandnews.com>
To: devl at freenetproject.org
Subject: RE: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
>From Tavin Cole <tavin at mailandnews.com>
>It seems totally reasonble for a node to reject caching documents greater
>in size than a certain fraction of the total datastore size (1% or less
>I should think). That prevents your worst-case example, but saves us from
>making relative value judgments using file size as the metric..
>
>--
>
>// Tavin Cole
1% is pretty tiny, that's 1MB on the default datastore size, and also leads to
the bizarre situation where all the insert nodes pass on the request and
return success, even though none of them store it. Maybe nodes should return
failure instead of passing through an insert they can't cache...
--
Benjamin Coates
--__--__--
Message: 13
Date: Sat, 27 Jan 2001 16:17:30 -0600 (CST)
From: "Mark J. Roberts" <m...@statesmean.com>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, 27 Jan 2001, Oskar Sandberg wrote:
> On Sat, Jan 27, 2001 at 03:45:32PM -0600, Mark J. Roberts wrote:
> > On Sat, 27 Jan 2001, Oskar Sandberg wrote:
> >
> > > > Solely for reasons of efficiency and memory use?
> > >
> > > Because DataStoreSize is the size of the routing table. Not all entries in
> > > the table contain data.
> >
> > I understand that--perhaps I was unclear. Is there any reason why bigger
> > routing tables are bad besides efficiency problems?
>
> Um, the reason we have a dataStoreSize var is that bigger routing tables
> are good, we don't want to get rid of the entries from the routing table
> just because we don't have space to store the data.
Of course. What prompted me to ask is that dataStoreSize is set quite low
by default (1000). A more efficient system could raise that value, which
would presumably improve routing.
--
Mark Roberts
mjr at statesmean.com
--__--__--
Message: 14
Date: Sat, 27 Jan 2001 23:48:11 +0100
From: Oskar Sandberg <md98-...@nada.kth.se>
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
Reply-To: devl at freenetproject.org
On Sat, Jan 27, 2001 at 05:14:55PM -0500, Benjamin Coates wrote:
<>
> 1% is pretty tiny, that's 1MB on the default datastore size, and also leads
> to
> the bizarre situation where all the insert nodes pass on the request and
> return success, even though none of them store it. Maybe nodes should return
> failure instead of passing through an insert they can't cache...
I haven't really been able to make up my mind on this. Should I reply
with a RequestFailed on Inserts for data larger than whatever the nodes
limit is?
>
> --
> Benjamin Coates
>
>
> _______________________________________________
> Devl mailing list
> Devl at freenetproject.org
> http://www.uprizer.com/mailman/listinfo/devl
--
'DeCSS would be fine. Where is it?'
'Here,' Montag touched his head.
'Ah,' Granger smiled and nodded.
Oskar Sandberg
md98-osa at nada.kth.se
--__--__--
Message: 15
To: devl at freenetproject.org
Subject: Re: [freenet-devl] Proposal: algorithm for forgetting documents in
datastore
From: Mr.Bad <mr....@pigdog.org>
Organization: Pigdog Journal
Date: 27 Jan 2001 14:47:30 -0800
Reply-To: devl at freenetproject.org
>>>>> "MJR" == Mark J Roberts <mjr at statesmean.com> writes:
MJR> Of course. What prompted me to ask is that dataStoreSize is
MJR> set quite low by default (1000). A more efficient system
MJR> could raise that value, which would presumably improve
MJR> routing.
One thing that seems strange to me about this discussion is that large
files are not necessarily less valuable than small files. I'd
personally want to make sure that, say, a Rodney King video stayed in
cache a lot more than the last 30 index pages of Pigdog Journal.
I wonder if we could come up with a metric that takes into
account -all- the variables that make an entry valuable, e.g.:
- recency: how recently was this file inserted or retrieved?
- frequency: how many times has this file been inserted or
retrieved?
- size: how much space in the cache does this file take up?
I dunno if it's possible, but it'd be interesting to make it explicit
and make the cache therefore a lot dumber (= simpler, more robust, la
la la). It could do something like this on insert:
dse.sortMetric = determineSortMetric(dse);
insertIntoCache(dse);
...where insertIntoCache just does an insert into a sorted array,
which is relatively easy to do. Similarly, on a retrieve, it'd be
something like:
dse.sortMetric = determineSortMetric(dse);
removeFromCache(dse);
insertIntoCache(dse);
The nice part about doing this is that all the smarts of the sorting
go into one method, which can get tweaked fairly easily without
messing with the rest of the DataStore code.
One piece of curiosity: as best I can tell, how the DataStore array is
sorted has little or no impact on routing. Is that right?
~Mr. Bad
P.S. One possible metric could be something like:
sort metric = (sum over hits (time of hit)) / size
...which would depend on storing records of every hit. It probably
also isn't very well tuned. We'd probably also have to throw in some
scaling factors or something.
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/\____/\ Mr. Bad <mr.bad at pigdog.org>
\ / Pigdog Journal | http://pigdog.org/ | *Stay*Real*Bad*
| (X \x)
( ((**) "If it's not bad, don't do it.
\ <vvv> If it's not crazy, don't say it." - Ben Franklin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--__--__--
_______________________________________________
Devl mailing list
Devl at freenetproject.org
http://www.uprizer.com/mailman/listinfo/devl
End of Devl Digest
