[freenet-dev] Early thoughts on HTL success rate data
Builds 1234-1236 included several changes to routing that we hope improved things. Specifically: - The "loop fix": when node A is choosing how to route a request it received from node B, when considering node C's FOAF locations, it should ignore locations among C's peers that exactly match B's location. That is, A shouldn't try to route a request back where it came from by an alternate route. Similarly, A should ignore A's own location, and the locations of any node A has already tried to route to and failed (rejectedLoop, rejectedOverload, etc). - FOAF tie breaking: When A is routing a request, if both B and C have the same minimum FOAF distance from the request location (presumably because they have a peer in common), A should tie break on which has the better immediate location. - A change to ULPR propagation that shouldn't have much direct impact, but may have had an indirect impact. This is also the first time we've made network-level changes since I started collecting the hourly HTL success rate data; hopefully, this gives me something to analyze and determine in detail whether the changes made an improvement. First, the caveats: 1236 is not yet self-mandatory, so there may still be some upgrade disruption happening. This is only data from my node; edt has been sending me his hourly stats data as well, but I don't yet have any post-1236 data from him, so I haven't included any of it. There are known local changes at my node (fewer persistent requests queued) that correlate, because my node lost its node.db4o with the 1236 upgrade (though I do attempt to model local request count impact on success rates). I don't yet have enough data to know if there are any weekly effects (though I have attempted to account for time-of-day effects). And all I've done are regression tests; I haven't yet done the various non-parametric statistics tests to produce an actual p-value on whether the change made a difference. I strongly suspect it did, but there's too much structure to my residuals and too much non-normality to the data to be able to say that from what I've done so far. Network load is meaningfully lower on the new data as well, and I don't know how to explain that. To summarize: these results are highly preliminary. Do *not* draw conclusions from them. They are, however, interesting and promising. I'll have better data soon, that we might be able to begin drawing tentative conclusions from. But for now, I think the data are intriguing enough that people would enjoy seeing them. I would be *very* appreciative of comments or suggestions. Now, the request: I'd like more people to send me their success rate data! So far I only have one; that's rather disappointing, given how many people want a faster Freenet. It's not hard; just turn on loglevel normal, make sure you have plenty of space allocated to logs so it doesn't drop old ones, and then run "zgrep HourlyStats logs/freenet-*.gz > output.txt" and send me the results by email. There's nothing non-anonymous in there. You don't have to remove duplicates; I can handle that easily. There's no particular need for you to worry about use of your node, internet connection, uptime, etc; I'm trying to model that, and I should be able to basically average it out. I'd like high-uptime nodes, but it's not required. So, the preliminary results. All data is based on the total observed CHK success rate (both local and remote; the % listed in the success rate by htl box on the stats page). First, the raw success rate vs HTL graph: freenet:CHK at u3EvtVA0k6wVFmcji-ww0-yBdmLKmYEf~hyXD6hbF-o,mphkKgatjdGtpHjf70t7APtBy82eW-GHwPjWWnNQFW4,AAIC--8/rate_both.png The success rates are rather noisy, but it looks at first glance like the new data (magenta) shows higher success rates than the older data (blue). Next, I created a linear regression model that attempted to explain success rates based on other factors. I included HTL, using 4th order polynomial fit, since it showed a fair bit of curving; an exponential or power model is probably better, but nonlinear regression is more complicated. The high order polynomial should match such curves reasonably well. I included a linear fit on local CHK blocks fetched, on the assumption that local load probably has an impact. I included a quadratic fit on total incoming CHK requests, as an indicator of local / global network load. And lastly, I included a sinusoidal fit (amplitude and phase, but not frequency) to time of day, and second and third harmonics. Overall R^2 was 0.84, with high significance values; all parameters were significant at the 1% level (most dramatically better than that), with the exception of the third harmonic of time of day. (The model was built against the unified set of data; we'll look for changes resulting from the new build in the residuals, rather than in model coefficients.) Having explained away a large fraction of the variation in the data, we look at the model residuals:
[freenet-dev] Freenet-Cards: ID cards with your node-ref
Hi, A few years back I wrote freenet-cards which you can give your friends so they can easily check your node-ref. For a long time these lived only on a freesite, but I just pushed them on a normal website to help spread freenet. -> http://freenetcard.draketo.de/ Please use them as you see fit. Best wishes, Arne --- --- --- --- --- --- --- --- --- Unpolitisch sein hei?t politisch sein, ohne es zu merken. - Arne (http://draketo.de) --- --- --- --- --- --- --- --- --- -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090921/37dc022d/attachment.pgp>
[freenet-dev] projectfreenet.org?!?
Projectfreenet.org runs a horribly outdated mirror of the project's site, looks at least a couple of years old. It seems to link to the actual freenet software at the real website, but info and instructions are very old and could confuse users. Can anything be done about that? If the Freenet bunny or the name "the free netweork project" are registered, it should be possible to make them stop, or at least agree to update their mirror more often. Note that some buzz about this mirror site "delivering false information" would get Freenet some pubblicity, which is never a bad thing.
[freenet-dev] BigInteger and NativeBigInteger
Matthew Toseland wrote: > On Saturday 19 September 2009 03:46:27 Ximin Luo wrote: >> http://github.com/freenet/fred-staging/commit/dea2134ba56462afb0806c5e126306bd0441c5bb >> >> Why is this commit necessary? The Java API documentation for >> BigInteger.toByteArray() says "This representation is compatible with the >> (byte[]) constructor." > > I thought you created some random arrays? Yeah, I use the random array to create a BigInteger, but then I create another array from it with toByteArray(), and that's what's passed into the constructors. But nvmd, this is trivial, they are all positive integers anyway. X > > > > > ___ > Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] projectfreenet.org?!?
Projectfreenet.org runs a horribly outdated mirror of the project's site, looks at least a couple of years old. It seems to link to the actual freenet software at the real website, but info and instructions are very old and could confuse users. Can anything be done about that? If the Freenet bunny or the name the free netweork project are registered, it should be possible to make them stop, or at least agree to update their mirror more often. Note that some buzz about this mirror site delivering false information would get Freenet some pubblicity, which is never a bad thing. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Freenet-Cards: ID cards with your node-ref
Hi, A few years back I wrote freenet-cards which you can give your friends so they can easily check your node-ref. For a long time these lived only on a freesite, but I just pushed them on a normal website to help spread freenet. - http://freenetcard.draketo.de/ Please use them as you see fit. Best wishes, Arne --- --- --- --- --- --- --- --- --- Unpolitisch sein heißt politisch sein, ohne es zu merken. - Arne (http://draketo.de) --- --- --- --- --- --- --- --- --- signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Early thoughts on HTL success rate data
Builds 1234-1236 included several changes to routing that we hope improved things. Specifically: - The loop fix: when node A is choosing how to route a request it received from node B, when considering node C's FOAF locations, it should ignore locations among C's peers that exactly match B's location. That is, A shouldn't try to route a request back where it came from by an alternate route. Similarly, A should ignore A's own location, and the locations of any node A has already tried to route to and failed (rejectedLoop, rejectedOverload, etc). - FOAF tie breaking: When A is routing a request, if both B and C have the same minimum FOAF distance from the request location (presumably because they have a peer in common), A should tie break on which has the better immediate location. - A change to ULPR propagation that shouldn't have much direct impact, but may have had an indirect impact. This is also the first time we've made network-level changes since I started collecting the hourly HTL success rate data; hopefully, this gives me something to analyze and determine in detail whether the changes made an improvement. First, the caveats: 1236 is not yet self-mandatory, so there may still be some upgrade disruption happening. This is only data from my node; edt has been sending me his hourly stats data as well, but I don't yet have any post-1236 data from him, so I haven't included any of it. There are known local changes at my node (fewer persistent requests queued) that correlate, because my node lost its node.db4o with the 1236 upgrade (though I do attempt to model local request count impact on success rates). I don't yet have enough data to know if there are any weekly effects (though I have attempted to account for time-of-day effects). And all I've done are regression tests; I haven't yet done the various non-parametric statistics tests to produce an actual p-value on whether the change made a difference. I strongly suspect it did, but there's too much structure to my residuals and too much non-normality to the data to be able to say that from what I've done so far. Network load is meaningfully lower on the new data as well, and I don't know how to explain that. To summarize: these results are highly preliminary. Do *not* draw conclusions from them. They are, however, interesting and promising. I'll have better data soon, that we might be able to begin drawing tentative conclusions from. But for now, I think the data are intriguing enough that people would enjoy seeing them. I would be *very* appreciative of comments or suggestions. Now, the request: I'd like more people to send me their success rate data! So far I only have one; that's rather disappointing, given how many people want a faster Freenet. It's not hard; just turn on loglevel normal, make sure you have plenty of space allocated to logs so it doesn't drop old ones, and then run zgrep HourlyStats logs/freenet-*.gz output.txt and send me the results by email. There's nothing non-anonymous in there. You don't have to remove duplicates; I can handle that easily. There's no particular need for you to worry about use of your node, internet connection, uptime, etc; I'm trying to model that, and I should be able to basically average it out. I'd like high-uptime nodes, but it's not required. So, the preliminary results. All data is based on the total observed CHK success rate (both local and remote; the % listed in the success rate by htl box on the stats page). First, the raw success rate vs HTL graph: freenet:c...@u3evtva0k6wvfmcji-ww0-ybdmlkmyef~hyxd6hbf-o,mphkKgatjdGtpHjf70t7APtBy82eW-GHwPjWWnNQFW4,AAIC--8/rate_both.png The success rates are rather noisy, but it looks at first glance like the new data (magenta) shows higher success rates than the older data (blue). Next, I created a linear regression model that attempted to explain success rates based on other factors. I included HTL, using 4th order polynomial fit, since it showed a fair bit of curving; an exponential or power model is probably better, but nonlinear regression is more complicated. The high order polynomial should match such curves reasonably well. I included a linear fit on local CHK blocks fetched, on the assumption that local load probably has an impact. I included a quadratic fit on total incoming CHK requests, as an indicator of local / global network load. And lastly, I included a sinusoidal fit (amplitude and phase, but not frequency) to time of day, and second and third harmonics. Overall R^2 was 0.84, with high significance values; all parameters were significant at the 1% level (most dramatically better than that), with the exception of the third harmonic of time of day. (The model was built against the unified set of data; we'll look for changes resulting from the new build in the residuals, rather than in model coefficients.) Having explained away a large fraction of the variation in the data, we look at the model residuals:
Re: [freenet-dev] BigInteger and NativeBigInteger
On Sat, Sep 19, 2009 at 7:26 AM, Ximin Luo xl...@cam.ac.uk wrote: I created a new branch (bigint7) which tests the performance of java 7's BigInteger implementation. can people please benchmark it? BigInteger use java.util.Arrays.copyOfRange and friends, which are java 6 only. ___ Devl mailing list Devl@freenetproject.org http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
