[freenet-support] Updated scripts (including a new dedupe.sh!)
OK, I've finally come up with a deduping script. It's not perfect, in that I
didn't know what to do with duplicate nodes with the same version, so I just
left them alone, but it's a start. Hopefully, over time, it'll work out OK,
the idea being that as later versions of nodes are added, the older dupes will
drop out.
For this reason, I've modified the addnodes.sh script to just add everything in
noderefs.txt (after first deduping it) to seednodes.ref, and then deduping
seednodes.ref. The problem with the earlier version of addnodes.sh was that if
a node was already present in seednodes.ref, the one in noderefs.txt would
never get added, even if it was newer.
I've also cleaned up and normalized all the other scripts, cleaning up the
headers and adding usage tips, as well as made them safer by having them backup
files before modifying them.
Hope you'll find them useful.
Conrad (my God, it's nearly 5 a.m.!)
--
Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas
#!/bin/sh
#
# versions.sh
#
# Show a count of the number of each unique node version in input file
#
# Author: dolphin
#
# Usage: versions.sh inputfile
if [ $# -ne 1 ]
then
echo Usage: versions.sh inputfile
exit 1
fi
grep ^version $1 | sort | uniq -c
#!/bin/sh
#
# Displays IP and version info for each node reference in a file
#
# Author: dolphin
#
# Usage: nodes.sh infile
if [ $# -ne 1 ]
then
echo Usage: nodes.sh infile
exit 1
fi
#function: read a single noderef from input, save to temp file
read_one_noderef()
{
rm -f /tmp/noderef.$$
while read line
do
echo $line /tmp/noderef.$$
if [ $line = End ]
then
break
fi
done
}
infile=$1
exec $infile
i=0
while :
do
read_one_noderef
if [ ! -f /tmp/noderef.$$ ]
then
break #end of file
fi
physical_tcp=$(awk '/^physical.tcp=/ {split($0, a, [=:]); print a[2]}' /tmp/n
oderef.$$)
echo -n physical.tcp: ${physical_tcp}:
port=$(awk -F':' '/^physical.tcp=/ {print $2}' /tmp/noderef.$$)
echo -n ${port}
version=$(awk -F',' '/^version=/ {print $4}' /tmp/noderef.$$)
echo version: ${version}
i=$((i + 1))
done
echo $i node references found
#!/bin/sh
#
# refresh_bookmarks.sh
#
# Keep freenet bookmarks refreshed
#
# Author: dolphin
#
# Usage: ./refresh_bookmarks.sh
#
# Put this script in your main freenet dir and run from there
# (expects to find freenet.conf and lib/freenet.jar in current dir)
# Uncomment the following line to skip searching the local store
# (better for improving your node's routing), or comment it out
# to use the local store
skipDS='--skipDS'
logLevel=error #adjust to taste
cd $(realpath $(dirname $0))
HTL=$(awk -F'=' '/maxHopsToLive=/ {print $2}' freenet.conf)
SleepOnFail=$(($(awk -F'=' '/failureTableTime=/ {print $2}' freenet.conf) / 500)
)
SleepOnSuccess=$((SleepOnFail / 2))
for key in $(awk -F'=' '/bookmarks\..*\.key=/ {print $2}' freenet.conf | sort -u
)
do
{ while :
do
echo Refreshing $key
if java -cp lib/freenet.jar freenet.client.cli.Main get $key \
--htl ${HTL} ${skipDS} --logLevel ${logLevel}
then
sleep ${SleepOnSuccess}
else
sleep ${SleepOnFail}
fi
done
} #don't remove this ampersand here! Not a typo! :-)
sleep 600 #wait 10 minutes before invoking next instance
done
#!/bin/sh
#
# dedupe.sh
#
# Dedupe nodes in input file (seednodes.ref or noderefs.txt)
#
# Author: dolphin
#
# Usage: ./dedupe.sh inputfile
#
# Put this script in your main freenet dir and run from there
# (companion script addnodes.sh expects to find it in current dir)
#
# Algorithm:
#
# Step 1: backup input file! :-)
#
# Step 2: read noderef records one at a time from input file,
# saving version info to a temp file for each unique IP
#
# Step 3: sort unique the version numbers in each IP's temp file
#
# Step 4: reread the input file one record at a time as above,
# saving any nodes whose version is found in its IP's
# temp file, discarding all others, writing output to
# a temporary nodes file
#
# Step 5: move temporary nodes file to original input file
#
# Not terribly elegant, but it gets the job one :-)
#
# One little bugaboo: if a node has only multiple
# references to a single version, none are deleted.
# Didn't know how else to handle this without more
# to go on. :-)
if [ $# -ne 1 ]
then
echo
[freenet-support] Re: [freenet-dev] Updated scripts (corrections)
D'oh! That's what I get for coding into the wee hours of the morning on too
much coffee and not enough food. :-)
I've attached corrections to the addnodes.sh and dedupe.sh scripts. It
occurred to me later that if addnodes.sh is run periodically without *really*
eliminating duplicate noderefs using the same version, then the seednodes.ref
file is going to just grow and grow and grow.
I've solved the problem by a sort of brute force method for now. In the case
of duplicate noderefs using the same version, only one is saved, the others are
mindlessly tossed in the bit bucket.
I've also improved the code in a few other places, removing some unnecessary
stuff, simplifying and speeding it up a good bit.
Enjoy!
On 17-Feb-2004 Conrad Sabatier wrote:
OK, I've finally come up with a deduping script. It's not perfect, in that I
didn't know what to do with duplicate nodes with the same version, so I just
left them alone, but it's a start. Hopefully, over time, it'll work out OK,
the idea being that as later versions of nodes are added, the older dupes
will
drop out.
For this reason, I've modified the addnodes.sh script to just add everything
in
noderefs.txt (after first deduping it) to seednodes.ref, and then deduping
seednodes.ref. The problem with the earlier version of addnodes.sh was that
if
a node was already present in seednodes.ref, the one in noderefs.txt would
never get added, even if it was newer.
I've also cleaned up and normalized all the other scripts, cleaning up the
headers and adding usage tips, as well as made them safer by having them
backup
files before modifying them.
Hope you'll find them useful.
Conrad (my God, it's nearly 5 a.m.!)
--
Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas
--
Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas
#!/bin/sh
#
# dedupe.sh
#
# Dedupe nodes in input file (seednodes.ref or noderefs.txt)
#
# Author: dolphin
#
# Usage: ./dedupe.sh inputfile
#
# Put this script in your main freenet dir and run from there
# (companion script addnodes.sh expects to find it in current dir)
#
# Algorithm:
#
# Step 1: backup input file! :-)
#
# Step 2: read noderef records one at a time from input file,
# saving version info to a temp file for each unique IP
#
# Step 3: remove all but the highest version number in each IP's
# temp file
#
# Step 4: reread the input file one record at a time as above,
# saving only the first node whose version is found in
# its IP's temp file, discarding all others, writing
# output to a temporary nodes file
#
# Step 5: move temporary nodes file to original input file
#
# Not terribly elegant, but it gets the job one :-)
#
# One little bugaboo: if a node has only multiple
# references to a single version, all but one are deleted.
# Didn't know how else to handle this without more
# to go on. :-)
if [ $# -ne 1 ]
then
echo Usage: ./dedupe.sh infile
exit 1
fi
infile=$1
cp ${infile} ${infile}.bak #save a copy just in case!
outfile=/tmp/$(basename ${infile}).$$
rm -f /tmp/node.* ${outfile}
#
#function: read a single noderef from input, save to temp file
#
read_one_noderef()
{
rm -f /tmp/noderef.$$
while read line
do
echo $line /tmp/noderef.$$
if [ $line = End ]
then
break
fi
done
}
# Collect node version info
i=0
while :
do
read_one_noderef
if [ ! -f /tmp/noderef.$$ ]
then
break #end of file
fi
physical_tcp=$(awk '/^physical.tcp=/ {split($0, a, [=:]); print a[2]}' /tmp/n
oderef.$$)
version=$(awk -F',' '/^version=/ {print $4}' /tmp/noderef.$$)
echo ${version} /tmp/node.${physical_tcp}
i=$((i + 1))
done $infile
echo $i node references found in $infile
echo Deduping...
#
# unique-ify version info (determine latest version of a node)
#
for f in /tmp/node.*
do
sort -urn -o ${f}.tmp $f
head -n 1 ${f}.tmp $f
done
i=0
while :
do
read_one_noderef
if [ ! -f /tmp/noderef.$$ ]
then
break #end of file
fi
physical_tcp=$(awk '/^physical.tcp=/ {split($0, a, [=:]); print a[2]}' /tmp/n
oderef.$$)
version=$(awk -F',' '/^version=/ {print $4}' /tmp/noderef.$$)
if grep -q $version /tmp/node.${physical_tcp}
then
cat /tmp/noderef.$$ $outfile
cat /dev/null /tmp/node.${physical_tcp}
i=$((i + 1))
fi
done $infile
mv $outfile $infile
echo Saved $i unique nodes to $infile
rm -f /tmp/node*
#!/bin/sh
#
# addnodes.sh
#
# Add nodes from current routing table to seednodes.ref
[freenet-support] Re: [freenet-dev] Updated scripts (one more correction)
One more correction to addnodes.sh: Rather than copying noderefs.txt to the end of seednodes.ref before deduping seednodes.ref, put it at the beginning. This way the (presumably) newer nodes from the freshly fetched noderefs.txt are guaranteed to win out over any older ones in seednodes.ref. Also, backup seednodes.ref in addnodes.sh before doing anything. Meant to do this on the last go-round, but it slipped my mind. :-) Enjoy! On 17-Feb-2004 Conrad Sabatier wrote: D'oh! That's what I get for coding into the wee hours of the morning on too much coffee and not enough food. :-) I've attached corrections to the addnodes.sh and dedupe.sh scripts. It occurred to me later that if addnodes.sh is run periodically without *really* eliminating duplicate noderefs using the same version, then the seednodes.ref file is going to just grow and grow and grow. I've solved the problem by a sort of brute force method for now. In the case of duplicate noderefs using the same version, only one is saved, the others are mindlessly tossed in the bit bucket. I've also improved the code in a few other places, removing some unnecessary stuff, simplifying and speeding it up a good bit. Enjoy! On 17-Feb-2004 Conrad Sabatier wrote: OK, I've finally come up with a deduping script. It's not perfect, in that I didn't know what to do with duplicate nodes with the same version, so I just left them alone, but it's a start. Hopefully, over time, it'll work out OK, the idea being that as later versions of nodes are added, the older dupes will drop out. For this reason, I've modified the addnodes.sh script to just add everything in noderefs.txt (after first deduping it) to seednodes.ref, and then deduping seednodes.ref. The problem with the earlier version of addnodes.sh was that if a node was already present in seednodes.ref, the one in noderefs.txt would never get added, even if it was newer. I've also cleaned up and normalized all the other scripts, cleaning up the headers and adding usage tips, as well as made them safer by having them backup files before modifying them. Hope you'll find them useful. Conrad (my God, it's nearly 5 a.m.!) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas #!/bin/sh # # addnodes.sh # # Add nodes from current routing table to seednodes.ref # Dedupes seednodes.ref after adding nodes # # Author: dolphin # # Usage: ./addnodes.sh # # Put this script in your main freenet dir and run from there # (expects to find noderefs.txt, seednodes.ref and dedupe.sh # in current dir) # # Requires: wget cd $(realpath $(dirname $0)) cp noderefs.txt noderefs.txt.bak echo Fetching noderefs.txt... wget -q -O noderefs.txt http://localhost:/servlet/nodestatus/noderefs.txt ./dedupe.sh noderefs.txt cp seednodes.ref seednodes.ref.bak cat noderefs.txt seednodes.ref.bak seednodes.ref ./dedupe.sh seednodes.ref ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: [freenet-dev] Updated scripts (one more *major* revision)
While going back to correct the problem of seednodes.ref being backed up in both addnodes.sh and dedupe.sh (which would cause the final backed up seednodes.ref.bak to actually be a copy of the raw concatenated noderefs.txt and seednodes.ref), I refined these two scripts even further and added a *lot* of bulletproofing. :-) Also added more echo statements to inform the user as to their progress. They should both be quite safe to use now, and are working just as I wanted them to. :-) Jeez, between this and all the work I've been doing modifying the spider, I'm whipped! I think I'll just *play* with the computer my remaining time off (yeah, right!) :-) Enjoy! On 17-Feb-2004 Conrad Sabatier wrote: One more correction to addnodes.sh: Rather than copying noderefs.txt to the end of seednodes.ref before deduping seednodes.ref, put it at the beginning. This way the (presumably) newer nodes from the freshly fetched noderefs.txt are guaranteed to win out over any older ones in seednodes.ref. Also, backup seednodes.ref in addnodes.sh before doing anything. Meant to do this on the last go-round, but it slipped my mind. :-) Enjoy! On 17-Feb-2004 Conrad Sabatier wrote: D'oh! That's what I get for coding into the wee hours of the morning on too much coffee and not enough food. :-) I've attached corrections to the addnodes.sh and dedupe.sh scripts. It occurred to me later that if addnodes.sh is run periodically without *really* eliminating duplicate noderefs using the same version, then the seednodes.ref file is going to just grow and grow and grow. I've solved the problem by a sort of brute force method for now. In the case of duplicate noderefs using the same version, only one is saved, the others are mindlessly tossed in the bit bucket. I've also improved the code in a few other places, removing some unnecessary stuff, simplifying and speeding it up a good bit. Enjoy! On 17-Feb-2004 Conrad Sabatier wrote: OK, I've finally come up with a deduping script. It's not perfect, in that I didn't know what to do with duplicate nodes with the same version, so I just left them alone, but it's a start. Hopefully, over time, it'll work out OK, the idea being that as later versions of nodes are added, the older dupes will drop out. For this reason, I've modified the addnodes.sh script to just add everything in noderefs.txt (after first deduping it) to seednodes.ref, and then deduping seednodes.ref. The problem with the earlier version of addnodes.sh was that if a node was already present in seednodes.ref, the one in noderefs.txt would never get added, even if it was newer. I've also cleaned up and normalized all the other scripts, cleaning up the headers and adding usage tips, as well as made them safer by having them backup files before modifying them. Hope you'll find them useful. Conrad (my God, it's nearly 5 a.m.!) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas dedupe.sh Description: dedupe.sh addnodes.sh Description: addnodes.sh ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: routing table
Hi Toad, on Mon, 16 Feb 2004 11:37:46 + you wrote: Before i start i must say i always like the idea of freenet. I tried it a couple time long ago but i never had enouth disk space to devote. I tried the current stable yesterday and i must say i was disapointed. First, like many here, by the insane amout of mem it sucked. But then i was more impressed by the reaction of the freenet ppl. So i put my rant too. But don't get me wrong i work on oss too, i know that you just do what you feel like doing. And it's good so :) As I have said on numerous occasions: RAM is cheap. Freedom or working software is not cheap, it's bloody expensive. RAM is NOT CHEAP ! You perhaps live in a wealty contry and have big money. But this is a privilege, not more. RAM is fucking expansive by my standards (and yes i live in a wealty contry) and freedom can be an end only if it's freedom for all. Otherwise you just create a new caste of privilegied. Do we really need one more ?? 79MB is way under what most nodes use. We may be able to get perm node usage down to 80MB at some point, but it's not an immediate priority. I just took a quick look at this list and could find several ppl in the same position as i. We have good old server (mine is a celeron 400 with 128MB) witch can handle ANY of todays servers easily. So we would like to run a frennet node. But we just can't Don't you think it would be nice to have a working network? Sure but again if it's a network of eltist who expect you to buy a new computer just to be able to run a permanent node i don't see the point. Albeu -- Everything is controlled by a small evil group to which, unfortunately, no one we know belongs. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: routing table
On Tue, Feb 17, 2004 at 07:19:28PM +0100, Alban Bedel wrote: Hi Toad, on Mon, 16 Feb 2004 11:37:46 + you wrote: Before i start i must say i always like the idea of freenet. I tried it a couple time long ago but i never had enouth disk space to devote. I tried the current stable yesterday and i must say i was disapointed. Yes, there are major problems with the network that are being worked out, agonizingly slowly.. First, like many here, by the insane amout of mem it sucked. It's unfortunate, however given the size of the network, it's performance level, and the consequent reasonable deduction that only geeks will use it, it's not so unreasonable for this to be only a moderately high priority. But then i was more impressed by the reaction of the freenet ppl. So i put my rant too. But don't get me wrong i work on oss too, i know that you just do what you feel like doing. And it's good so :) Well, I work on what is best for the project, because I get paid for it. But obviously the volunteers do what they feel like. As I have said on numerous occasions: RAM is cheap. Freedom or working software is not cheap, it's bloody expensive. RAM is NOT CHEAP ! You perhaps live in a wealty contry and have big money. But this is a privilege, not more. RAM is fucking expansive by my standards (and yes i live in a wealty contry) and freedom can be an end only if it's freedom for all. Otherwise you just create a new caste of privilegied. Do we really need one more ?? RAM is subject to Moore's Law. Freedom, and software development, are not. Having said that we do have a problem with memory usage with all the reports of nodes crashing with OOMs even when given 250MB+. 79MB is way under what most nodes use. We may be able to get perm node usage down to 80MB at some point, but it's not an immediate priority. I just took a quick look at this list and could find several ppl in the same position as i. We have good old server (mine is a celeron 400 with 128MB) witch can handle ANY of todays servers easily. So we would like to run a frennet node. But we just can't Upgrade it to 256MB. Your motherboard doesn't support 256MB? That sounds unlikely if it can take a celeron 400. If you are so poor that you can't afford the $50 upgrade, well frankly, can you afford the bandwidth? Don't you think it would be nice to have a working network? Sure but again if it's a network of eltist who expect you to buy a new computer just to be able to run a permanent node i don't see the point. We can improve on the memory usage issue once it works. And MEMORY FOR A GIVEN COST DOUBLES EVERY YEAR TO TWO YEARS. We have been working on freenet since ~ 1997, and it certainly has not improved that quickly. Albeu -- Everything is controlled by a small evil group to which, unfortunately, no one we know belongs. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Way to much RAM! Build 5064
On Wed, Jan 28, 2004 at 06:56:42AM -0600, S wrote: On Wed, 28 Jan 2004 13:33:42 +0100 Maximilian Mehnert [EMAIL PROTECTED] wrote: Freenet is one of the most beautiful ideas I ever hit on. But it should be possible to run it on a small pentium machine with no more than 100MB of RAM. I agree 100%. I have a machine dedicated to Freenet. It doesn't do anything else, period. It's a P3 600mhz with 192 megs of RAM. Both stable and unstable will max out its CPU most of the time. I suspect that the core issue is RAM, but I don't know for sure. IF the core issue is CPU, then a working implementation of rate limiting will allow the node to receive just as many requests as it can deal with. If the CPU is maxed out, even if this is due to memory usage/swapping, this will result in high routingTime, and messageSendTimeRequest, both of which are supposed to be taken into account by rate limiting, so again, WHEN rate limiting works properly, it should be able to deal with substandard-performance nodes reasonably well. Actually, I have this exact same problem on my development node, not because of the hardware (which is solid but not excessive - XP 2800+, 1GB RAM, striped IDE), but because of the logging I use, which uses a LOT of RAM, and a LOT of CPU. I've repeatedly seen old machines like my P3-600 disregarded as irrelevant, and not worth optimizing for, in terms of the Freenet network. See above. The best thing I can do for you is get rate limiting working properly. And I think Freenet should easily run on a 600MHz machine, or something is wrong. I'm just skeptical about running on 128MB, or on 200MHz machines. I hesitate to call this particular box old. I have an IBM Aptiva, with a whopping Pentium 75, 40 megs of RAM, running FreeBSD, acting as the gatekeeper for my LAN. It pushes a few gigs worth of data each day, ipfw filtering included, with a load of 0.01 most of the time, and doesn't complain! Now that's what I call old, but the damn thing keeps on rolling. Nothing wrong with that for a firewall machine. Yet I continue to devote the P3 to doing nothing but running a Freenet node, and I will keep doing so for the forseeable future. To me, it's worth it. There have been some significant improvements over the past few months, and I don't doubt that the improvements will continue. You didn't elaborate about how long you'd been away from Freenet, but within the past 6 months, there have been ups and downs. Recently there have been several ups, especially multiplexing. Having 400MB of RAM used by the node's java processes seems out of whack. In fact that sounds insane. Which threadFactory is your configuration file set to use? If you set it to use the YThreadFactory, do things improve? If you can, please keep running your node! -s -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Way to much RAM! Build 5064
On Thu, Jan 29, 2004 at 10:13:05AM +0100, Maximilian Mehnert wrote: Am Mi, den 28.01.2004 schrieb Maximilian Mehnert um 15:28: Having 400MB of RAM used by the node's java processes seems out of whack. In fact that sounds insane. Which threadFactory is your configuration file set to use? If you set it to use the YThreadFactory, do things improve? Sorry. Being online for about 12 hours freenet again succeeded in overloading an acceptable machine (1.5GHz, 512MB Ram), leaving it doing nothing but swapping RAM. I think it's time to take a break. Perhaps I'll check back in a year ;-) I'm still of the opinion that freenet will only spread if people are able to run it on a small router or in background with no noticeable impact on performance. Only _REAL_ geeks will do this. Freenet will spread if it can unobtrusively run on a windoze PC, only needing to be turned off for Quake III. If I had a second life I would help redoing the whole thing in ocaml or something like that. But I have the miserable feeling that studying medicine will keep me busy for the next years. :-( Ok, no more flame wars ;-) :) It is possible to compile java to native code, that may help CPU-wise. Memory-wise, progress needs to be made, but the important thing at the moment is to get rate limiting working properly. Regards, Max -- Maximilian Mehnert [EMAIL PROTECTED] -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Way to much RAM! Build 5064
On Fri, Jan 30, 2004 at 09:28:27PM +1300, Phillip Hutchings wrote: On 29/01/2004, at 10:13 PM, Maximilian Mehnert wrote: Am Mi, den 28.01.2004 schrieb Maximilian Mehnert um 15:28: Having 400MB of RAM used by the node's java processes seems out of whack. In fact that sounds insane. Which threadFactory is your configuration file set to use? If you set it to use the YThreadFactory, do things improve? Sorry. Being online for about 12 hours freenet again succeeded in overloading an acceptable machine (1.5GHz, 512MB Ram), leaving it doing nothing but swapping RAM. I think it's time to take a break. Perhaps I'll check back in a year ;-) I'm still of the opinion that freenet will only spread if people are able to run it on a small router or in background with no noticeable impact on performance. I agree here. My router is a 1.53Ghz Athlon (XP1800+) with 512MB of RAM. The CPU isn't taxed, but the memory is. Also, bandwidth is used quite readily. I have a quota, and it'd be nice to be able to give Freenet a maximum monthly allocation, and have it shut down after that's passed the limit. Have you tried averageOutputBandwidthLimit etc? I have no problem donating 2-3GB/month of traffic, but it takes 4-5 if I don't watch it, that's with a limit of 2kb/sec both ways. : It would be nice if anyone with a spare P266 box could fire up freenet and just let it sit there. Absolutely. If I had a second life I would help redoing the whole thing in ocaml or something like that. But I have the miserable feeling that studying medicine will keep me busy for the next years. :-( Ok, no more flame wars ;-) If I knew the protocol, and knew enough about networking, I'd do a Cocoa client. I have no problems with continually changing the protocol, I'd just have to participate on the developer mailing list. Unfortunately there's no easy place to start from. I guess that's what you get with pre-release software. Another good idea would be a 'freenet browser', something like Gecko or WebKit (for OS X) embedded in to a freenet thing, with privacy options auto set. Yikes. Please, keep the client separate. It can talk to the node via FCP. A freenet specific browser might be nice though. -- Phillip Hutchings [EMAIL PROTECTED] http://www.sitharus.com/ -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Way to much RAM! Build 5064
On Sat, Jan 31, 2004 at 01:21:13PM +1300, Phillip Hutchings wrote: And how many browsers do that? Sure, I'm not sure about writing a plugin, since most of the time they can only add processing for different MIME types, whereas a different browser using a freenet:// protocol could connect through FCP and do things like simplify splitfiles, insertions and the like with an interface that normal users could use. Sure, there's things such as fiw, but it's not the easiest of things to use. Having the whole feature set in one application would make it a lot nicer. Uhm, what is the problem with splitfile downloads? Yes, it LOOKS intimidating, but it's quite usable by the average cubicle monkey let alone freenet-using geek. We can't make it much more transparent, because of issues with resource usage - we MUST have the user confirm the download at some stage, otherwise it can be exploited very badly. There may be workarounds for this of course. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: routing table
Please read my other mail on this subject before passing judgement - what I am working on at the moment may actually have some bearing on this. On Wed, Feb 18, 2004 at 12:15:22AM +, Toad wrote: On Tue, Feb 17, 2004 at 07:19:28PM +0100, Alban Bedel wrote: Hi Toad, on Mon, 16 Feb 2004 11:37:46 + you wrote: Before i start i must say i always like the idea of freenet. I tried it a couple time long ago but i never had enouth disk space to devote. I tried the current stable yesterday and i must say i was disapointed. Yes, there are major problems with the network that are being worked out, agonizingly slowly.. First, like many here, by the insane amout of mem it sucked. It's unfortunate, however given the size of the network, it's performance level, and the consequent reasonable deduction that only geeks will use it, it's not so unreasonable for this to be only a moderately high priority. But then i was more impressed by the reaction of the freenet ppl. So i put my rant too. But don't get me wrong i work on oss too, i know that you just do what you feel like doing. And it's good so :) Well, I work on what is best for the project, because I get paid for it. But obviously the volunteers do what they feel like. As I have said on numerous occasions: RAM is cheap. Freedom or working software is not cheap, it's bloody expensive. RAM is NOT CHEAP ! You perhaps live in a wealty contry and have big money. But this is a privilege, not more. RAM is fucking expansive by my standards (and yes i live in a wealty contry) and freedom can be an end only if it's freedom for all. Otherwise you just create a new caste of privilegied. Do we really need one more ?? RAM is subject to Moore's Law. Freedom, and software development, are not. Having said that we do have a problem with memory usage with all the reports of nodes crashing with OOMs even when given 250MB+. 79MB is way under what most nodes use. We may be able to get perm node usage down to 80MB at some point, but it's not an immediate priority. I just took a quick look at this list and could find several ppl in the same position as i. We have good old server (mine is a celeron 400 with 128MB) witch can handle ANY of todays servers easily. So we would like to run a frennet node. But we just can't Upgrade it to 256MB. Your motherboard doesn't support 256MB? That sounds unlikely if it can take a celeron 400. If you are so poor that you can't afford the $50 upgrade, well frankly, can you afford the bandwidth? Don't you think it would be nice to have a working network? Sure but again if it's a network of eltist who expect you to buy a new computer just to be able to run a permanent node i don't see the point. We can improve on the memory usage issue once it works. And MEMORY FOR A GIVEN COST DOUBLES EVERY YEAR TO TWO YEARS. We have been working on freenet since ~ 1997, and it certainly has not improved that quickly. Albeu -- Everything is controlled by a small evil group to which, unfortunately, no one we know belongs. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED] -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: routing table
Toad schrieb: Upgrade it to 256MB. Your motherboard doesn't support 256MB? That sounds unlikely if it can take a celeron 400. If you are so poor that you can't afford the $50 upgrade, well frankly, can you afford the bandwidth? Well there are Countrys where bandwidth is very cheap or free for home users while memory is really expensive. For around 10$ per month I get unlimited bandwitdh on my adsl line. Take a look at the attached Image to see how cheap brand memory can be (noname memory isn't a subject cause it is a shure way to make your system unstable). This is one of the most expensive here, but it is still a quite good example for way higher prices in different countrys. inline: cheap-memory.jpg___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Millions(sic) of errors
Came home tonight to find that my node had crashed earlier on. Now after restarting I'm seeing millions (literally) of this message in the logfile: 18-Feb-2004 01:44:20 (freenet.ConnectionHandler, YThread-106, ERROR): Caught java.lang.IllegalArgumentException: Should be mux protocol, was [EMAIL PROTECTED] in forceSendPacket java.lang.IllegalArgumentException: Should be mux protocol, was [EMAIL PROTECTED] at freenet.RateLimitingViolationMessage.resolve(RateLimitingViolationMessage.ja va:50) at freenet.PeerHandler.getPacket(PeerHandler.java:1092) at freenet.ConnectionHandler.forceSendPacket(ConnectionHandler.java:2170) at freenet.PeerHandler.sendSinglePacket(PeerHandler.java:1025) at freenet.PeerHandler.innerSendMessageAsync(PeerHandler.java:900) at freenet.PeerHandler.receivedRequest(PeerHandler.java:1771) at freenet.node.states.FNP.NewRequest.genReceived(NewRequest.java:67) at freenet.node.states.FNP.NewDataRequest.received(NewDataRequest.java:35) at freenet.node.StateChain.received(StateChain.java:183) at freenet.node.StateChain.received(StateChain.java:67) at freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:2 36) at freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.j ava:171) at freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler .java:122) at freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:70) at freenet.Ticker$Event.run(Ticker.java:395) In 15 minutes the logfile has reached 1.52GB (yes, GB, that wasn't a typo) with logging at the default level of normal, and it all seems to be this 'should be mux protocol' message. In the few moments it's taken to write this mail, the log is now up to 1.85GB. Aaah. something new now: 18-Feb-2004 01:55:32 (freenet.RateLimitingViolationMessage, Network writing thread, NORMAL): AARGH: couldn't send [EMAIL PROTECTED] Thousands of these too, and then the logfile seems to have stopped growing except for a few: 18-Feb-2004 01:57:39 (freenet.node.rt.StandardNodeEstimator, YThread-203, NORMAL): Caught freenet.node.rt.EstimatorFormatException: Estimate 1088779 over max 100 reading tcp/[snip], sessions=1, presentations=3,1, ID=DSA([snip]), version=Fred,0.5,STABLE-1.50,5069 from FieldSet :( freenet.node.rt.EstimatorFormatException: Estimate 1088779 over max 100 at freenet.node.rt.RoutingPointStore$RoutingPoint.init(RoutingPointStore.java :93) at freenet.node.rt.RoutingPointStore.init(RoutingPointStore.java:351) at freenet.node.rt.HistoryKeepingRoutingPointStore.init(HistoryKeepingRouting PointStore.java:31) at freenet.node.rt.DecayingKeyspaceEstimator$DecayingKeyspaceEstimatorFactory.c reate(DecayingKeyspaceEstimator.java:851) at freenet.node.rt.StandardNodeEstimator.init(StandardNodeEstimator.java:531) at freenet.node.rt.StandardNodeEstimatorFactory.create(StandardNodeEstimatorFac tory.java:110) at freenet.node.rt.StandardNodeEstimatorFactory.create(StandardNodeEstimatorFac tory.java:170) at freenet.node.rt.NGRoutingTable.reference(NGRoutingTable.java:463) at freenet.node.rt.FilterRoutingTable.reference(FilterRoutingTable.java:52) at freenet.node.Node.reference(Node.java:4863) at freenet.node.states.request.AwaitingStoreData.relayStoreData(AwaitingStoreDa ta.java:139) at freenet.node.states.request.AwaitingStoreData.receivedMessage(AwaitingStoreD ata.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at freenet.node.State.received(State.java:131) at freenet.node.StateChain.received(StateChain.java:183) at freenet.node.StateChain.received(StateChain.java:67) at freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:2 36) at freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.j ava:171) at freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler .java:122) at freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:70) at freenet.Ticker$Event.run(Ticker.java:395) at freenet.thread.YThreadFactory$YThread.run(YThreadFactory.java:249) Logfile is now 1.98GB and not growing rapidly. What can cause this? ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Millions(sic) of errors
On Wed, Feb 18, 2004 at 02:04:45AM -, Kevin Bennett wrote: Came home tonight to find that my node had crashed earlier on. Now after restarting I'm seeing millions (literally) of this message in the logfile: This definitely should not happen, it means your node is using the pre-multiplexing connection protocol... there may be a quick fix... 18-Feb-2004 01:44:20 (freenet.ConnectionHandler, YThread-106, ERROR): Caught java.lang.IllegalArgumentException: Should be mux protocol, was [EMAIL PROTECTED] in forceSendPacket java.lang.IllegalArgumentException: Should be mux protocol, was [EMAIL PROTECTED] at freenet.RateLimitingViolationMessage.resolve(RateLimitingViolationMessage.ja va:50) at freenet.PeerHandler.getPacket(PeerHandler.java:1092) at freenet.ConnectionHandler.forceSendPacket(ConnectionHandler.java:2170) at freenet.PeerHandler.sendSinglePacket(PeerHandler.java:1025) at freenet.PeerHandler.innerSendMessageAsync(PeerHandler.java:900) at freenet.PeerHandler.receivedRequest(PeerHandler.java:1771) at freenet.node.states.FNP.NewRequest.genReceived(NewRequest.java:67) at freenet.node.states.FNP.NewDataRequest.received(NewDataRequest.java:35) at freenet.node.StateChain.received(StateChain.java:183) at freenet.node.StateChain.received(StateChain.java:67) at freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:2 36) at freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.j ava:171) at freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler .java:122) at freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:70) at freenet.Ticker$Event.run(Ticker.java:395) In 15 minutes the logfile has reached 1.52GB (yes, GB, that wasn't a typo) with logging at the default level of normal, and it all seems to be this 'should be mux protocol' message. In the few moments it's taken to write this mail, the log is now up to 1.85GB. Aaah. something new now: 18-Feb-2004 01:55:32 (freenet.RateLimitingViolationMessage, Network writing thread, NORMAL): AARGH: couldn't send [EMAIL PROTECTED] Hmmm. Thousands of these too, and then the logfile seems to have stopped growing except for a few: 18-Feb-2004 01:57:39 (freenet.node.rt.StandardNodeEstimator, YThread-203, NORMAL): Caught freenet.node.rt.EstimatorFormatException: Estimate 1088779 over max 100 reading tcp/[snip], sessions=1, presentations=3,1, ID=DSA([snip]), version=Fred,0.5,STABLE-1.50,5069 from FieldSet :( freenet.node.rt.EstimatorFormatException: Estimate 1088779 over max 100 at freenet.node.rt.RoutingPointStore$RoutingPoint.init(RoutingPointStore.java :93) at freenet.node.rt.RoutingPointStore.init(RoutingPointStore.java:351) at freenet.node.rt.HistoryKeepingRoutingPointStore.init(HistoryKeepingRouting PointStore.java:31) at freenet.node.rt.DecayingKeyspaceEstimator$DecayingKeyspaceEstimatorFactory.c reate(DecayingKeyspaceEstimator.java:851) at freenet.node.rt.StandardNodeEstimator.init(StandardNodeEstimator.java:531) at freenet.node.rt.StandardNodeEstimatorFactory.create(StandardNodeEstimatorFac tory.java:110) at freenet.node.rt.StandardNodeEstimatorFactory.create(StandardNodeEstimatorFac tory.java:170) at freenet.node.rt.NGRoutingTable.reference(NGRoutingTable.java:463) at freenet.node.rt.FilterRoutingTable.reference(FilterRoutingTable.java:52) at freenet.node.Node.reference(Node.java:4863) at freenet.node.states.request.AwaitingStoreData.relayStoreData(AwaitingStoreDa ta.java:139) at freenet.node.states.request.AwaitingStoreData.receivedMessage(AwaitingStoreD ata.java:88) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at freenet.node.State.received(State.java:131) at freenet.node.StateChain.received(StateChain.java:183) at freenet.node.StateChain.received(StateChain.java:67) at freenet.node.StandardMessageHandler$Ticket.run(StandardMessageHandler.java:2 36) at freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.j ava:171) at freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler .java:122) at freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:70) at freenet.Ticker$Event.run(Ticker.java:395) at freenet.thread.YThreadFactory$YThread.run(YThreadFactory.java:249) Logfile is now 1.98GB and not growing rapidly. What can cause this? ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
