Re: [freenet-support] Would someone please re-insert these files?
On 12-Aug-2005 Anonymous via Panta Rhei wrote:
I've had a helluva time trying to d/l these and i'd be gratefull for
a re-insert by anyone who has them.
File: xpcorp.ISO
Key: [EMAIL PROTECTED],t0N0QCiz0-a~LL08UL4XtA
Bytes: 512360448
File: WinXPPlus.ISO
Key: [EMAIL PROTECTED],gNGGF1n0TvCRjdzgtA-NRA
Bytes:
Thank you.
You may be interested in a pair of bash scripts I wrote to help with
large splitfile downloads.
The first, get_splitfile, will try indefinitely to fetch the minimum
number of blocks needed to decode a file, looping forever over the list
of data/check blocks that make up each segment until either the
required number has been successfully retrieved or a user interrupt.
It will automatically spawn a number of background fetches equal to the
number of segments in the splitfile times two (one thread for data
blocks, one for check blocks).
Note that no actual decoding is done. The idea is to get the raw
blocks into the local datastore, so that a later request via fproxy
will succeed more quickly and easily.
Note also that each successfully retrieved block is stored in a
directory *outside* of your actual freenet datastore, so a little
additional disk space will be required.
The second script, watch_splitfile, allows you to monitor the progress
of the first script's operation.
Read the header comments in each file for usage instructions and other
information.
Enjoy!
P.S. Toad, should we maybe add these to our scripts collection in CVS?
:-)
--
Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas
#!/usr/local/bin/bash
#
# get_splitfile -- download an FEC splitfile
#
# usage: get_splitfile key output_filename
#
# Uses a separate thread for each of the splitfile's segments to improve
# throughput
#
# Requires: fcpget (freenet tools) for fetching metadata,
# fnget (zzed's freenet tools) for fetching the actual data blocks
#
# Note: doesn't actually decode the file, but just downloads the blocks so they'
ll
# (hopefully) be ready and available in the local datastore when a real attempt
to
# download/decode the file is done via fproxy
#
HTL=$(awk -F'=' '/^%?maxHopsToLive=/ {print $2}' ../freenet/freenet.conf)
##
#
# Subroutine definitions precede main program
#
##
#
# usage - display correct command line usage on stderr
#
usage()
{
{
echo
echo usage: get_splitfile key output_filename
echo
} 2
}
#
# number format conversion routines
#
#
# d2h - convert unsigned decimal to hex
#
d2h()
{
local d=$1
local h=
local nybble
while [ ${d} -ne 0 ]
do
nybble=$((d 15))
case ${nybble} in
10) nybble=a;;
11) nybble=b;;
12) nybble=c;;
13) nybble=d;;
14) nybble=e;;
15) nybble=f;;
*) nybble=${nybble};;
esac
h=${nybble}${h}
((d = 4))
done
echo ${h}
}
#
# h2d - convert hex string (without any leading 0x) to decimal
#
h2d()
{
local h=$1
local d=0
local nybble
while [ -n ${h} ]
do
nybble=${h:0:1}
case ${nybble} in
a) nybble=10;;
b) nybble=11;;
c) nybble=12;;
d) nybble=13;;
e) nybble=14;;
f) nybble=15;;
*) ;;
esac
d=$(((d 4) + nybble))
h=${h:1}
done
echo ${d}
}
#
#
# Begin main program
#
#
# Check command line, exit if incorrect
if [ ${#} -ne 2 ]
then
usage
exit 1
fi
key=${1}
output_filename=${2}
# Import variables from spider
#
# Note: if not using spider, simply define the variables FCPGET (full path to
# freenet tools' fcpget program) and FCP_HOST (the address of the node to connec
t to)
#
. ${HOME}/freenet/spider/spider.vars
# Prepend freenet: to key if necessary
if [[ ${key#freenet:*} == ${key} ]]
then
key=freenet:${key}
fi
# Replace slashes in key with colons to create filename (without leading
# freenet:) prefix for metadata file and segment files
filename_prefix=$(echo ${key:8} | sed -e 's#/#:#g')
# Create a temporary directory for downloaded data
tempdir=${output_filename}.tmp
mkdir -p ${tempdir}
cd ${tempdir}
# Get the raw metadata for the file
metadata_file=${output_filename}.metadata
while [[ ! -s ${metadata_file} ]]
do
${FCPGET} -n ${FCP_HOST} -R -l ${HTL} -v 3 ${key} ${metadata_file}
|| exit
1
done
# Get the splitfile segment map info
Re: [freenet-support] Would someone please re-insert these files?
Oops, forgot to include a third, supporting script
(FECSegmentSplitFile, attached here).
On 12-Aug-2005 Conrad J. Sabatier wrote:
You may be interested in a pair of bash scripts I wrote to help with
large splitfile downloads.
The first, get_splitfile, will try indefinitely to fetch the minimum
number of blocks needed to decode a file, looping forever over the
list of data/check blocks that make up each segment until either the
required number has been successfully retrieved or a user interrupt.
It will automatically spawn a number of background fetches equal to
the number of segments in the splitfile times two (one thread for
data blocks, one for check blocks).
Note that no actual decoding is done. The idea is to get the raw
blocks into the local datastore, so that a later request via fproxy
will succeed more quickly and easily.
Note also that each successfully retrieved block is stored in a
directory *outside* of your actual freenet datastore, so a little
additional disk space will be required.
The second script, watch_splitfile, allows you to monitor the
progress of the first script's operation.
Read the header comments in each file for usage instructions and
other information.
Enjoy!
P.S. Toad, should we maybe add these to our scripts collection in
CVS? :-)
--
Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas
#!/usr/local/bin/bash
#
# FECSegmentSplitFile -- Do an FECSegmentSplitFile for the splitfile metadata
# contained in the file named in argument $1, save the file's segment info to
# individual header and blockmap files, using the output filename prefix named
# in argument $2
#
# usage: FECSegmentSplitFile metadata_filename filename_prefix
#
#
# usage - display correct command line usage on stderr
#
usage()
{
{
echo
echo usage: FECSegmentSplitFile metadata_filename
filename_prefix
echo
} 2
}
# Check command line, exit if incorrect
if [ ${#} -ne 2 ]
then
usage
exit 1
fi
metadata_file=$1
filename_prefix=$2
metadata_size=$(stat -f %Xz ${metadata_file})
# Import variables from spider
#
# Note: if not using spider, simply define the variable FCP_HOST
# (the address of the node to connect to)
#
. ${HOME}/freenet/spider/spider.vars
# Open a connection to the node on file descriptor 3
exec 3/dev/tcp/${FCP_HOST}/8481
{
# Send the FCP message preamble to the node
echo -ne \00\00\00\02
# Send command to node
echo FECSegmentSplitFile
echo DataLength=${metadata_size}
echo Data
} 3
# Send the data from the metadata file to the node
cat ${metadata_file} 3
# Finish out the command
echo EndMessage 3
# Read and save the node's response to segment files prefixed with
# output file name.
#
# Node's reply should be in the form of SegmentHeader/BlockMap pairs
#
i=0
exec 3
while read -r
do
if [[ ${REPLY} != SegmentHeader ]]
then
exit 1
fi
cat /dev/null ${filename_prefix}.segment.${i}.header
# read and save segment header
while read -r
do
if [[ ${REPLY} == EndMessage ]]
then
break
else
echo ${REPLY} ${filename_prefix}.segment.${i}.header
fi
done
read -r
if [[ ${REPLY} != BlockMap ]]
then
exit 1
fi
cat /dev/null ${filename_prefix}.segment.${i}.blockmap
# read and save segment blockmap
while read -r
do
if [[ ${REPLY} == EndMessage ]]
then
break
else
echo ${REPLY}
${filename_prefix}.segment.${i}.blockmap
fi
done
sort -o ${filename_prefix}.segment.${i}.blockmap
${filename_prefix}.segment.${i
}.blockmap
((++i))
done
exit 0
#!/usr/local/bin/bash
#
# get_splitfile -- download an FEC splitfile
#
# usage: get_splitfile key output_filename
#
# Uses a separate thread for each of the splitfile's segments to improve
# throughput
#
# Requires: fcpget (freenet tools) for fetching metadata,
# fnget (zzed's freenet tools) for fetching the actual data blocks
#
# Note: doesn't actually decode the file, but just downloads the blocks so they'
ll
# (hopefully) be ready and available in the local datastore when a real attempt
to
# download/decode the file is done via fproxy
#
HTL=$(awk -F'=' '/^%?maxHopsToLive=/ {print $2}' ../freenet/freenet.conf)
##
#
# Subroutine definitions precede main program
#
##
#
# usage - display correct command line usage on stderr
#
usage()
{
{
echo
echo usage: get_splitfile key
Re: [freenet-support] Recent mail problems
On 25-Jul-2005 Matthew Toseland wrote: Recently dodo has been having problems. The lists should work now. Yes, it looks like things are back to normal again now. Thanks for taking care of this so quickly! -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Relocating datastore causes total loss of keys
Today I finally finished a little project I had undertaken to reorganize my filesystems. In a nutshell, I decided to dedicate this 200 GB USB drive I had bought recently (for another intended purpose which I've since abandoned) exclusively for use as my freenet datastore. I figured that this way, if one of my machines should suddenly fail, it would be a simple matter just to disconnect the USB drive, plug it into the other machine, and continue right along with barely a hiccup, almost as if nothing had even happened. Not to mention the luxury of having 200 gigs of storage dedicated exclusively to freenet! :-) So anyway, I moved all the directories under my pre-existing datastore over to the newly repartitioned, relabeled, newfs'ed and remounted USB drive, adjusted the storeFile= setting in my freenet.conf to point to the drive's mount point, restarted Freenet, and lo and behold! I was suddenly back to square zero! All the keys I had just relocated were gone! Any idea why such a simple procedure as this would result in the complete loss of my datastore? I can't for the life of me think of any good reason why Fred would not only totally disregard all of my pre-existing keys just because I had relocated them, but even go so far as to delete them all, too. Very strange behavior, if you ask me. Certainly a major violation of the famouse POLA that's bandied about frequenty on the FreeBSD mailing lists. How bizarre! :-) -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] GCJ 4.1
I'm currently in the process of compiling and installing the GCC suite version 4.1. Has anyone yet tried compiling Freenet using this version of GCJ? If so, how successful was it? -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Important security advisory re: vulnerability in multiple web browsers
web browsers -- window injection vulnerabilities Description: A Secunia Research advisory reports: Secunia Research has reported a vulnerability in multiple browsers, which can be exploited by malicious people to spoof the content of websites. The problem is that a website can inject content into another site's window if the target name of the window is known. This can e.g. be exploited by a malicious website to spoof the content of a pop-up window opened on a trusted website. Secunia has constructed a test, which can be used to check if your browser is affected by this issue: http://secunia.com/multiple_browsers_window_injection_vulnerability_test/ A workaround for Mozilla-based browsers is available. http://www.freebsd.org/ports/portaudit/b0911985-6e2a-11d9-9557-000a95bc6fae.html -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Important news for users of stable dual-network nodes -- Please read!
Please excuse the crossposting, but I felt this was important enough to make sure it was seen by those who may not be subscribed to this or that list. The following will be appearing later today in DFI's News section: Feb 25, 2005: Finally got around to committing to CVS an overlooked update to the stable version of src/freenet/Version.java. I had updated the unstable version of this file several weeks ago (see Jan 20, 2005 below), but forgot to sync up the one in stable. As a result, there was a mismatch between the protocol version stable was expecting unstable nodes to be using, and the one unstable nodes actually were using. Those of you running dual-network nodes should start seeing better results now, as more and more stable users update their nodes to include this latest update. Sorry for not taking care of this sooner! -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Two questions
On Thu, 24 Feb 2005 07:02:39 +0100, Marco A. Calamari [EMAIL PROTECTED] wrote: On Wed, 2005-02-23 at 21:44 -0800, Todd Walton wrote: On Wed, 23 Feb 2005 10:41:36 +0100, Marco A. Calamari [EMAIL PROTECTED] wrote: the fact that [FIND] is still the one of 24/12/2004 is due I'm using the stable Freenet ? No. FIND is a DBR. If you can load it, then it's been inserted very recently. 23/12/2004 is the date of the proprietor's most recent comment. It's just that he hasn't found reason to comment since then. Many thanks for your answer, but the site say index generated 2004 12 22. Of course it is inserted every day, but seems to me unmantained since then. That is correct. Sonax, maintainer of FIND, has been having problems with the spider used to generate the index, and hasn't had the time or inclination to fix it. He's still inserting the site daily, but the actual index data has not been updated in a couple of months now. I've offered to help him resolve whatever issues are involved, but he says he just doesn't have the time or motivation to do it at this time. Perhaps at some (hopefully near) future date. SOmenone haave suggestion about the HTL to unse for insertion in both stable and unstable Freenet ? I use 25, and let the network reduce it as it sees fit. MaxHTL (what the network reduces HTL to) is something like 20 these days. I had positive results inserting with htl=6 in stable. Someone else would likely have something better to say about that. I really hope so ;) Ciao. Marco I follow the advice given by FIW (the Freesite Insertion Wizard), i.e., for DBR sites, use a lower HTL (say, 15) than for an edition site or a one-shot site (where you may want to use, say, 25). This makes sense, and seems to work well enough. The reasoning behind this is that the more frequently a site's data is updated, the less need for very deep insertions, as much of the data will be unchanged from one insert to the next, therefore a certain amount of redundancy is involved, resulting in an automatic reinforcement of the data within the network. Less frequently inserted data, on the other hand, basically only gets one chance to take, and will benefit from the deeper insertion (God, this sounds dirty, doesn't it?), distributing the data as far as possible into the network initially, helping it to later disseminate to other nodes more easily. Of course, there are other factors to consider as well. Some of the more popular edition/one-shot sites may be accessed much more frequently than some DBR sites, thereby helping the data to propagate throughout the network, whereas some less popular DBR sites may actually benefit from deeper insertion. It's not a perfect science; there are really no hard-and-fast rules. Just common sense and good judgement, basically, combined with how long you're willing to wait for your inserts to complete. :-) HTH -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Freenet and Solaris 10
broadband connection has either a 128kbps or a 256kbps uplink, this equates to 16kB/sec and 32kB/sec respectively. You will need to keep some bandwidth back for other apps and for downloads (yes, downloading uses a small amount of upload bandwidth). We suggest therefore limits of 12000 for a 128kbps upload connection, or 24000 for a 256kbps upload connection. Most broadband connections have far more download bandwidth than upload bandwidth... just because you have 1Mbps download, does not mean you have 1Mbps upload; if you do not know what your connection's upload speed is, use one of the above options. outputBandwidthLimit [12288] Setting: averageInputBandwidthLimit If nonzero, specifies an independent limit for incoming data only (averaged over a week). (overrides averageBandwidthLimit if nonzero) averageInputBandwidthLimit [0] Setting: averageOutputBandwidthLimit If nonzero, specifies an independent limit for outgoing data only (averaged over a week). (overrides bandwidthLimit if nonzero) averageOutputBandwidthLimit [0] Setting: logLevel The error reporting threshold, one of: Error: Errors only Normal: Report significant events, and errors Minor: Report minor events, significant events, and errors Debug: Report everything that can be reported logLevel [normal] Setting: mainport.params.servlet.7.params.sfDefaultSaveDir Default folder to save large downloaded files to. Defaults to a folder called freenet-downloads in your home directory. mainport.params.servlet.7.params.sfDefaultSaveDir [/export/home/bob/freenet-downloads] Sun java detected. Sun Java 1.4.2 detected. start-freenet.sh: test: unknown operator == $ Any idea what is going wrong? ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED] -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: What's up with the mailing lists?
On Fri, 14 Jan 2005 00:40:28 -0600, Conrad J. Sabatier [EMAIL PROTECTED] wrote: There's something very peculiar going on here. I've only received a handful of messages in a single list (devl) since subscribing. Just out of curiosity, I went back to the mailman interface and resubscribed to all the lists. I got a message back in from only two of the lists saying I was already subscribed. Very odd. Err...nevermind. It was a stupid procmail rule malfunction. :-) Sorry for the useless noise. -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: ANNOUNCE: DFI (Dolphin's Freenet Index) is now back online!
On Fri, 14 Jan 2005 05:59:56 -0800, Sonax [EMAIL PROTECTED] wrote: Date: Thu, 13 Jan 2005 17:37:27 -0800 From: Todd Walton [EMAIL PROTECTED] Subject: Re: [freenet-support] ANNOUNCE: DFI (Dolphin's Freenet Index) is now back online! On Wed, 12 Jan 2005 01:06:46 + (UTC), Conrad J. Sabatier [EMAIL PROTECTED] wrote: [EMAIL PROTECTED]/DFI// Since Fillament has decided to leave Freenet, and give up maintaining YoYo, maybe YoYo ought to be replaced with DFI. It wouldn't matter until there's a new build, of course, but the change could be made now. -todd I would be honored, for sure! :-) By the way, Toad, now's a good time to tell you how mightily impressed I am at how quickly my node got up to speed. In just a couple of days, my spider's database had pretty well fleshed itself out with all the currently available links. Nice! If you read Fillament's last entry he plans to hand over the YoYo privkey to anyone who get's yoyo running (he published the yoyo software). I plan to (with fillaments help when i catch up with him on IIP), to try and get YoYo running again. So don't write off YoYo as dead just yet. I've been wondering what's up with YoYo; it's been pretty funky since I came back online. I'm not saying that DFI should not be on the web-interface, but it should not take YoYo's place. If it should replace anything make it replace FIND (which is just a DFI clone anyway), or CofE. (If i get YoYo running there is no need for two sites on the web- interface to go down when/if i mess up, they get to me, i go on holiday, my hardware blows up etc.) Heh. :-) Also, consider the fact that when the 0.7 starts to become somthing that can be run, you might get two split networks (again), and my guess is that dolphin will want to play with the big boys over at unstable. I can't talk for dolphin, but that would be my guess. Leaving stable without a updated spidered index (again) would be a bad move. Actually, funny you mention that. I just enabled the dual-network hack on my node yesterday and seeded it with a bunch of stable and unstable nodes. So there's a good possibility that it could be a useable index for both branches. But who says it should replace anything? Is there some magic limit on the number of sites on the web-interface? Why not just add DFI (and maby mr.X index also)? PS. Conrad, great to see you around again! :) Thanks. It's nice to be back. I'd forgotten how much I enjoyed tinkering with this stuff. :-) If you have not already, you might wanna look into what Hopekiller has done with your spider. No need for you to fix things he has already fixed (or the other way around). [EMAIL PROTECTED]/hopeless-index/17//files/spider.r ar Yes, I noticed that somewhere (on your site, I think). I'll have to check it out. Right now, I'm mainly preoccupied with getting all of the spider's database files up to snuff (mainly categorizing all the new sites that have come up while I was away), and also with tuning my node. I've been poring over Sun's docs, and I'm experimenting right now with running the JVM using FreeBSD's 1:1 thread mapping library (libthr.so) in conjunction with some of the more esoteric JVM command line options. Interesting stuff! :-) -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Either the lists are really dead today or...
On Fri, 14 Jan 2005 03:39:21 -0600, S [EMAIL PROTECTED] wrote: On Fri, 14 Jan 2005 00:05:52 -0600 Conrad J. Sabatier [EMAIL PROTECTED] wrote: Is anyone else having problems with the lists? Here is what showed up on my end for devel only, local January 13th: mysql SELECT senders.sender, subjects.subject FROM messages INNER JOIN senders ON messages.sender = senders.id INNER JOIN subjects ON messages.subject = subjects.id WHERE (date = 1105596000 AND date = 1105682399);+--+- -+| sender | subject | +--+- -+| Constantine Dokolas cdokolas:sunsoftgr.com | Re: [freenet-dev] Re: Freenet 0.7 | | Toad toad:amphibian.dyndns.org | Re: [freenet-dev] Re: Freenet 0.7| | Toad toad:amphibian.dyndns.org | Re: [freenet-dev] Re: Freenet 0.7| | Robert roberts83:gmail.com | [freenet-dev] bug report | | \Conrad J. Sabatier\ conrads:cox.net | [freenet-dev] CVS privs?| | conrads:cox.net (Conrad J. Sabatier) | [freenet-dev] Re: ANNOUNCE: DFI (Dolphin\'s Freenet Index) is| | \Conrad J. Sabatier\ conrads:cox.net | [freenet-dev] Another performance-enahancing command line option | | \Conrad J. Sabatier\ conrads:cox.net | Re: [freenet-dev] Another performance-enahancing command line| +--+- -+ 8 rows in set (0.09 sec) There were some posts from tech and support also. I don't put those into a database, though, and I really don't feel like grokking date headers to see which ones truly showed up during the 24 hours that I call yesterday.. :) FWIW, up until the discussion about 0.7, the lists haven't been nearly as active over the past few months as they were when you left. Welcome back, by the way! Ah, OK. I remember a time when these lists were just teeming with activity (not sure if that was a good or a bad thing). :-) As I posted in another thread, it turned out to be a misconfiguration in my .procmailrc. :-) Thanks for the welcome back. :-) -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] ANNOUNCE: DFI (Dolphin's Freenet Index) is now back online!
[EMAIL PROTECTED]/DFI// I just completed my first successful insert of DFI today since returning to freenet just yesterday. Please try using it so that it will begin to propagate again as quickly as possible. Due to the fact that I'm just starting back with freenet, the index is still rather smallish, but we should be seeing it grow with each passing day as my node's routing improves and the spider successfully locates more and more sites. Please try to bear with us during this early phasing back in of the site. Thanks! -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: jcpuid and jbigi
In article [EMAIL PROTECTED], Anders Bruun Olsen support@freenetproject.org wrote: Hi, I have just installed Freenet on my Gentoo machine and have noticed that after it running for a while the java processes start to use quite alot of CPU. In /var/freenet/freenet.stderr.log I get these lines which I think might be part of the reason why it is so slow: ERROR: Resource name [freenet/support/CPUInformation/libjcpuid-x86-linux.so] was not found WARN: Native CPUID library jcpuid not loaded - will not be able to read CPU information using CPUID NOTICE: Resource name [net/i2p/util/libjbigi-linux-none.so] was not found INFO: Native BigInteger library jbigi not loaded - using pure java I can see that both jcpuid and NativeBigInteger is in the Contrib module in CVS, and at least compiling libjcpuid-x86-linux.so isn't a problem, but how do I get freenet to actually use that, and how do I get jbigi compiled and get freenot to use that as well? And will it actually help with my CPU usage/speed problem? I am on an Athlon XP 2600+ with 512Mb RAM, so it should be sufficient I would think. Thanks! -- Anders See the hints/caveats below before attempting the following procedure. 1) cd Contrib/NativeBigInteger 2) sh build.sh 3) cd ../jcpuid 4) sh build.sh 3) cd ../../freenet 4) remove any existing freenet.jar/freenet-ext.jar under your freenet source tree's build dir 5) ant (this will create new jars with the new libraries included) 6) copy the resulting jar files to wherever it is you run freenet from Ta-da! That's all, folks! A few caveats/hints, however: Some of the build scripts under Contrib may require some editing (they did for me when I was building the libs under FreeBSD). You can save a *lot* of time building the integer libraries if you edit the build.sh script and change the following line to only build for the desired CPU (if you're running 64-bit Java on the AMD 64, use athlon64): for x in none pentium pentiummmx pentium2 pentium3 pentium4 k6 k62 k63 athlon Also, you may need to add some environment variable settings to the lines in the script that invoke libgmp's configure script, namely, CPPFLAGS=-I/path/to/your/java/includes. Until I did this, the build kept failing to find jni.h at a certain point. For example, you'd want to change the line in the build.sh script from: ../../gmp-4.1.3/configure to CPPFLAGS=-I/path/to/your/java/includes ../../gmp-4.1.3/configure If you need any more help, let us know. Hope this helps. -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: jcpuid and jbigi
In article [EMAIL PROTECTED], Toad support@freenetproject.org wrote: -=-=-=-=-=- -=-=-=-=-=- We need the athlon64 binaries in the freenet-ext.jar. Are they long mode? I imagine they would be. :-) However, we still don't have native Java support for AMD 64 under FreeBSD. In fact, we're still waiting for the jdk-1.5 port for *any* architecture (rumored to be near completion), so I'm left with no choice at this time but to run the jdk-1.4.2 port on my Athlon (i386) box. I could try building the libs under AMD 64, just for the hell of it, but they won't do anybody any good at this time, I'm afraid. Incidentally, did you happen to see my request to have my CVS privs reinstated? There *are* some real issues with some of the scripts under Contrib on non-Win/non-Linux platforms that I'd like to try and clear up for anyone who may try to build on some other platform(s). I'm also *very* interested in getting involved in the major overhaul of the actual daemon code. So, whaddya say, huh? :-) Incidentally, I re-subscribed to all the lists, but something's not working right. I'm seeing messages here on gmane that I have yet to see in my mailer. Is there a known problem with the lists at this time? -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: ANNOUNCE: DFI (Dolphin's Freenet Index) is now back online!
In article [EMAIL PROTECTED], Alex R. Mosteo support@freenetproject.org wrote: Conrad J. Sabatier wrote: [EMAIL PROTECTED]/DFI// I just completed my first successful insert of DFI today since returning to freenet just yesterday. Please try using it so that it will begin to propagate again as quickly as possible. Due to the fact that I'm just starting back with freenet, the index is still rather smallish, but we should be seeing it grow with each passing day as my node's routing improves and the spider successfully locates more and more sites. Please try to bear with us during this early phasing back in of the site. Thanks! The 5100 build is working quite well for me, and today I've tried a bunch of random sites in TFE and all of them have appeared in a very short time. It will be interesting to see how a crawler performs starting from scratch. I'm running from the CVS repo, myself (HEAD). The spider has already picked up quite a few sites, more than double when it started. Not bad! -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Re: ANNOUNCE: DFI (Dolphin's Freenet Index) is now back online!
In article [EMAIL PROTECTED], Someone support@freenetproject.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Conrad J. Sabatier schrieb: | [EMAIL PROTECTED]/DFI// | | I just completed my first successful insert of DFI today since returning to | freenet just yesterday. Please try using it so that it will begin to | propagate again as quickly as possible. | | Due to the fact that I'm just starting back with freenet, the index is still | rather smallish, but we should be seeing it grow with each passing day as my | node's routing improves and the spider successfully locates more and more | sites. | | Please try to bear with us during this early phasing back in of the site. | Thanks! | It's not coming through, what HTL did you use to insert? Ack! I already had one failed (automated, scheduled) insert, due to my falling asleep and the node going off the deep end for some reason. :-) So the site wasn't ready at the time of the DBR rollover, unfortunately. It should be better now. Got it inserted on time today (at HTL 15). -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Either the lists are really dead today or...
...there's something wrong with the list server. I've only received a total of five messages today in all of my freenet mailing lists (I'm subscribed to all of the lists). All of them were in the devel list, and out of those five, three were messages I posted myself. Is anyone else having problems with the lists? -- Conrad J. Sabatier [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] What's up with the mailing lists?
There's something very peculiar going on here. I've only received a handful of messages in a single list (devl) since subscribing. Just out of curiosity, I went back to the mailman interface and resubscribed to all the lists. I got a message back in from only two of the lists saying I was already subscribed. Very odd. -- Conrad J. Sabatier -- [EMAIL PROTECTED] -- In Unix veritas ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] Why are so few people upgrading unstable lately?
Build 6449 has been out for several days, 6450 just became available yesterday, yet my routing table still has quite a few nodes running 6448. What's up with that? -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: Freenet haiku errmm
On Wed, Jan 21, 2004 at 04:05:08AM -0600, S wrote: On Tue, 20 Jan 2004 21:58:08 -0600 Conrad J. Sabatier [EMAIL PROTECTED] wrote: On Tue, Jan 20, 2004 at 11:33:58PM +, Toad wrote: What happens when you try to insert? The usual very very very long verification times? Just thought I'd chime in on this subject. Most recently, I'm finding inserts are working much better. Are you using FIW, and if so, have you noticed any improvements since 0.08 came out? Yes, I'm using fiw, latest version. I *think* 0.08 is a little faster, in that it doesn't seem to error out as much doing the verifications (although this could be something entirely outside of fiw, too). I have a DBR site which I quit bothering to attempt insertions for late last year. I picked back up recently, and then found FIW 0.08 on Monday, but I see no difference. Half an hour or more to successfully insert a 5KB index.html plus the slot (the rest get skipped since they're unchanged) with HTL 15... I've inserted DFI recently in as little as about 20 minutes. Granted, usually the only things that need updating are the actual index.html and the DBR. Still, for about 370K of insert data, that's not bad. Even at that, the insert doesn't actually complete, I get a status of Done for inserting the slot and then I quit FIW without waiting for it to verify the insertion. If I leave it running, it'll sit there forever on Getting with tons of RouteNFs, though the site is accessible through my fproxy. I'm seeing a lot less of these lately. Can't tell whether it's a problem with FIW or with the network, but I used to be able to insert the whole site in under 5 minutes (thinking back to September or October). Now it takes half an hour just to insert the index.html and the slot. Granted I haven't been trying very often, but it seems like insertion is a real problem lately. I'm seeing pretty much just the opposite here. :-) I also have noticed that the number of keys in my datastore isn't increasing very much anymore. I've been hovering between 39900 and 40100 keys for several days now (when the node dies or I restart, I typically lose some key count due to whatever was stored in the temporary files). I last nuked the datastore on November 26, so I've accumulated ~40K keys in less than two months, bringing my store to about 60% full. Then all of a sudden the datastore hasn't grown by 200 persistent keys in several days. Seems to me either nobody's inserting anything, or nobody's requesting anything new, or maybe everyone's inserting and requesting as usual but the data isn't getting propagated. One thing of interest is that in Frost, on Monday I got a whole bunch of troll messages (Oprah Winfrey everywhere) but I haven't seen much since. It's as if nothing's getting through, or perhaps nobody's posting anything, or perhaps people are posting but nobody's able to see it, which takes me back to I don't know what the problem is :) BTW this experience is all on stable, I know you're inserting from unstable so I'll have to give it a shot there and see how things turn out. What HTL are you using to insert? I'm inserting with HTL=25. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: Freenet haiku errmm
On Wed, Jan 21, 2004 at 09:27:06AM +0100, [EMAIL PROTECTED] wrote: I would be very happy if you can publish a complete listing of tweaking you've done to your freenet.conf to optimize insertion; your complete reasoning doing each change would be very interesting too Well, I haven't really done any tweaks with insertions in mind specifically, just trying to smooth out overall performance. I finally realized recently that I was running too big of a node, i.e., too many threads, too many connections, etc. I'm finding now that with the smaller settings, I'm seeing much more consistent and reliable performance overall, with far fewer of those long periods of sluggishness/unresponsiveness, which I think, in the long run, makes for a better-contributing node to the rest of the network, even if it can't handle as many requests. Some of the settings I'm using are only applicable to unstable, by the way. If you're really interested, I'll try and followup later with more specifics. I just really don't have that much time at the moment. :-) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] Re: Freenet haiku errmm
On Tue, Jan 20, 2004 at 11:33:58PM +, Toad wrote: What happens when you try to insert? The usual very very very long verification times? Just thought I'd chime in on this subject. Most recently, I'm finding inserts are working much better. I have DFI's insert scheduled to begin two hours before its DBR rollover, just to be on the safe side. In the past, this was absolutely necessary, and often the insert would only complete just in time, or even late. This hasn't been the case at all the last several days. I'm seeing the inserts finish anywhere from around a half-hour to an hour early now. The node is working beautifully, I must say. It seems the combination of my recent JDK update and reconfiguring the node with more conservative limits has done a world of good. CPU load (as reported in top) is now hovering around 20-30% most of the time, the node is doing very little rejecting, and finding lots of stuff that it hadn't in a while. Definitely positive progress going on here. :-) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] DFI being inserted late today -- sorry!
I upgraded my Sun JDK today via FreeBSD's ports collection. Building the entire beast from source. Took longer than expected. I probably should have just killed everything else instead of letting fred and other stuff run as usual the whole time. Anyway, it's finally done, fred was just restarted, and once it settles down from its startup shakedown, I'll be inserting DFI. Just wanted to let everyone know, so you wouldn't be suspecting bad routing or whatever. :-) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] Oopsie
Sorry about DFI not being inserted earlier today as usual. Last night, just out of curiosity, I added -Xprof to the script that generates the new index and forgot to remove it later. As a result, the scheduled update run today got totally bogged down and nothing ever did get inserted. Rectifying that right now. My apologies. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
Re: [freenet-support] Re: DFI -- new and old keys
On Tue, Dec 30, 2003 at 12:11:45PM +0100, news.gmane.org wrote: Great, but can you please attach the both keys as a text file because gmane screwed up everything :-) I've attached the original key to this mail. Hopefully, the new one isn't really needed. I must confess, I've been remiss in inserting the new key the last few days. It seems like the original has settled back down to normal (at least as far as I can tell from here). Have you been able to retrieve it (the original DFI) successfully lately? Inserts have been decidedly slow and difficult the last few days. Not sure what the problem is (probably the fact that I'm running the latest unstable). :-) But they *do* eventually succeed. I just hope it's propagating OK. Thanks! BTW, DFI is a great index! Gee, thanks! :-) My main goal with DFI was to provide an index that's continually updated and fast to load (hence the very limited graphics). TFE and YoYo are great, but too static and too slow (what with all the activelinks) for my tastes. I also wanted an index that didn't have tons of dead links or links to things that weren't html pages (has anyone noticed that if you access FreenetHelp, your browser starts downloading fmb.jar in the background?). :-) And incidentally, if the author of colours is reading this, could you *please* use conventional edition numbering for your inserts? Pretty please? :-) It's hell to keep up with, it really is, which is why I still haven't linked any of them on my site. Can't even tell which is supposed to be the latest edition. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas [EMAIL PROTECTED]/DFI// ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] A couple of shell scripts for maintaining your node's health
Here are a couple of little scripts I put together today. They should both be placed in your main freenet dir to work properly (or adjust the paths yourself in the scripts). The first, addnodes.sh, downloads your noderefs.txt file and then adds any new nodes it finds in it to seednodes.ref. This is a very convenient way to keep both your noderefs.txt and seednodes.ref up-to-date automatically. Just run it as a cron job. The second, refresh_bookmarks.sh, came about as a result of some very unhealthy routing over the last 24 hours or so. I decided it might help to at least try to keep my bookmarks refreshed (most of them were blank on the gateway page). The script launches a sub-process for each of your bookmarks, checking them periodically. The interval between checks is derived from the failureTableTime setting in freenet.conf. You can, if you wish, have it ignore the local datastore, simply by uncommenting a line in the script. Both are pretty straighforward and easy to understand, I think. Hope you'll find them useful. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas addnodes.sh Description: Bourne shell script refresh_bookmarks.sh Description: Bourne shell script ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] DFI -- new and old keys
Well, it looks like I managed to hopelessly screw up DFI. :-) Still trying to get it straightened out, but in the meantime (and possibly/probably for all future uses), I've reinserted it under a new key, which appears to be working normally. It will be great if the original key eventually settles back down to normal, but just in case... Let's not rush into changing the default bookmark just yet; we may just get lucky. :-) I'd appreciate it if you could keep trying both keys for a while and let me know if you notice anything I need to know about. The new one will need some help to get propagated at first, too, of course. Thanks. Old key: [EMAIL PROTECTED]/DFI// New key: [EMAIL PROTECTED]/DFI// -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] Eureka! Saner threads config settings
D'oh! Why didn't I think of trying this before? maximumThreads=0 targetMaxThreads=512 This seems to be rendering the desired behavior with YThreads now, i.e., the factory is still generating threads beyond the target limit as needed, but the target limit *is* being factored in for estimating load. And the overflow threads are not running amok like before (no 2000+ threads). Not bad, not bad at all. :-) -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] unstable.ref contains only stable nodes!
What is going on lately with the noderefs files on the snapshots site? -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] Update re: DFI
I've finally gotten around to adding a NIM feedback form to DFI ([EMAIL PROTECTED]/DFI//), so if anyone has any comments, please feel free. I'm still in the process of refining the spider and its results. One thing I've done recently, which I'm not sure is a good idea or not, is to ignore CHK keys. It's just my feeling that a real freesite should use an SSK. Similarly, KSKs are passed right over as well. If anyone knows why this may be a bad idea, please let me know. Thanks. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas freenet:[EMAIL PROTECTED]/DFI// ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support
[freenet-support] DFI (Dolphin's Freenet Index) has moved
After a lot of trial-and-error (and a few stupid blunders), the spider-based index of unstable sites I've been working on is now looking pretty darn good, if I do say so myself. The new key is: [EMAIL PROTECTED]/DFI// Hopefully, the more people use/bookmark this thing, the better it will work. Still trying to get some of the other search engines to update/add this. If you're the maintainer of one of these, please check out this site and add it to your listings. Thanks. -- Conrad Sabatier [EMAIL PROTECTED] - In Unix veritas ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
