Re: to find fastest debian mirror
Bruce Perens [EMAIL PROTECTED] writes: I would suggest that the program create its own temporary directory under /tmp and delete it when it is done, rather than insist that ~/tmp exist. Set your umask so that others do not have write permissions on the directory and its files _before_ you create the directory. Mkdir will fail if any of the security problems exist. Instead of using /tmp, I think it should firast try the environment variable $TMPDIR, and then go to /tmp if that doesn't exist. ($TMPDIR seems to be a fairly standard variable, so why not use it?) That's just a small nit - this looks like a really cool idea. I have done the same thing a few times by hand trying to find the fastest mirror. Now I can check each and every time :-) Later, Dale -- + finger for pgp public key -+ | Dale E. Martin | University of Cincinnati Savant Research Laboratory | | [EMAIL PROTECTED]| http://www.ececs.uc.edu/~dmartin | +--+ -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: to find fastest debian mirror
hi Nice program. thanks Adding the -s 1024 flag to ping would make the packets closer to the size of FTP packets, and make the results more accurate. well, indeed I had done that after sending it I would suggest that the program create its own temporary directory under /tmp and delete it when it is done, rather than insist that ~/tmp exist. Set your umask so that others do not have write permissions on the directory and its files _before_ you create the directory. Mkdir will fail if any of the security problems exist. I follow BUGTRAQ because I would like to learn more about security (that I do not understand) so I understood that creating stuff (either files or dirs) in /tmp is quite a delicate thing : if the script is run by root and the temporary name is easily predictable, then someome in the system can use this to do damage. I have in plan to ask a few questions on the subject; maybe the answer is to use names with an extension like this: /bin/dd if=/dev/random count=1 bs=16 2/dev/null |\ /bin/sed -n l | /usr/bin/tr --delete --complement [:alnum:]_\n this generates a fairly random name I would implement it right now, but it doesn't work perfectly: indeed if you try /bin/dd if=/dev/random count=10 2/dev/null |\ /bin/sed -n l | /usr/bin/tr --delete --complement [:alnum:]_\n you will notice that only a few bytes get thru... why? Please do some AWK work to take the number of dropped packets into account. Speed should probably be something like: reliability = (100 - dropped_packet_percentage) / 100 real_speed = speed * reliability Indeed, because of the re-transmission delay for dropped packets, you may want to increase the penalty for them above this. Thanks ok I will put it in my TODO list I have a few scritps like this maybe in the future I will build a package of them. a.m. -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: to find fastest debian mirror
Andrea Mennucci [EMAIL PROTECTED] wrote: This message is in MIME format. Yuk - as this was normal text, this wasn't necessary. this script takes as its argument the file README.mirrors (that can be downloaded from the debian ftp sites), extracts the list of hosts and pings them all (it takes roughly 30 seconds, since it parallelizes) then sorts the output by fastest time I had some trouble running your script, because `ulimit -u` was set at my system to 32 (for whatever reason), and I ran it under X - so I got lots of fork: try again messages. They seemed to prevent your script from terminating normally. Not knowing (then) how to solve the maximum processes problem, I tried to change your script so it would be configurable how many pings could run in parallel. I added MAX_PINGS=5 to the top, and changed: # we do not want to flood the net; we test 30 sites at a time # then we wait 8 seconds so that most of the pings will be done if [ `expr $n % 30 ` = 0 ] ; then sleep 8 ; fi to: while [ `ps -axu | grep -v grep | grep ping | wc --lines` -gt $MAX_PINGS ] do sleep 1 done This seems to limit the number of pings while not waiting longer than necessary. (Please note that I'm not good at shell scripts; there may be a much better way to do this - it just worked for me). I think it would be fantastic if this script could be made part of dpkg-ftp , (lets say, as an option when in dselect you choose Access and then ftp); It would need some additional work to make it more resistant to problems like the ulimit -u thing, though. E.g., there is an unknown host (debian.med.miami.edu) in README.mirrors, and the ping error message regarding this host should be killed, I think. I also noticed during testing that there is a big difference in results between subsequent runs of the script - perhaps it would be better to make ping run a little longer. Lastly, it would be nice if the script FTPed the README.mirrors file itself (from ftp.debian.org) if not present. Other than the above, I found your script very useful, and I'll save it here - it may be useful to check Simtel, sunsite, etc. mirrors too ;-) Gertjan. --- -- Gertjan Klein [EMAIL PROTECTED] The Boot Control home page: http://www.xs4all.nl/~gklein/bcpage.html -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
Re: to find fastest debian mirror
Nice program. Adding the -s 1024 flag to ping would make the packets closer to the size of FTP packets, and make the results more accurate. I would suggest that the program create its own temporary directory under /tmp and delete it when it is done, rather than insist that ~/tmp exist. Set your umask so that others do not have write permissions on the directory and its files _before_ you create the directory. Mkdir will fail if any of the security problems exist. Please do some AWK work to take the number of dropped packets into account. Speed should probably be something like: reliability = (100 - dropped_packet_percentage) / 100 real_speed = speed * reliability Indeed, because of the re-transmission delay for dropped packets, you may want to increase the penalty for them above this. Thanks Bruce -- Can you get your operating system fixed when you need it? Linux - the supportable operating system. http://www.debian.org/support.html Bruce Perens K6BP [EMAIL PROTECTED] NEW PHONE NUMBER: 510-620-3502 -- TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to [EMAIL PROTECTED] . Trouble? e-mail to [EMAIL PROTECTED] .
to find fastest debian mirror
hello to everybody
I had a strange slowdown on the internet, and
I wanted to locate the fastest debian mirror for me so I wrote a simple
shell script that I call sort_mirrors_by_speed
this script takes as its argument the file README.mirrors
(that can be downloaded from the debian ftp sites), extracts the list
of hosts and pings them all (it takes roughly 30 seconds, since
it parallelizes) then sorts the output by fastest time
I think it would be fantastic if this script could be made
part of dpkg-ftp , (lets say, as an option when in dselect
you choose Access and then ftp);
so the average user could ask dselect to test the mirror speeds and
then choose the fastest for its network from a convenient menu
(of course this means that the file README.mirrors should be shipped as a
part of dpkg-ftp , and maybe updated when Update is chosen)
hope you will find this useful
a.m.
#!/bin/sh
# sort_mirrors_by_speed by A.Mennucci Nov 97
#
# this program is subject to the
# GNU general public license
#
# this program will scan the file in $1 using
# grep \.[a-zA-Z]*: $1 | awk '{print $1}'
# to find mirror site host names.
#
# the string before the : character is considered to be a
# mirror site host name
#
# Mirror sites are pinged and statistics are collected and sorted
#
if [ $1 = -h -o $1 = --help ] ; then
echo Usage: $0 mirrorfile
echo Argument must be the file README.mirrors
echo containing a list of debian mirrors.
echo This program will ping all mirrors with 4 packets
echo to test their speed and reliability, and then will
echo sort the result, put it in file mirrors_by_speed and show the best.
exit 0
fi
tmpdir=~/tmp
# warning: using the directory /tmp and running this script
# as root is a potential security problem
if ! [ -w $tmpdir ] ; then
echo $0 ERROR dir $tmpdir is not writable
echo please create it if it does not exist
exit 1
fi
if [ ! -r $1 -o $1 = ] ; then
echo $0 ERROR Give as argument the file README.mirrors
echo containing a list of debian mirrors
exit 1
fi
if [ -r mirrors_by_speed ] ; then
echo $0 ERROR the file mirrors_by_speed already exists
exit 1
fi
n=0
echo -n Testing
grep \.[a-zA-Z]*: $1 | awk '{print $1}' | \
wc --lines | tr \n
echo debian mirror sites for speed and reliability.
echo -n Tests done :
for i in ` grep \.[a-zA-Z]*: $1 | awk '{print $1}' ` ;
do
n=`expr $n + 1 `
(
h=$n
#echo TEST $h for $i
if ping -c 6 `echo $i | cut -d: -f1 ` $tmpdir/ping$$_$h ; then
grep round-trip $tmpdir/ping$$_$h |\
cut -d/ -f 4 | tr \n $tmpdir/mirrors_by_speed_$h
echo -n ms AVERAGE , $tmpdir/mirrors_by_speed_$h
grep received $tmpdir/ping$$_$h |\
cut -d, -f 3 | tr \n , $tmpdir/mirrors_by_speed_$h
echo SITE $i $tmpdir/mirrors_by_speed_$h
#cat $tmpdir/mirrors_by_speed_$h
cat $tmpdir/mirrors_by_speed_$h mirrors_by_speed
rm $tmpdir/mirrors_by_speed_$h
fi
rm $tmpdir/ping$$_$h
echo -n $h
)
# we do not want to flood the net; we test 30 sites at a time
# then we wait 8 seconds so that most of the pings will be done
if [ `expr $n % 30 ` = 0 ] ; then sleep 8 ; fi
done
wait
echo .
echo Fastest mirrors are:
mv mirrors_by_speed mirrors_by_speed~
sort -n mirrors_by_speed~ mirrors_by_speed
head mirrors_by_speed
