08.01.2010 15:30, Bob Beck writes: > 2010/1/8 Bob Beck <b...@ualberta.ca>: > And what I mean by that Is that I would be willing to put together a similar > trick for this on ftp.openbsd.org as I did for the installer - if > someone was willing > to integrate it into the pkg_add tools.
Maybe it is not polite to answer to this old thread, but I've integrated (somehow) AutoMirrorDiscovery functionality into pkg_add tool, just to prove myself that i can do that without crashing pkg_add functionality. It is very clumsy right now, but it works for me (with some bugs of course). This is how it works (currently there is no knob to shut it up): (/tmp)[213]% sudo pkg_add -d Pinging anga.funkfeuer.at...OK Pinging carroll.cac.psu.edu...OK Pinging filedump.se.rit.edu...OK Pinging ftp-stud.fht-esslingen.de...OK Pinging ftp.arcane-networks.fr...OK Pinging ftp.aso.ee...no response Pinging ftp.belnet.be...OK Pinging ftp.bytemine.net...OK Pinging ftp.ca.openbsd.org...no response Pinging ftp.cc.uoc.gr...no response Pinging ftp.chg.ru...OK Pinging ftp.crans.org...OK Pinging ftp.cs.pu.edu.tw...no response Pinging ftp.cse.buffalo.edu...no response Pinging ftp.das.ufsc.br...OK Pinging ftp.df.lth.se...OK Pinging ftp.dkuug.dk...no response Use of uninitialized value $ret in numeric eq (==) at /usr/libdata/perl5/OpenBSD/AutoMirrorDiscovery.pm line 58. Pinging ftp.duth.gr...no response Pinging ftp.esat.net...OK Pinging ftp.estpak.ee...OK Pinging ftp.eu.openbsd.org...OK Pinging ftp.fmed.uc.pt...no response Pinging ftp.fr.openbsd.org...OK Pinging ftp.freebsdchina.org...OK Pinging ftp.freenet.de...OK Pinging ftp.fsn.hu...OK Pinging ftp.gamma.ru...OK Pinging ftp.heanet.ie...OK Pinging ftp.iinet.net.au...OK Pinging ftp.inet.no...OK Pinging ftp.irisa.fr...OK Pinging ftp.is.co.za...OK Pinging ftp.jaist.ac.jp...OK Pinging ftp.jyu.fi...OK Pinging ftp.kaist.ac.kr...no response Pinging ftp.kddlabs.co.jp...no response Pinging ftp.lambdaserver.com...OK Pinging ftp.mirrorservice.org...OK Pinging ftp.netbsd.se...OK Pinging ftp.nluug.nl...OK Pinging ftp.obsd.si...OK Pinging ftp.openbsd.dk...OK Pinging ftp.openbsd.or.id...OK Pinging ftp.openbsd.org.ar...no response Pinging ftp.openbsd.org...OK Pinging ftp.piotrkosoft.net...no response Pinging ftp.plig.net...OK Pinging ftp.rediris.es...OK Pinging ftp.spline.de...OK Pinging ftp.task.gda.pl...OK Pinging ftp.tcc.edu.tw...OK Pinging ftp.tpnet.pl...OK Pinging ftp.tw.openbsd.org...OK Pinging ftp.udc.es...no response Pinging ftp.ulak.net.tr...OK Pinging ftp.uninett.no...OK Pinging ftp.wu-wien.ac.at...OK Pinging ftp2.fr.openbsd.org...OK Pinging ftp3.usa.openbsd.org...OK Pinging ftp5.usa.openbsd.org...OK Pinging gulus.usherbrooke.ca...no response Pinging mirror.aarnet.edu.au...OK Pinging mirror.cdmon.com...OK Pinging mirror.corbina.net...OK Pinging mirror.hostfuss.com...OK Pinging mirror.iawnet.sandia.gov...no response Pinging mirror.internode.on.net...OK Pinging mirror.pacific.net.au...OK Pinging mirror.planetunix.net...OK Pinging mirror.public-internet.co.uk...no response Pinging mirror.rit.edu...OK Pinging mirror.roothell.org...OK Pinging mirror.switch.ch...OK Pinging mirrors.24-7-solutions.net...OK Pinging mirrors.localhost.net.ar...no response Pinging mirrors.nic.funet.fi...OK Pinging mirrors.ucr.ac.cr...no response Pinging obsd.cec.mtu.edu...OK Pinging openbsd.arcticnetwork.ca...OK Pinging openbsd.bsdforen.de...OK Pinging openbsd.ftp.fu-berlin.de...no response Pinging openbsd.mirror.frontiernet.net...OK Pinging openbsd.mirrors.pair.com...OK Pinging openbsd.mirrors.tds.net...OK Pinging openbsd.noc.jgm.gov.ar...no response (/tmp)[214]% cat /var/db/ftpmirror.cache ftp://mirrors.nic.funet.fi/pub/OpenBSD/4.6/packages/i386/ ftp://ftp.gamma.ru/pub/OpenBSD/4.6/packages/i386/ ftp://ftp.eu.openbsd.org/pub/OpenBSD/4.6/packages/i386/ (/tmp)[228]% echo $PKG_PATH (/tmp)[229]% sudo pkg_add -i fdm No packages available in the PKG_PATH fdm-1.6:tdb-1.0.6p0: ok (1 to go) fdm-1.6: ok Here is the code: /usr/src/usr.sbin/pkg_add/OpenBSD/AutoMirrorDiscovery.pm # ex:ts=8 sw=4: # $OpenBSD$ # # Copyright (c) 2009 Igor Zinovik <zino...@petrsu.ru> # # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. # functionality for automatic fastest ftp mirror discovery use strict; use warnings; package OpenBSD::AutoMirrorDiscovery; use OpenBSD::Paths; use Net::Ping; use Net::FTP; sub discover_ftp_mirrors() { my ($state) = @_; #my $verbose = $state->verbose >= 3; if (-e "/var/db/ftpmirror.cache") { #$state->print("Mirrors cache already exist\n") if $verbose; return 1; } my (@mirrors) = @_; if (-e "/etc/ftp.mirrors") { open my $fh, '<', "/etc/ftp.mirrors" or die("Permirrsion denied"); @mirrors = <$fh>; close $fh; } # Root priveledges are required to do ICMP pings my $p = Net::Ping->new("icmp"); $p->hires(); my (%hosts) = (); foreach my $mirror (@mirrors) { my ($host) = ($mirror =~ /ftp:\/\/([\w\.-]*)\//); chomp $host; print "Pinging $host...";#if $verbose; my ($ret, $duration) = $p->ping($host, 1.0); if ($ret == 1) { print "OK\n";# if $verbose; $hosts{$mirror} = $duration; } else { print "no response\n";# if $verbose; } } $p->close(); my $cmd = OpenBSD::Paths->uname." -r"; my ($rev) = split(/\s+/o, `$cmd`); chomp $rev; $cmd = OpenBSD::Paths->arch." -s"; my ($app_arch) = split(/\s+/o, `$cmd`); chomp $app_arch; # Sort mirrors by response time, fast are closer to the beggining my @ftp_mirrors = sort {$hosts{$a} cmp $hosts{$b}} keys %hosts; open my $fh, '>', "/var/db/ftpmirror.cache" or die("Cannot create mirror cache file"); my $pkgpath = "NULL"; my $tries = 0; foreach my $p (@ftp_mirrors) { # Check only 3 available mirrors if ($tries gt 2) { last; } chomp $p; my ($ftp_host) = ($p =~ /ftp:\/\/([\w\.-]*)\//); my ($ftp_dir) = ($p =~ /ftp:\/\/$ftp_host\/([\w\.-_]*)/); # Check mirror availability my $ftp = Net::FTP->new($ftp_host, Passive=>1) or next; if ($ftp->login('anonymous', '-anonymous@')) { $ftp_dir .= $rev."/packages/".$app_arch; if ($ftp->cwd($ftp_dir)) { # Seems that this mirror contains packages we need $pkgpath = $p.$rev."/packages/".$app_arch."/"; print $fh "$pkgpath\n"; $tries++; } $ftp->quit; next; } } close $fh; if ($pkgpath eq "NULL") { die("Seems that all mirrors are unavailable"); } return 0; } 1; Index: Makefile =================================================================== RCS file: /OpenBSD/src/usr.sbin/pkg_add/Makefile,v retrieving revision 1.59 diff -u -r1.59 Makefile --- Makefile 19 Jan 2010 14:26:24 -0000 1.59 +++ Makefile 10 Feb 2010 21:03:39 -0000 @@ -14,6 +14,7 @@ OpenBSD/Add.pm \ OpenBSD/AddDelete.pm \ OpenBSD/ArcCheck.pm \ + OpenBSD/AutoMirrorDiscovery.pm \ OpenBSD/CollisionReport.pm \ OpenBSD/Delete.pm \ OpenBSD/Dependencies.pm \ Index: pkg_add =================================================================== RCS file: /OpenBSD/src/usr.sbin/pkg_add/pkg_add,v retrieving revision 1.471 diff -u -r1.471 pkg_add --- pkg_add 26 Jan 2010 11:31:08 -0000 1.471 +++ pkg_add 10 Feb 2010 21:03:39 -0000 @@ -23,6 +23,7 @@ use OpenBSD::AddDelete; package OpenBSD::AddDelete; +use OpenBSD::AutoMirrorDiscovery; use OpenBSD::Dependencies; use OpenBSD::PackingList; use OpenBSD::PackageInfo; @@ -858,10 +859,10 @@ -our ($opt_a, $opt_A, $opt_P, $opt_Q, $opt_r, $opt_u, $opt_U, $opt_l, $opt_z); +our ($opt_a, $opt_A, $opt_d, $opt_P, $opt_Q, $opt_r, $opt_u, $opt_U, $opt_l, $opt_z); -handle_options('aqchruUzl:A:P:Q:', {}, - 'pkg_add [-acIinqrsUuvxz] [-A arch] [-B pkg-destdir] [-D name[=value]]', +handle_options('adqchruUzl:A:P:Q:', {}, + 'pkg_add [-adcIinqrsUuvxz] [-A arch] [-B pkg-destdir] [-D name[=value]]', '[-L localbase] [-l file] [-P type] [-Q quick-destdir] pkg-name [...]'); local $SIG{'INFO'} = sub { $state->status->print($state); }; @@ -906,6 +907,12 @@ $state->{allow_replacing} = $opt_r || $opt_u || $opt_U; $state->{hard_replace} = $opt_r; $state->{newupdates} = $opt_u || $opt_U; + +if (defined $opt_d) { + OpenBSD::AutoMirrorDiscovery->discover_ftp_mirrors($state); + exit(0); +} + if (@ARGV == 0 && !$opt_u && !$opt_l) { Usage "Missing pkgname"; Index: OpenBSD/PackageLocator.pm =================================================================== RCS file: /OpenBSD/src/usr.sbin/pkg_add/OpenBSD/PackageLocator.pm,v retrieving revision 1.87 diff -u -r1.87 PackageLocator.pm --- OpenBSD/PackageLocator.pm 10 Jan 2010 11:32:41 -0000 1.87 +++ OpenBSD/PackageLocator.pm 10 Feb 2010 21:03:39 -0000 @@ -30,14 +30,32 @@ if (!defined $default_path) { $default_path = OpenBSD::PackageRepositoryList->new; + my $v; if (defined $ENV{PKG_PATH}) { - my $v = $ENV{PKG_PATH}; + $v = $ENV{PKG_PATH}; $v =~ s/^\:+//o; $v =~ s/\:+$//o; while (my $o = OpenBSD::PackageRepository->parse(\$v)) { $default_path->add($o); } } else { + if (-e "/var/db/ftpmirror.cache") { + my (@mirrors) = @_; + open my $fh, '<', "/var/db/ftpmirror.cache" or + die("Permirrsion denied"); + @mirrors = <$fh>; + close $fh; + foreach my $m (@mirrors) { + chomp $m; + $v .= ":$m"; + } + } + $v =~ s/^\:+//o; + $v =~ s/\:+$//o; + while (my $o = OpenBSD::PackageRepository->parse(\$v)) { + $default_path->add($o); + } + $default_path->add(OpenBSD::PackageRepository->new("./")); }
