Revision: 764
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=764
Author: ineiev
Date: 2024-02-11 09:26:31 -0500 (Sun, 11 Feb 2024)
Log Message:
-----------
update
Modified Paths:
--------------
trunk/sviki/Mirroring.mdwn
Modified: trunk/sviki/Mirroring.mdwn
===================================================================
--- trunk/sviki/Mirroring.mdwn 2024-02-02 09:27:29 UTC (rev 763)
+++ trunk/sviki/Mirroring.mdwn 2024-02-11 14:26:31 UTC (rev 764)
@@ -80,9 +80,18 @@
The HTTPS CGI configuration for `gnu mirror` is:
location / {
+ gzip off;
+
+ # In this case, the ENTIRE path is passed on the to cgi script, e.g.
+ # given http://ftpmirror.sv.gnu.org/datamash/datamash-1.1.0.tar.gz we
pass
+ # '/datamash/datamash-1.1.0.tar.gz' to the CGI script.
+ # So there is no need to use 'fastcgi_split_path_info' command.
fastcgi_param PATH_INFO $uri;
- fastcgi_param MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
- fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+
+ fastcgi_param MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/gnu.txt;
+ fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
+ fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
+ fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
@@ -100,32 +109,53 @@
# Actual redirection mechanism.
location /releases-redirect {
+ gzip off;
+
+ # This command tells NGINX how to extract the path info part
+ # from the requested URL: it trims the 'releases-redirect' part,
+ # leaving only what follows it.
fastcgi_split_path_info ^(/releases-redirect)(/?.+)$;
+
+ # This command takes NGINX's path_info variable (extracted above),
+ # and sends it to the FastCGI daemon as the PATH_INFO environment
variable.
fastcgi_param PATH_INFO $fastcgi_path_info;
- fastcgi_param MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
- fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+
+ fastcgi_param MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/download.txt;
+ fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
+ fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
+ fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
+
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
-The HTTPS versions substitute *mirrors.txt with *mirrors-https.txt
+The HTTPS versions substitute gnu.txt and download.txt with gnu-https.txt
+and download-https.txt
-The mirror list text files (`gnu-ftpmirror.txt` and `download-mirrors.txt`)
-contain an auto-generated list of active mirrors:
+The mirror list text files (`gnu.txt` and `download.txt`)
+contain an auto-generated list of active mirrors and region redirections:
- # head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt
- http://mirror.tochlab.net/pub/gnu/ ru
- http://mirror.cedia.org.ec/gnu/ ec
- http://www.gutscheinrausch.de/mirror/gnu/ de
- http://www.artfiles.org/gnu.org/ de
- http://gnu.mirrors.linux.ro/ ro
+ # head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
+ http://us.mirrors.cicku.me/gnu/ us
+ http://mirror.freedif.org/GNU/ sg
+ http://fosszone.csd.auth.gr/gnu/ gr
+ http://mirror.marwan.ma/gnu/ ma
+ http://ftp.task.gda.pl/pub/gnu/ pl
+ # tail -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
+ ye az
+ yt za
+ za za
+ zm za
+ zw za
+(Yemen [YE] uses the mirrors located in Azerbaijan [AZ], Zimbabwe [ZW] uses
+the mirrors located in South Africa [ZA].)
+
See 'mirmon' section below to learn how these files are generated.
-The Perl script `mirror-redirect.pl` uses
-[Geo::IP](https://metacpan.org/pod/Geo::IP) and `Geo::Mirror` modules
-to read the mirror file, detect the closest region based on the user's
-IP, and return an HTTP redirect to that mirror.
+The Perl script `mirror-redirect` uses
+[Geo::IP](https://metacpan.org/pod/Geo::IP) module
+to detect users' regions based on their IP.
## Server-side Mirmon overview
@@ -161,7 +191,7 @@
-rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt
lrwxrwxrwx 1 28 Feb 20 23:28 nongnu-mirror-list.txt ->
/srv/download/00_MIRRORS.txt
- The script `/opt/savannah/mirrors/scripts/gnu-mirmon.pl`
+ The script `/opt/savane/bin/sv_gnu-mirmon`
fetches the gnu list and saves it locally in the two versions
(`gnu` and `allgnu`). The `nongnu` list is symlinked to the
manually-managed
file.
@@ -191,22 +221,20 @@
A 'state' file contains the status for each mirror:
$ head -n5 /var/lib/mirmon/state-gnu
- http://ca.mirror.babylon.network/gnu/ 1487646961 ok 1487651999 s
1487651999-s 1487651999
- http://download.nust.na/pub/ftp.gnu.org/gnu/ 1487643361 ok 1487651999
s 1487651999-s 1487651999
- http://fr.mirror.babylon.network/gnu/ 1487643361 ok 1487651999 s
1487651999-s 1487651999
- http://ftp-gnu-org.ip-connect.vn.ua/ 1487625361 ok 1487651999 s
1487651999-s 1487651999
- http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1487556961 ok 1487651999 s
1487651999-s 1487651999
+ http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.gnu.org/ 1707624961 ok
1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721
+ http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1707617761 ok 1707654121
ssssssssssssss 1707626399-ssssssssssssss 1707654121
+ http://ftp.cc.uoc.gr/mirrors/gnu/ 1707581761 ok 1707654121
ssssssssssssss 1707626399-ssssssssssssss 1707654121
+ http://ftp.eq.uc.pt/software/unix/gnu/ 1707621361 ok 1707657721
ssssssssssssfs 1707628921-ssssssssssssss 1707657721
+ http://ftp.fau.de/gnu/ 1707646561 ok 1707657721 ssssssssssssss
1707632521-ssssssssssssss 1707657721
5. Two auxiliary scripts
`/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh`
- read the state files, and generate simple text files
- containing only the active and up-to-date mirrors:
+ invoke sv_mirmon-to-geip; that script reads the state file and the mirror
+ list and generates simple text files containing the active and up-to-date
+ mirrors and region redirections:
- $ ls -1 /opt/savannah/mirrors/active-mirror-lists/
- download-mirrors-https.txt
- download-mirrors.txt
- gnu-ftpmirrors-https.txt
- gnu-ftpmirrors.txt
+ # ls /opt/savannah/mirrors/active-mirror-lists/
+ download-https.txt download.txt gnu-https.txt gnu.txt
These are the four files used in the CGI scripts (see above section).
@@ -214,8 +242,8 @@
`/etc/cron.d/sv-mirmon`:
# Update the authoritative list of GNU ftp mirrors (mirroring
ftp.gnu.org).
- 3 1 * * * root /opt/savannah/mirrors/scripts/gnu-mirmon.pl --http-only
/opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
- 6 1 * * * root /opt/savannah/mirrors/scripts/gnu-mirmon.pl --all
/opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
+ 3 1 * * * root sv_gnu-mirmon --http-only
/opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
+ 6 1 * * * root sv_gnu-mirmon --all
/opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
# Check the status of each mirror (and regenarate the HTML report
page).
# For gnu/nongnu - also update the list of active mirrors (for the
mirror CGIs).
@@ -229,22 +257,26 @@
## Diagnostics and Troubleshooting
1. The three mirmon HTML reports are viewable here:
- <http://download0.savannah.gnu.org/mirmon/>.
+ <http://download.savannah.gnu.org/mirmon/>.
-2. An additional CGI script exercise the Geo::IP
- module and the generated files:
- <http://download0.savannah.gnu.org/mirror-check/> and
- <http://ftpmirror0.savannah.gnu.org/mirror-check/>.
+2. An additional page is used for testing:
+ <http://download.savannah.gnu.org/mirror-check/>,
+ <http://ftpmirror.savannah.gnu.org/mirror-check/>,
+ <https://download.savannah.gnu.org/mirror-check/>,
+ <https://ftpmirror.savannah.gnu.org/mirror-check/>.
- The script is `/opt/savannah/cgi-bin/mirrors/mirror-check.pl`.
+ The NGINX configurations are `*-common.inc`:
- The NGINX configurations are `ftpmirror-common.inc` and
- `download-common.inc`:
location /mirror-check/ {
- fastcgi_param GNU_MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
- fastcgi_param NONGNU_MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
- fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-check.pl;
+ gzip off;
+ fastcgi_split_path_info ^(/mirror-check)(/?.+)$;
+ fastcgi_param PATH_INFO $fastcgi_path_info;
+ fastcgi_param MIRROR_FILE /path/to/active/mirror/list;
+ fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
+ fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
+ fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
+ fastcgi_param TEST 1;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
@@ -256,16 +288,18 @@
of each hour (defined in the cron file):
# ls -lhog /var/lib/mirmon/
- total 60K
- -rw-r--r-- 1 7.3K Feb 21 20:22 state-gnu
- -rw-r--r-- 1 3.5K Feb 21 20:32 state-nongnu
- -rw-r--r-- 1 16K Feb 21 20:42 state-allgnu
+ total 84K
+ -rw-r--r-- 1 27K Feb 11 09:17 debian-mirror-check.html
+ -rw-r--r-- 1 3.9K Feb 11 09:17 state
+ -rw-r--r-- 1 24K Feb 11 08:42 state-allgnu
+ -rw-r--r-- 1 18K Feb 11 08:22 state-gnu
+ -rw-r--r-- 1 4.2K Feb 11 08:32 state-nongnu
The active mirror list files should have similar timestamp:
# ls -lhog /opt/savannah/mirrors/active-mirror-lists/
- total 16K
- -rw-r--r-- 1 415 Oct 15 16:32 download-mirrors-https.txt
- -rw-r--r-- 1 1.1K Oct 15 16:32 download-mirrors.txt
- -rw-r--r-- 1 626 Oct 15 16:22 gnu-ftpmirrors-https.txt
- -rw-r--r-- 1 2.8K Oct 15 16:22 gnu-ftpmirrors.txt
+ total 24K
+ -rw-r--r-- 1 1.7K Feb 11 08:32 download-https.txt
+ -rw-r--r-- 1 2.1K Feb 11 08:32 download.txt
+ -rw-r--r-- 1 4.4K Feb 11 08:22 gnu-https.txt
+ -rw-r--r-- 1 4.2K Feb 11 08:22 gnu.txt
