Revision: 331
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=331
Author: agn
Date: 2017-04-18 00:50:30 -0400 (Tue, 18 Apr 2017)
Log Message:
-----------
MailingLists: add new page and update existing pages
Modified Paths:
--------------
trunk/sviki/ListServer.mdwn
trunk/sviki/MailSystem.mdwn
Added Paths:
-----------
trunk/sviki/MailingListsInternals.mdwn
Modified: trunk/sviki/ListServer.mdwn
===================================================================
--- trunk/sviki/ListServer.mdwn 2017-03-04 22:39:37 UTC (rev 330)
+++ trunk/sviki/ListServer.mdwn 2017-04-18 04:50:30 UTC (rev 331)
@@ -1,18 +1,38 @@
-List Server
-===========
+Mailing Lists Server
+====================
+All gnu and nongnu mailing lists are processed on `lists.gnu.org`
+using the Mailman package. This machine is managed by FSF admins.
+Savannah admins have only non-root access and are able to perform
+limited mailing-list related administration tasks.
+
+This page describes commands to manually manage mailing lists
+by savannah administrators. See also:
+
+* [[MailSystem]] - various email and mailing-list related details
+ in Savannah and gnu systems.
+* [[MailingListsInternals]] - the interface between savannah's php
+ frontend and Mailman.
+
+
+General
+-------
+
Some Mailman admin tasks need to be performed on the actual machine
where Mailman is installed, while others, such as changing the password
for a list, can be done via the web interface and the site-wide admin
-mailman password. See [[MailSystem]] for architecture details.
+mailman password.
-In order to do this you need to have shell access to the machine that
-runs the master Mailman installation.
+In order to do this you need to have shell access to `lists.gnu.org` -
+the machine that runs the master Mailman installation. If you require
+such access, ask one of the other savannah admins to add your ssh
+public key.
Once your ssh public key has been added, you can log in as the user
'list'. You will be greeted with the bash prompt:
- [mailman@lists list]$
+ $ ssh l...@lists.gnu.org
+ lists:~ $
From here you can perform all sorts of Mailman administrative tasks,
such as those discussed below. See also: [[ResetListPassword]].
@@ -27,8 +47,8 @@
-------------
Project admins can create their own mailing lists through the normal web
-interface. (What that web interface does is ssh to the list server and
-run `newlist`; see the [[MailSystem]] page.)
+interface. (This results in an ssh connection to `lists.gnu.org`
+and `newlist` being run there; see the [[MailSystem]] page.)
For lists whose names do not follow an allowed pattern, savannah-hackers
must manually run `~/mycreate_list.pl` on lists, and then attach the
Modified: trunk/sviki/MailSystem.mdwn
===================================================================
--- trunk/sviki/MailSystem.mdwn 2017-03-04 22:39:37 UTC (rev 330)
+++ trunk/sviki/MailSystem.mdwn 2017-04-18 04:50:30 UTC (rev 331)
@@ -1,7 +1,7 @@
MailSystem
==========
-See also [[ListServer]].
+See also [[ListServer]] and [[MailingListsInternals]].
The principal GNU (and FSF?) email server is lists.gnu.org, managed by
FSF sysadmin. Savannah people and programs do not have root access.
@@ -36,27 +36,13 @@
- public archives are managed by mharc: `~mharc/mbox` and
`~mharc/html` (check [[ImportMailingListArchive]]).
-Mailing list creation
----------------------
-I (Sylvain) basically installed sv_mailman on internal.sv.gnu.org,
-modified it to support mailman virtual hosts (and use the proper admin
-mail instead of usern...@sv.gnu.org). Then I wrote a fake
-`/usr/sbin/newlist` that connects to l...@lists.gnu.org via SSH and call
-an helper script there (`savannah_wrapper.pl`). This happens via a
-Savannah public key in l...@lists.gnu.org's `authorized_keys` with a
-`command=` restriction.
+Mailing-List interface to Savannah's frontend
+---------------------------------------------
-There is also an empty `/usr/sbin/config_list` on internal that does
-nothing; we don't want what Savane currently does there.
+See [[MailingListsInternals]].
-The cron job runs on internal every five minutes, file
-`/etc/cron.d/sv`. The script being run is named sv_mailman.
-On lists.gnu.org, there is a script `mycreate_list.pl` which can be used
-to create lists by hand. It duplicates what `savannah_wrapper.pl` does
-for Savannah actions. (The scripts must be kept in sync by hand. They
-should be merged so there is only one.)
Valid sender
------------
Added: trunk/sviki/MailingListsInternals.mdwn
===================================================================
--- trunk/sviki/MailingListsInternals.mdwn (rev 0)
+++ trunk/sviki/MailingListsInternals.mdwn 2017-04-18 04:50:30 UTC (rev
331)
@@ -0,0 +1,345 @@
+Mailing Lists Management in Savannah
+====================================
+
+All gnu and nongnu mailing lists are processed on `lists.gnu.org`
+using the Mailman package. This machine is managed by FSF admins.
+Savannah admins have only non-root access and are able to perform
+limited mailing-list related administration tasks.
+
+This page describes the interface between savannah's php frontend
+and the Mailman package running on `lists.gnu.org`. See also:
+
+* [[MailSystem]] - various email and mailing-list related details
+ in Savannah and gnu systems.
+* [[ListServer]] - manually running mailing-list related commands.
+
+
+
+Overview
+--------
+
+Savannah provides mailing lists to hosted projects.
+Project admins can create as many mailing lists as needed through
+the PHP web frontend.
+
+The general workflow is as follows:
+
+1. A project admin on savannah creates a new mailing list.
+2. Mailing list information is updated in the `mail_group_list` table
+ in the `savane` MySQL database.
+3. A cronjob on `mgt0` reads the `mail_group_list` table and determines
+ which operations are needed (e.g. list-creation, password-reset).
+4. The cronjob runs local scripts (on mgt0) for each operation.
+5. Each script proxies the commands over SSH to `lists.gnu.org`
+6. The script `scannah_wrapper.pl` on `lists.gnu.org` processes the commands
+ and runs the appropriate Mailman scripts locally on `lists.gnu.org`.
+
+
+
+Database
+--------
+
+The `mail_group_list` table contains the following fields:
+
+ mysql> select * from mail_group_list where list_name = 'sed-devel' \G
+ *************************** 1. row ***************************
+ group_list_id: 3272
+ group_id: 6711
+ list_name: sed-devel
+ is_public: 1
+ password: NULL
+ list_admin: 829
+ status: 5
+ description: Development discussion for GNU Sed
+
+To see the mailing-lists for a savannah project:
+
+ mysql> select list_name from mail_group_list,groups
+ where groups.unix_group_name = 'coreutils'
+ and groups.group_id = mail_group_list.group_id ;
+ +---------------+
+ | list_name |
+ +---------------+
+ | bug-coreutils |
+ | coreutils |
+ +---------------+
+ 2 rows in set (0.00 sec)
+
+The meaning of the `status` field:
+
+ Status 0: list is deleted (ie, does not exist).
+ Status 1: list is marked for creation.
+ Status 2: list is marked for reconfiguration.
+ Status 5: list has been created (ie, it exists).
+
+The `password` field means:
+
+ NULL: default value
+ '1': A password-reset was requested for this mailing list.
+
+The password field should not contain any real passwords,
+though the database does contain values besides NULL and '1'
+(TODO: find out how/why):
+
+ mysql> select count(*) from mail_group_list where password is not NULL and
password != '1';
+ +----------+
+ | count(*) |
+ +----------+
+ | 1674 |
+ +----------+
+ 1 row in set (0.00 sec)
+
+
+
+PHP web frontend code
+---------------------
+
+Mailing list administration is implemented in
+[./frontend/php/mail/admin/index.php](https://git.savannah.gnu.org/cgit/administration/savane.git/tree/frontend/php/mail/admin/index.php).
+
+The PHP code updates *only* the `mail_group_list` table in the MySQL
+database, and queues future actions by changing the `status` and
+`password` fields in the table.
+
+No other actions are directly performed by the PHP frontend code.
+
+The PHP code contains these additional comments:
+
+ This frontend php script sets status to:
+ 0 if user deletes a list before the backend ever actually created it.
+ 1 if user adds a list
+ 2 if user reconfigures an _existing_ list (ie, status was 5)
+
+ The backend sv_mailman.pl script sets status to:
+ 0 when a list is actually deleted
+ 5 when a list is actually created
+
+ - when we create an alias, which mean someone was able, according to
+ group type restriction to add to his project a list that was already
+ inside the database, we add the list inside the database with a status
+ of 5, so sv_mailman does not try to recreate it.
+ In the worse case, if two persons creates the same list at the same
+
+ The field password will not contact real password, it will contain
+ '1' when the backend is supposed to reset it.
+
+
+Backend cronjob
+---------------
+
+The mailing list cronjob runs on `mgt0` (in the old setup
+before the 2017 migration to newer VM, the script was on `internal` -
+outdated wiki pages might still refer to it there).
+
+In `mgt0:/etc/cron.d/savane`:
+
+ PATH=...:/opt/savannah/bin:...
+ # New list creation
+ */5 * * * * root sv_mailman --cron
+
+The script is `mgt0:/opt/savannah/bin/sv_mailman`.
+
+The above script is the installed version, originating from
+`mgt0:/opt/savannah/savane/backend/mail/sv_mailman.in` (and also in
+git
+<https://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in>).
+If you want to modify the script, NEVER update the installed version
+in `/opt/savannah/bin`. ALAWYS update the source version in
+`/opt/savannah/savane/backend/mail/sv_mailman.in` then run `make
+instal` (and of course, check-in the updated version too).
+
+The script reads the `mail_group_list` MySQL table,
+and performs the required actions based on the `status` and `password`
+fields.
+
+The script runs the following local scripts (on `mgt0`):
+
+- `/usr/sbin/newlist` - creating new mailing list
+- `/usr/sbin/config_list` - configuring a list (currently a no-op)
+- `/usr/sbin/rmlist` - removing a list
+- `/usr/lib/mailman/bin/change_pw` - resetting a mailing list password.
+
+Sadly, the paths are hard-coded in `sv_mailman` (there's even a comment
+about how bad they are).
+
+On `mgt0`, symlinks were created to the actual scripts:
+
+ agn@mgt0:~$ ls -lhog /usr/sbin/{newlist,config_list,rmlist}
/usr/lib/mailman/bin/change_pw
+ lrwxrwxrwx 1 28 Apr 17 22:44 /usr/lib/mailman/bin/change_pw ->
/opt/savannah/sbin/change_pw
+ lrwxrwxrwx 1 30 Apr 17 22:43 /usr/sbin/config_list ->
/opt/savannah/sbin/config_list
+ lrwxrwxrwx 1 26 Apr 17 22:43 /usr/sbin/newlist ->
/opt/savannah/sbin/newlist
+ lrwxrwxrwx 1 25 Apr 17 22:43 /usr/sbin/rmlist -> /opt/savannah/sbin/rmlist
+
+FIXME: remove the hard-coded PATHs from `sv_mailman`, and install an
+updated version.
+
+NOTE: These scripts are proxies to `lists.gnu.org` - they do not
+perform any local action on `mgt0`. Their names are likely left-over
+from the time the entire savannah system (including Mailman)
+was running on the same local machine.
+
+
+SSH Proxing to lists.gnu.org
+----------------------------
+
+The `newlist`/`rmlist`/`change_pw` scripts on `mgt0`
+proxy the needed commands to `lists.gnu.org`,
+where the Mailman package is installed.
+
+1. On `mgt0`, the script `sv_mailman` runs as root from cron-jobs.
+2. The `sv_mailman` runs one of the action scripts (e.g. `newlist`).
+ the script also generates random passwords if needed.
+3. The `newlist` script (on `mgt0`) connets with SSH to `l...@lists.gnu.org`
+ (this happens as user `root@mgt0`).
+4. The corresponding pubkey in `lists.gnu.org:/home/list/.ssh/authorized_keys`
+ is defined as follows:
+
+ command="./savannah_wrapper.pl" ssh-rsa AAAAB3NzaC1yc2...
+
+5. When the `newlist` script from `mgt0` connects with SSH to `lists.gnu.org`
+ instead of getting a shell, the
`lists.gnu.org:/home/list/savannah_wrapper.pl`
+ script is executed.
+6. This scripts parses limited input from STDIN, and if it contains
+ recognized commands and parameters, it runs the corresponding program
+ locally on `lists.gnu.org`.
+7. `sv_mainman` sends email notification to the administrators of the mailing
+ lists, with the newly generated random passwords.
+ (FIXME: emails are sent even if commands fail - there's no error
+ checking).
+
+Example: running the following on `mgt0` as user `root` will result in
+changing the mailing-list password on `lists.gnu.org`:
+
+ # ssh -T l...@lists.gnu.org <<EOF
+ COMMAND=change_pw
+ LIST_NAME=bug-datamash
+ PASSWORD=123456
+ EOF
+
+The corresponding code in `lists.gnu.org:/home/list/savannah_wrapper.pl` (
+lots of code omitted for brevity):
+
+ ...
+ my %vars;
+ while (<>) {
+ chomp;
+ my ($variable, $value) = split('=');
+ $vars{lc($variable)} = $value;
+ }
+ ...
+ if ($vars{'command'} eq 'change_pw') {
+ (system('change_pw',
+ '-l', $vars{'list_name'},
+ '-p', $vars{'password'},
+ '--quiet')
+ == 0) or die "change_pw: $!";
+ }
+
+
+
+TODO: document the "virtual host" hack (e.g. allowing gnu/nongnu mailing
lists).
+
+
+Logs
+----
+
+On `mgt0`, the log file is `mgt0:/var/log/sv_database2system.log`:
+
+ [sv_mailman] Sun Apr 9 01:50:01 2017 - starting
+ [sv_mailman] Sun Apr 9 01:50:02 2017 - List www-zh-cn-translators
<Fossilet> config_list.
+ [sv_mailman] Sun Apr 9 01:50:02 2017 - List www-zh-cn-translators
<Fossilet> reconfigured.
+ [sv_mailman] Sun Apr 9 01:50:02 2017 - List www-zh-cn-translators
password was reset.
+ [sv_mailman] Sun Apr 9 01:50:02 2017 - Mail sent to f...@gnu.org,
b...@gnu.org, b...@gnu.org.
+ [sv_mailman] Sun Apr 9 01:50:02 2017 - work finished
+
+On `lists.gnu.org`, the log file is
`lists.gnu.org:/home/list/savannah_wrapper.log`.
+It contains all the passed parameters (and thus passwords in clear-text):
+
+ Tue Apr 18 02:52:06 2017
+ password = 12345
+ list_name = pretest-users
+ command = change_pw
+
+
+Manual walk-through example - changing password
+-----------------------------------------------
+
+When a project-admin on savannah requests password-reset
+on their mailing list, the `password` field is set to `1`.
+Equivalent command:
+
+ $ mysql savane
+ mysql> UPDATE mail_group_list SET password='1' where
list_name='pretest-users' LIMIT 1;
+
+On `mgt0` the `sv_mailman` is run periodically from cron.
+Equivalent command:
+
+ $ ssh root@mgt0
+ # cd /opt/savannah/bin
+ # ./sv_mailman --cron
+ New pretest-users password: foobar12345
+
+The log file `mgt0:/var/log/sv_database2system.log` will show:
+
+ [sv_mailman] Mon 17 Apr 2017 10:52:05 PM EDT - starting
+ [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - List pretest-users password
was reset.
+ [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - Mail sent to
assafgor...@gmail.com for list pretest-users.
+ [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - work finished
+
+The `sv_mailman` generated a new random password and executed `change_pw`.
+Equivalent command (still on `mgt0`):
+
+ $ ssh root@mgt0
+ # cd /opt/savannah/sbin
+ # ./change_pw -n pretest-users -p 123456
+ New pretest-users password: 123456
+
+The `change_pw` script connects to `lists.gnu.org`
+and transmits the commands as STDIN.
+Equivalent command (still on `mgt0`):
+
+ # ssh -T l...@lists.gnu.org <<EOF
+ COMMAND=change_pw
+ LIST_NAME=pretest-users
+ PASSWORD=123456
+ EOF
+
+
+The script `savannah_wrapper.pl` on `lists.gnu.org`
+reads the commands from STDIN and execute the needed
+programs.
+Equivalent command (on `lists.gnu.org`):
+
+ $ ssh l...@list.gnu.org
+ lists:~$ which change_pw
+ /home/list/mailman/bin/change_pw
+ lists:~$ change_pw -l pretest-users -p 123456 --quiet
+ New pretest-users password: 123456
+
+
+
+
+Mailing list creation - FOR REFERENCE ONLY
+------------------------------------------
+
+**The Following section describes the original setup. It is kept for
+historical purposes**
+
+I (Sylvain) basically installed sv_mailman on internal.sv.gnu.org,
+modified it to support mailman virtual hosts (and use the proper admin
+mail instead of usern...@sv.gnu.org). Then I wrote a fake
+`/usr/sbin/newlist` that connects to l...@lists.gnu.org via SSH and call
+an helper script there (`savannah_wrapper.pl`). This happens via a
+Savannah public key in l...@lists.gnu.org's `authorized_keys` with a
+`command=` restriction.
+
+There is also an empty `/usr/sbin/config_list` on internal that does
+nothing; we don't want what Savane currently does there.
+
+The cron job runs on internal every five minutes, file
+`/etc/cron.d/sv`. The script being run is named sv_mailman.
+
+On lists.gnu.org, there is a script `mycreate_list.pl` which can be used
+to create lists by hand. It duplicates what `savannah_wrapper.pl` does
+for Savannah actions. (The scripts must be kept in sync by hand. They
+should be merged so there is only one.)
