Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Markus Gschwendt]

Hi Jonas!
On Mon, 2018-10-08 at 00:43 +0200, Jonas Meurer wrote:
> Yes. At least with mailman3 3.1.1-8
> 
> Sorry to poke you again. Can you still reproduce it with latest mailman3
> 3.2 packages from Debian unstable? Did you try to do a fresh
> installation of mailman3 in the meantime?

No, but i'm very busy at the moment.
I will try in about a month.

A friend (who is running several small mailman2 servers) tried to
install mailman3 without the Web-GUI and was very disappointed about
the huge memory footprint for a mailman with only one list and a few
members and doing nothing. This would be a reason to move away from
mailman.

Markus

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Jonas Meurer]

Hey Markus,

On Wed, 18 Apr 2018 15:33:11 +0200 Markus Gschwendt
 wrote:
> On Wed, 2018-04-18 at 12:06 +0200, Jonas Meurer wrote:
> > do you still encounter this problem?
> 
> Yes. At least with mailman3 3.1.1-8

Sorry to poke you again. Can you still reproduce it with latest mailman3
3.2 packages from Debian unstable? Did you try to do a fresh
installation of mailman3 in the meantime?

> 
> >  I bet there's something wrong on
> > your side. Mailman3 works fine with Exim4 and if you configured it
> > correctly, it certainly doesn't run the Postfix postmap command.
> 
> It's an install of the mailman3-Debian-Sid packages. Nothing else is on
> that machine.
> 
> exim4: 4.90.1-5
> nginx-light: 1.13.12-1
> mailman3: 3.1.1-9
> 
> I just did upgrade from 3.1.1-8 to 3.1.1-9 and get this on restarting
> mailman:
> 
> service mailman3 stopTraceback (most recent call last):
>   File "/usr/lib/mailman3/bin/master", line 11, in 
> load_entry_point('mailman==3.1.1', 'console_scripts', 'master')()
>   File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line
> 528, in main
> with open(config.PID_FILE, 'w') as fp:
> PermissionError: [Errno 13] Permission denied:
> '/run/mailman3/master.pid'

This permission error smells suspcious to me. /run/mailman/ should be
owned by 'list'list' and the mailman3 daemon should put the pid file
into it with same ownership.

Maybe something got screwed during installation/configuration/debugging
of the packages in early state. We indeed had some nasty permission bugs
in the first releases of mailman3 packages. The latest packages should
have fixed them (at least the ones we know of).

If you don't discover this bug anymore or are unable to reproduce, I'd
like to close the bugreport.

Cheers
 jonas



signature.asc
Description: OpenPGP digital signature

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Markus Gschwendt]

On Wed, 2018-04-18 at 12:06 +0200, Jonas Meurer wrote:
> do you still encounter this problem?

Yes. At least with mailman3 3.1.1-8

>  I bet there's something wrong on
> your side. Mailman3 works fine with Exim4 and if you configured it
> correctly, it certainly doesn't run the Postfix postmap command.

It's an install of the mailman3-Debian-Sid packages. Nothing else is on
that machine.

exim4: 4.90.1-5
nginx-light: 1.13.12-1
mailman3: 3.1.1-9

I just did upgrade from 3.1.1-8 to 3.1.1-9 and get this on restarting
mailman:

service mailman3 stopTraceback (most recent call last):
  File "/usr/lib/mailman3/bin/master", line 11, in 
load_entry_point('mailman==3.1.1', 'console_scripts', 'master')()
  File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line
528, in main
with open(config.PID_FILE, 'w') as fp:
PermissionError: [Errno 13] Permission denied:
'/run/mailman3/master.pid'


> If you still encounter this problem, please paste the output of the
> following commands:
> 
> # mailman info

mailman info
GNU Mailman 3.1.1 (Between The Wheels)
Python 3.6.5rc1 (default, Mar 22 2018, 10:45:39) 
[GCC 7.3.0]
config file: /etc/mailman3/mailman.cfg
db url: sqlite:var/lib/mailman3/data/mailman.db
devmode: DISABLED
REST root url: http://localhost:8001/3.1/
REST credentials: restadmin:x


> # mailman conf | grep -F "[mta]"

[mta] configuration: python:mailman.config.exim4
[mta] delivery_retry_period: 5d
[mta] incoming: mailman.mta.exim4.LMTP
[mta] lmtp_host: 127.0.0.1
[mta] lmtp_port: 8024
[mta] max_autoresponses_per_day: 10
[mta] max_delivery_threads: 0
[mta] max_recipients: 500
[mta] max_sessions_per_connection: 0
[mta] outgoing: mailman.mta.deliver.deliver
[mta] remove_dkim_headers: no
[mta] smtp_host: localhost
[mta] smtp_pass: 
[mta] smtp_port: 25
[mta] smtp_user: 
[mta] verp_confirm_format: $address+$cookie
[mta] verp_confirm_regexp:
^(.*<)?(?P[^+]+?)\+(?P[^@]+)@.*$
[mta] verp_confirmations: no
[mta] verp_delimiter: +
[mta] verp_delivery_interval: 0
[mta] verp_format: ${bounces}+${local}=${domain}
[mta] verp_personalized_deliveries: no
[mta] verp_probe_format: $bounces+$token@$domain
[mta] verp_probe_regexp: ^(?P[^+]+?)\+(?P[^@]+)@.*$
[mta] verp_probes: no
[mta] verp_regexp:
^(?P[^+]+?)\+(?P[^=]+)=(?P[^@]+)@.*$


> # cat /etc/mailman3/mailman.cfg

# Copyright (C) 2008-2017 by the Free Software Foundation, Inc.
#
# This file is part of GNU Mailman.
#
# GNU Mailman is free software: you can redistribute it and/or modify
it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your
option)
# any later version.
#
# GNU Mailman is distributed in the hope that it will be useful, but
WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
for
# more details.
#
# You should have received a copy of the GNU General Public License
along with
# GNU Mailman.  If not, see .

# This file contains the Debian configuration for mailman.  It uses
ini-style
# formats under the lazr.config regime to define all system
configuration
# options.  See  for details.


[mailman]
# This address is the "site owner" address.  Certain messages which
must be
# delivered to a human, but which can't be delivered to a list owner
(e.g. a
# bounce from a list owner), will be sent to this address.  It should
point to
# a human.
site_owner: x...@example.com

# This is the local-part of an email address used in the From field
whenever a
# message comes from some entity to which there is no natural reply
recipient.
# Mailman will append '@' and the host name of the list involved.  This
# address must not bounce and it must not point to a Mailman process.
noreply_address: noreply

# The default language for this server.
default_language: en

# Membership tests for posting purposes are usually performed by
looking at a
# set of headers, passing the test if any of their values match a
member of
# the list.  Headers are checked in the order given in this
variable.  The
# value From_ means to use the envelope sender.  Field names are case
# insensitive.  This is a space separate list of headers.
sender_headers: from from_ reply-to sender

# Mail command processor will ignore mail command lines after
designated max.
email_commands_max_lines: 10

# Default length of time a pending request is live before it is evicted
from
# the pending database.
pending_request_life: 3d

# How long should files be saved before they are evicted from the
cache?
cache_life: 7d

# A callable to run with no arguments early in the initialization
process.
# This runs before database initialization.
pre_hook:

# A callable to run with no arguments late in the initialization
process.
# This runs after adapters are initialized.
post_hook:

# Which paths.* file system layout to use.
# You should not change this 

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Jonas Meurer]

Hi Markus,

do you still encounter this problem? I bet there's something wrong on
your side. Mailman3 works fine with Exim4 and if you configured it
correctly, it certainly doesn't run the Postfix postmap command.

If you still encounter this problem, please paste the output of the
following commands:

# mailman info
# mailman conf | grep -F "[mta]"
# cat /etc/mailman3/mailman.cfg

Cheers
 jonas

On Thu, 29 Mar 2018 18:15:42 +0200 Markus Gschwendt 
wrote:
> On Thu, 2018-03-29 at 17:57 +0200, Jonas Meurer wrote:
> > 
> > > ...
> > >   File "/usr/lib/python2.7/smtplib.py", line 292, in _get_socket
> > > return socket.create_connection((host, port), timeout)
> > > >   File "/usr/lib/python2.7/socket.py", line 575, in create_connection
> > > raise err
> > > error: [Errno 111] Connection refused
> > ...
> 
> my fault. don't know why it worked from command line. exim could not
> bind to an unconfigured IPv6 address.
> 
> but again the postfix issue on an exim install during adding a list:
> 
> RuntimeError: command failure: /usr/sbin/postmap
> /var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
> command failure: /usr/sbin/postmap
> /var/lib/mailman3/data/postfix_domains, 127, Key has expired
> Mar 29 16:11:38 2018 (8096) 127.0.0.1 - - "POST /3.0/lists HTTP/1.1"
> 500 59
> Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET /3.0/domains HTTP/1.1"
> 200 294
> Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
> /3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
> Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
> /3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
> Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
> /3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
> Mar 29 16:12:02 2018 (8096) command failure: /usr/sbin/postmap
> /var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
> Mar 29 16:12:02 2018 (8096) command failure: /usr/sbin/postmap
> /var/lib/mailman3/data/postfix_domains, 127, Key has expired
> Mar 29 16:12:02 2018 (8096) REST request handler error:
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/wsgiref/handlers.py", line 137, in run
> self.result = application(self.environ, self.start_response)
>   File "/usr/lib/python3/dist-
> packages/mailman/database/transaction.py", line 50, in wrapper
> rtn = function(*args, **kws)
>   File "/usr/lib/python3/dist-packages/mailman/rest/wsgiapp.py", line
> 214, in __call__
> return super().__call__(environ, start_response)
>   File "falcon/api.py", line 215, in falcon.api.API.__call__
>   File "falcon/api.py", line 189, in falcon.api.API.__call__
>   File "/usr/lib/python3/dist-packages/mailman/rest/lists.py", line
> 245, in on_post
> mlist = create_list(**validator(request))
>   File "/usr/lib/python3/dist-packages/mailman/app/lifecycle.py", line
> 100, in create_list
> call_name(config.mta.incoming).create(mlist)
>   File "/usr/lib/python3/dist-packages/mailman/mta/postfix.py", line
> 66, in create
> self.regenerate()
>   File "/usr/lib/python3/dist-packages/mailman/mta/postfix.py", line
> 103, in regenerate
> raise RuntimeError(NL.join(errors))
> RuntimeError: command failure: /usr/sbin/postmap
> /var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
> command failure: /usr/sbin/postmap



signature.asc
Description: OpenPGP digital signature

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Markus Gschwendt]

On Thu, 2018-03-29 at 17:57 +0200, Jonas Meurer wrote:
> 
> > ...
> >   File "/usr/lib/python2.7/smtplib.py", line 292, in _get_socket
> > return socket.create_connection((host, port), timeout)
> > >   File "/usr/lib/python2.7/socket.py", line 575, in create_connection
> > raise err
> > error: [Errno 111] Connection refused
> ...

my fault. don't know why it worked from command line. exim could not
bind to an unconfigured IPv6 address.

but again the postfix issue on an exim install during adding a list:

RuntimeError: command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_domains, 127, Key has expired
Mar 29 16:11:38 2018 (8096) 127.0.0.1 - - "POST /3.0/lists HTTP/1.1"
500 59
Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET /3.0/domains HTTP/1.1"
200 294
Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
/3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
/3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "GET
/3.0/domains/mm3a.runout.at HTTP/1.1" 200 189
Mar 29 16:12:02 2018 (8096) command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
Mar 29 16:12:02 2018 (8096) command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_domains, 127, Key has expired
Mar 29 16:12:02 2018 (8096) REST request handler error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
  File "/usr/lib/python3/dist-
packages/mailman/database/transaction.py", line 50, in wrapper
rtn = function(*args, **kws)
  File "/usr/lib/python3/dist-packages/mailman/rest/wsgiapp.py", line
214, in __call__
return super().__call__(environ, start_response)
  File "falcon/api.py", line 215, in falcon.api.API.__call__
  File "falcon/api.py", line 189, in falcon.api.API.__call__
  File "/usr/lib/python3/dist-packages/mailman/rest/lists.py", line
245, in on_post
mlist = create_list(**validator(request))
  File "/usr/lib/python3/dist-packages/mailman/app/lifecycle.py", line
100, in create_list
call_name(config.mta.incoming).create(mlist)
  File "/usr/lib/python3/dist-packages/mailman/mta/postfix.py", line
66, in create
self.regenerate()
  File "/usr/lib/python3/dist-packages/mailman/mta/postfix.py", line
103, in regenerate
raise RuntimeError(NL.join(errors))
RuntimeError: command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_lmtp, 127, Key has expired
command failure: /usr/sbin/postmap
/var/lib/mailman3/data/postfix_domains, 127, Key has expired
Mar 29 16:12:02 2018 (8096) 127.0.0.1 - - "POST /3.0/lists HTTP/1.1"
500 59

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Jonas Meurer]

Hi Markus,

thanks again for your feedback!

Am 29.03.2018 um 15:32 schrieb Markus Gschwendt:
> Maybe this bug should be renamed to something more generic. I don't
> want to open a new bug for everything i run into on debugging the
> install process until it's stable.

I don't mind abusing this bugreport for issues you run into.

> I just tried again with the latest sid repo.
> Mail/SMTP from commandline does work (eg `mail -v -s test root`)
> 
> 8<
> [mta]
> incoming: mailman.mta.exim4.LMTP
> outgoing: mailman.mta.deliver.deliver
> smtp_host: localhost
> smtp_port: 25
> smtp_user:
> smtp_pass:
> lmtp_host: 127.0.0.1
> lmtp_port: 8024
> configuration: python:mailman.config.exim4
> >8

That's great and a big step forward :) So at least mailman3 now works on
sysvinit systems! Let's see what's up with mailman3-web ...

> Now i get until the WebUI-Login page. After entering User/PW i get
> following in /var/log/mailman3/web/mailman.log:
> 
> 8<
> ERROR 2018-03-29 15:13:29,516 12710 django.request Internal Server
> Error: /accounts/login/
> Traceback (most recent call last):
> [...]
>   File "/usr/lib/python2.7/dist-packages/django/core/mail/message.py",
> line 348, in send
> return self.get_connection(fail_silently).send_messages([self])
>   File "/usr/lib/python2.7/dist-
> packages/django/core/mail/backends/smtp.py", line 104, in send_messages
> new_conn_created = self.open()
>   File "/usr/lib/python2.7/dist-
> packages/django/core/mail/backends/smtp.py", line 64, in open
> self.connection = self.connection_class(self.host, self.port,
> **connection_params)
>   File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
> (code, msg) = self.connect(host, port)
>   File "/usr/lib/python2.7/smtplib.py", line 317, in connect
> self.sock = self._get_socket(host, port, self.timeout)
>   File "/usr/lib/python2.7/smtplib.py", line 292, in _get_socket
> return socket.create_connection((host, port), timeout)
>   File "/usr/lib/python2.7/socket.py", line 575, in create_connection
> raise err
> error: [Errno 111] Connection refused

The Django app fails at sending a mail. Is the app allowed to send mail
via email backend 'django.core.mail.backends.smtp.EmailBackend' to the
mail address you gave for your superuser?

The Django app sends a confirmation mail at first login, asking to
verify the mail address of the account. Only afterwards you're allowed
to login as the user.

Cheers,
 jonas



signature.asc
Description: OpenPGP digital signature

Bug#891937: [Pkg-mailman-hackers] Bug#891937: mailman3-full install issues

[Markus Gschwendt]

Maybe this bug should be renamed to something more generic. I don't
want to open a new bug for everything i run into on debugging the
install process until it's stable.

I just tried again with the latest sid repo.
Mail/SMTP from commandline does work (eg `mail -v -s test root`)

8<
[mta]
incoming: mailman.mta.exim4.LMTP
outgoing: mailman.mta.deliver.deliver
smtp_host: localhost
smtp_port: 25
smtp_user:
smtp_pass:
lmtp_host: 127.0.0.1
lmtp_port: 8024
configuration: python:mailman.config.exim4
>8

Now i get until the WebUI-Login page. After entering User/PW i get
following in /var/log/mailman3/web/mailman.log:

8<
ERROR 2018-03-29 15:13:29,516 12710 django.request Internal Server
Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", 
line 249, in _legacy_get_response
response = self._get_response(request)
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", 
line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", 
line 185, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", 
line 68, in view
return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/utils/decorators.py",
line 67, in _wrapper
return bound_func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-
packages/django/views/decorators/debug.py", line 76, in
sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/utils/decorators.py",
line 63, in bound_func
return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/usr/lib/python2.7/dist-packages/allauth/account/views.py",
line 137, in dispatch
return super(LoginView, self).dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/allauth/account/views.py",
line 80, in dispatch
**kwargs)
  File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", 
line 88, in dispatch
return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/allauth/account/views.py",
line 103, in post
response = self.form_valid(form)
  File "/usr/lib/python2.7/dist-packages/allauth/account/views.py",
line 150, in form_valid
return form.login(self.request, redirect_url=success_url)
  File "/usr/lib/python2.7/dist-packages/allauth/account/forms.py",
line 197, in login
redirect_url=redirect_url)
  File "/usr/lib/python2.7/dist-packages/allauth/account/utils.py",
line 146, in perform_login
send_email_confirmation(request, user, signup=signup)
  File "/usr/lib/python2.7/dist-packages/allauth/account/utils.py",
line 314, in send_email_confirmation
signup=signup)
  File "/usr/lib/python2.7/dist-packages/allauth/account/models.py",
line 60, in send_confirmation
confirmation.send(request, signup=signup)
  File "/usr/lib/python2.7/dist-packages/allauth/account/models.py",
line 166, in send
get_adapter(request).send_confirmation_mail(request, self, signup)
  File "/usr/lib/python2.7/dist-packages/allauth/account/adapter.py",
line 444, in send_confirmation_mail
ctx)
  File "/usr/lib/python2.7/dist-packages/allauth/account/adapter.py",
line 137, in send_mail
msg.send()
  File "/usr/lib/python2.7/dist-packages/django/core/mail/message.py",
line 348, in send
return self.get_connection(fail_silently).send_messages([self])
  File "/usr/lib/python2.7/dist-
packages/django/core/mail/backends/smtp.py", line 104, in send_messages
new_conn_created = self.open()
  File "/usr/lib/python2.7/dist-
packages/django/core/mail/backends/smtp.py", line 64, in open
self.connection = self.connection_class(self.host, self.port,
**connection_params)
  File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
(code, msg) = self.connect(host, port)
  File "/usr/lib/python2.7/smtplib.py", line 317, in connect
self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python2.7/smtplib.py", line 292, in _get_socket
return socket.create_connection((host, port), timeout)
  File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
ERROR 2018-03-29 15:13:29,516 12710 django.request Internal Server
Error: /accounts/login/
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-
packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", 
line 249, in _legacy_get_response
response = self._get_response(request)
  File