[Mailman-Developers] Re: Mailman, postfix and docker

[Danil Smirnov]

Hey,

As promised a while ago, I've prepared Kubernetes PoC for Mailman 3 (Helm
chart) which is currently deployed into GCP (GKE Autopilot) and exposed at
http://test.mailman4.com/postorius/lists/

The setup is based on Abhilash's container images
 (up to version 0.3.11 - see
below why). Outbound delivery is relayed via Mailgun (due to the outbound
port 25 is blocked in GCP), while income emails received by the local
Postfix container.

When tried previously,

...I've faced only two disappointing issues: 1) HK whitelisting mechanism
> which makes no sense in a dynamic Kubernetes environment and 2) a shared
> volume, which is needed for Core and Postfix to communicate.


The first issue has been resolved by allowing Django's any host notation
 (thank you
folks!), and the second one I've resolved by myself with help of a sidecar
container built from the following code:
https://github.com/danil-smirnov/mailman-postfix-extender

This solution moves Postfix configuration from file-based to network-based
without touching a line of Mailman code. (Of course, it's PoC still and
would look much better if it's part of the Mailman 3 codebase, covered by
tests, etc.)

There are a couple of other issues that I've worked around but have to
address to move to a production-grade solution:

   1. Hardcoded hostnames issue (
   https://github.com/maxking/docker-mailman/issues/449)
   Worked around by adding some redundant code
   

   .
   2. lmtp_host parameter issue (
   https://gitlab.com/mailman/mailman/-/issues/852)
   Worked around by using IP addresses in postfix_lmtp file (possible up to
   0.3.11 version of docker images only - due to recent change
   

   ).

The code could be found in this repository:
https://github.com/danil-smirnov/mailman-helm-chart

I'm glad I've finally shared this, I hope it'll help to improve Mailman 3
compatibility with the modern orchestration tools.

With my best regards,
Danil Smirnov
Mailman3.com


On Sun, Dec 20, 2020 at 1:51 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Danil Smirnov writes:
>
>  > Sorry for the confusion - I should write "would allow" instead of
> "allows"
>  > as using socketmap_table would be a great feature but it obviously
> requires
>  > a new endpoint in Mailman core which does not exist yet.
>
> Ah, OK, so what exists now is the Postfix docker (which you provided a
> reference).
>
>  > Otherwise everything looks fine and with those two issues
>  > resolved/mitigated there is no problem to deploy Mailman 3 in
> Kubernetes, I
>  > can prepare an example Helm chart for it.
>
> That would be great!
>
> Steve
>
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Stephen J. Turnbull]

Danil Smirnov writes:

 > Sorry for the confusion - I should write "would allow" instead of "allows"
 > as using socketmap_table would be a great feature but it obviously requires
 > a new endpoint in Mailman core which does not exist yet.

Ah, OK, so what exists now is the Postfix docker (which you provided a
reference).

 > Otherwise everything looks fine and with those two issues
 > resolved/mitigated there is no problem to deploy Mailman 3 in Kubernetes, I
 > can prepare an example Helm chart for it.

That would be great!

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Danil Smirnov]

Hey Steve,

вс, 20 дек. 2020 г., 6:03 Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp>:

> Danil Smirnov writes:
>
>  > But how about that idea to offload Postfix-Mailman communication from a
>  > filesystem to a network using Postfix socketmap_tables feature which I
> like
>  > a lot as it allows me to get rid of using any shared volumes in the
> stack
>  > completely - it's quite convenient in multi-host setups like Docker
> Swarm
>  > or Kubernetes...
>
> Would you be willing to write up a description of your configuration
> that we can add to the wiki and/or documentation?  Communication in
> multihost setups, especially those intended to isolate hosts exposed
> to the internet from the intranet, is a sore point for us since
> Mailman 3 is really designed as a single-host, three application
> system.


Sorry for the confusion - I should write "would allow" instead of "allows"
as using socketmap_table would be a great feature but it obviously requires
a new endpoint in Mailman core which does not exist yet.

I've experimented a lot with Mailman 3 in Kubernetes and I've faced only
two disappointing issues: 1) HK whitelisting mechanism
 which makes no sense in
a dynamic Kubernetes environment and 2) a shared volume, which is needed
for Core and Postfix to communicate.

Otherwise everything looks fine and with those two issues
resolved/mitigated there is no problem to deploy Mailman 3 in Kubernetes, I
can prepare an example Helm chart for it.

Danil
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Stephen J. Turnbull]

Danil Smirnov writes:

 > But how about that idea to offload Postfix-Mailman communication from a
 > filesystem to a network using Postfix socketmap_tables feature which I like
 > a lot as it allows me to get rid of using any shared volumes in the stack
 > completely - it's quite convenient in multi-host setups like Docker Swarm
 > or Kubernetes...

Would you be willing to write up a description of your configuration
that we can add to the wiki and/or documentation?  Communication in
multihost setups, especially those intended to isolate hosts exposed
to the internet from the intranet, is a sore point for us since
Mailman 3 is really designed as a single-host, three application
system.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Danil Smirnov]

Hi Hans-Martin,

On Sat, Dec 19, 2020 at 1:05 PM Hans-Martin Mosner  wrote:

> We're currently working on moving our mail system over to a "mailu"
> dockerized implementation, so I will gather some
> experience in connecting Mailman3 with a dockerized postfix.
>

Yeah, using dockerised postfix is straight-forward in the same
docker-compose stack - one just need to mount var/data folder of Mailman
core into the Postfix container, set transport_maps, local_recipient_maps
and relay_domains parameters to correct values and set Postfix service name
into SMTP_HOST environment variable of Core and Web services.

(I'm using my simple Postfix Docker image for this purpose - works pretty
well:
https://hub.docker.com/r/danilsmirnov/postfix )

But how about that idea to offload Postfix-Mailman communication from a
filesystem to a network using Postfix socketmap_tables feature which I like
a lot as it allows me to get rid of using any shared volumes in the stack
completely - it's quite convenient in multi-host setups like Docker Swarm
or Kubernetes...

Best regards,
Danil
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Hans-Martin Mosner]

Am 18.12.20 um 18:49 schrieb costavitor...@gmail.com:
> So, my questions are:
> - Is there a way to share aliases db between containers(different hosts) of 
> mailman and postfix? I thought about sharing volumes but I'm not quite sure 
> if it's the best solution
It is possible to share volumes between containers (or preferably use a host 
directory that's mounted into both
containers). You need to be careful with userid issues, though, if you're 
running the processes within the containers as
non-root users.
>
> - I've seen:
>  virtual_alias_maps = mysql:/etc/postfix/virtual/existing_aliases.cf, 
> var/lib/mailman/data/virtual-mailman
>   to add in postfix main.cf. Is it possible to do the same thing to aliases 
> and transport_maps? And if so, how to set it in mailman configuration, 
> because this is for mailman

As Mark already wrote, Mailman3 uses LMTP, but you still need to share the 
transport map created by Mailman3 with
postfix. With 2 containers having access to the same shared filesystem (either 
running on the same docker host, or using
a network filesystem when you've got a cluster) it is pretty straightforward.

In a configuration where Mailman3 and Postfix are running on separate hosts, I 
have used the Mailman3 "postmap-command"
option in var/etc/postfix-mailman.cfg to call a shell script which first runs 
postmap and then copies the map to the
postfix host. That works well enough, but it took some time experimenting with 
several options. You need to set the
lmtp_host and lmtp_port options in mailman.cfg to point to the container 
running Mailman3. If you expose the container
LMTP port to the docker host, it might be easier, but it's likely safer to keep 
the inter-container communication really
internal.

We're currently working on moving our mail system over to a "mailu" dockerized 
implementation, so I will gather some
experience in connecting Mailman3 with a dockerized postfix.

Cheers,
Hans-Martin
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Mark Sapiro]

On 12/18/20 9:49 AM, costavitor...@gmail.com wrote:
> Hi all
> 
> I'm upgrading a mailman2 installation to mailman3. And I using a docker 
> implementation, created by me and not Maxking's implementation.
> 
> Everything is installed and is served correctly. 
> I have created a postfix container that relays emails to a smtp server, and I 
> can send emails by telnet from it.
> When I create a new thread in Hyperkitty, I receive an error from postfix 
> container that the user (mailing list name) is not found.
> 
> And I believe that is obvious because the aliases db are not accessible by 
> postfix.


In Mailman 3, Postfix uses transport_maps to deliver to Mailman via
LMTP. It doesn't use aliases.


> So, my questions are:
> - Is there a way to share aliases db between containers(different hosts) of 
> mailman and postfix? I thought about sharing volumes but I'm not quite sure 
> if it's the best solution


I don't know Docker and can't answer this.


> - I've seen:
>  virtual_alias_maps = mysql:/etc/postfix/virtual/existing_aliases.cf, 
> var/lib/mailman/data/virtual-mailman
>   to add in postfix main.cf.

That looks very strasnge. I don't think Postfix would like a relative
path, but ignoring that, it certainly wants a table type.


> Is it possible to do the same thing to aliases and transport_maps? And if so, 
> how to set it in mailman configuration, because this is for mailman


I don't know what you mean by "the same thing". See

for configuring Postfix for Mailman 3.

Mailman can generate either hash tables or regexp tables. The above doc
has details.

-- 
Mark Sapiro The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9