[Mailman-Developers] Re: Help on Gitlab

[Stephen J. Turnbull]

Usama Nadeem writes:

 > How did you set up your local machine? The only doc I could find on
 > setting up was this: https://wiki.list.org/DEV/HowToContributeGit

As a specific example of setting up a dev environment, I did

mkdir Mailman
cd Mailman

then cloned

mailman
mailman-web
mailman-hyperkitty
django-mailman3
mailmanclient
postorius
hyperkitty

from the GitLab mailman project.  Then

python -m venv VENV
. VENV/bin/activate
python -m pip --upgrade install pip
python -m pip install build wheel
mkdir DIST
for d in $the_repos_above; do
  pushd $d
  python -m build -o ../DIST
  popd
done
python -m pip install DIST/*.whl

After that you need to do a certain amount of setup, creating files in
/etc/mailman3, at least mailman.cfg, settings.py, and gunicorn-conf.py
(that last is my preferred name, I forget the exact standard name,
which doesn't express that it's just Python code, and tends to draw
warnings because it lacks the .py extension).

Something that I don't recall being documented is that "mailman-web
compress" expects to be run from Mailman's $var_dir (see mailman.cfg
for its definition).  Also, recently the django-allauth module which
is downloaded from PyPI has been quite unstable (in the sense of API
compatibility, AFAIK it works fine once APIs match).  The HEAD commits
of postorius, hyperkitty, and django-mailman3 should be compatible
with the most recent django-allauth, but if you get errors from
"mailman-web" subcommands about missing methods or wrong type
arguments, there's a good chance it's django-allauth (especially if
for some reason you are working with past versions).  Feel free to ask
on the list about those.

OTOH, if you're at all experienced with Python, they're not hard to
debug.  If you are thinking about contributing to the mailman-web side
of the suite, debugging such errors yourself is a good exercise to dip
your toe into the code.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support

[Stephen J. Turnbull]

Justus Winter writes:

 > But currently Mailman3 does fork+exec, so it doesn't get to share
 > the parent's pages.  I experimented with fork-and-dont-exec [0],
 > but the results were underwhelming, because reference counting can
 > cause pages to diverge.  Surprisingly, gc.freeze didn't seem to
 > help much, so there may have been issues beyond the reference
 > counts.
 > [...]
 > I think Python just doesn't support sharing code across processes
 > well.

Seems likely.  I know that Emacsen have always advised running just
one process for this reason (also because users usually want all their
recent hacks available in all buffers, but memory hogging is a big
reason).

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] OpenPGP support

[Stephen J. Turnbull]

This is split thread #3.

Justus Winter writes:

 > >  >   - Implement OpenPGP support
 > >
 > > What does that mean?
 > 
 > OpenPGP can be used to provide confidentiality and integrity for
 > email.  What exactly that means in the setting of mailing lists
 > varies by threat model and policy.

I was afraid you'd say that.  I mean, it's the right generic answer,
but I've yet to see a viable use case with a plausible threat model
for any of the implementations proposed.

 > My prototype [2] simply records associations between addresses and
 > OpenPGP certificates by consuming Autocrypt headers [3] and when
 > sending an outgoing mail opportunistically encrypting it if a
 > certificate is known.

Except for the Autocrypt part, this has been done.  But there are two
problems: nobody wants it very badly (see this post specifically

and the surrounding thread is also valuable because you'll see all the
reasons why I don't want to do this in Mailman at present, and you're
the first person in decades I think has a good shot at convincing me
otherwise! :-)  The second problem is I don't see a convincing use
case.  Note: I don't consider the opportunistic encryption aspect a
serious flaw.  Obviously this initial proposal is mostly a proof-of-
concept and most (all?) serious applications simply wouldn't send
unencrypted mail.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Threads and robustness against runner crashes

[Stephen J. Turnbull]

Split thread #2.

Justus Winter writes:

 > >  > Here are the things I did so far:
 > >  > 
 > >  >   - I have Mailman running with runners in threads instead of
 > >  > processes, but that is in a proof-of-concept stage at this
 > >  > point and needs some cleaning up
 > >
 > > After working with Mailman 3 and Postfix, I've become fond of
 > > the HUPD (HUPD of Uncontrolled Proliferation of Daemons) model
 > > of application design, at least for email.
 > 
 > My prototype let's you chose, for every kind of runner, whether to
 > use the process or thread model

That's not a sales point, as far as I'm concerned.  It adds complexity
for the installer and the site manager, as well as in the code.

 > I don't quite buy (or maybe I'm not understanding the whole picture)
 > into the argument that having individual processes improves the
 > robustness of the whole system.

I'm talking about the developer/maintainer experience, not about run
time.

 > From my experience, having individual runners killed can render
 > Mailman unusable [0] (and to my then untrained eye it was
 > impossible to see that a runner was missing,

That's some combination of documentation, logging, and tooling bugs.
At the very least "mailman status" should report whether all the
runners that were started are still present (it doesn't at present).

It's really not hard to detect a crashed or stalled runner, even in a
sliced (multirunner) queue -- queuefiles start to pile up.  (By "not
hard" I mean you can use "ls" or "du", not that it should be obvious
what to do.)

 > if on the other hand Mailman would have been a single process, or a
 > significantly smaller number of processes, a single missing process
 > would have been more apparent),

True, but to me crashes in a monolithic program are less acceptable,
expecially threaded, because other concurrent operations may depend on
that program staying alive.  The way exception handling is done in
Mailman 2 with a big "except Exception" around the whole program, you
mostly would not get a crash at all, just a log message with an
traceback, probably unintelligible to a non-developer of Mailman.  Not
clear that's a win over the current situation for you.  Sure, you can
probably arrange for exception handling to be per-thread in some
sense, but that's going to be conceptually harder than the the "log
the exception, let it crash, have the master restart it and pray"
approach we use in the multiprocess model.

 > and when a runner has picked up a mail from a queue, and then
 > crashes, that mail is lost forever (i.e. runner operations are not
 > atomic).

Please report such incidents in as much detail as you can.  The whole
point of "store and forward" is to prevent that.  Runners should not
alter the queuefile until they're done.  If they crash in the middle,
they should leave the queuefile they received and maybe a work file.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Optimization for constrained environments and latency

[Stephen J. Turnbull]

I'm going to split this into three separate threads with appropriate
titles.  This is #1.

Regarding optimization for constrained environments, I wrote:

 > > But I'm pretty sure people have run Mailman 3 on a Raspberry Pi.
 > > How constrained an environment are you aiming for?

Justus Winter writes:

 > I had problems on my shared hoster that provided 1 gigabyte of RAM
 > per user (I'm not a 100% on how they measure that).

OK, yes, my estimate says that's going to be too tight.  I'm seeing
about 80MB per runner with a full complement of processes without any
slicing being 18.  Some of that is shared (IIRC about 5MB/runner
according to top), but that only buys back 1 runner's worth.  There
are several somewhat optional processes (the nntp, archive, command,
and 2 WSGI processes for Mailman Web) but that's probably still not
quite going to fit into 1GB.

Re polling queues:

 > the runners are polling their queues in loops.  My installations
 > that hardly see any traffic at all are all doing: do I have work,
 > no, sleep 1, do I have work, no, sleep 1... I can see that this
 > will amortize in big installations, but for small ones this is
 > quite sad.

I guess, but if it doesn't show in the load average, I'm not sure why
one should care.  I don't know about your installation, but Mailman
consumes less than 1% of CPU when idle as far as I can tell.  For me
to support a change here, either you'd need to show a non-negligible
improvement or it would have to be "free" (see below).

 > >  >   - Improve latency of messages
 > >
 > > What latency are you observing?

 > And even for big installations, or if we say that efficiency is not
 > important, if a mail goes through the hands of three queue runners,
 > the worst-case latency is three seconds in an otherwise idle
 > installation!  We can definitively improve upon that.

Who will notice?  Is there anybody who cares about a 3s latency in
list email?  If there is, that would be a user-visible improvement to
set against any increase in code complexity.

 > The key insight here is that emails in queues don't appear out of
 > thin air, another runner is putting them there.  If each runner
 > that goes to sleep does so by waiting on a condition variable
 > associated with its queue, and every runner that deposits a mail
 > into the queue signals the sleeping runners, that latency goes away
 > while at the same time improving efficiency by no longer having to
 > poll the queue every second.

Thing is, email (and Mailman specifically) operates on a store-and-
forward model.  The queue file *must* be present for a runner to do
its work, and conversely, if a file is present the runner has work to
do.  Polling is a little ugly, but it's a perfect fit for the problem
semantically, and very simple to explain and implement.

If the condition-variable-based code is equally simple, equally
reliable, and identical across our supported platforms, sure, that's
worth looking at because we get your developer-visible efficiency
enhancements for "free".  But if any of those requirements fails, I
would want to see an improvement in user-visible performance.

 > > I will take a look at the work you mention, but it will be a
 > > couple of weeks at least before I have useful comments.

Still need some time for this, but I wanted to get some stuff out of
my inbox. :-)
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Introduction, FOSDEM, scaling down, latency, OpenPGP support

[Stephen J. Turnbull]

Hi Justus!

Justus Winter writes:

 > I'd like to contribute a little to Mailman, and I'd like to better
 > understand how the Mailman project is doing nowadays.  I have
 > gotten a bugfix merged in the past, but I now have what I think is
 > a fairly uncontroversial cleanup merge request that has neither
 > been merged nor has it gotten comments.

I think Abhilash has been very busy with day job and some major
changes at home (congratulations!), and I certainly have been busy
with that.  I hope to have more time for Mailman development starting
in April, since I will be retiring from $DAYJOB on March 31.  Mark has
been good about routine requests, especially merging in contributions
from translator.  He and I have been mostly concentrating on
list-based support (most of my work recently), and relatively urgent
bug-fixes (Mark and Abhilash).  There are a few other core devs but
all core currently inactive.

 > As FOSDEM is around the corner, are any of you going to be there
 > and are up for a chat?

I will not be.  Haven't heard anything from the rest of the crew, but
we mostly meet at PyCon.  This is kinda difficult this year though as
my funding for travel from Japan is gone due to preretirement and
Abhilash I believe is in India.

 > Besides cleanups and bugfixes, there are three things I'd like to
 > do:
 > 
 >   - Improve Mailman to better scale down to small installations

Not sure what you can really do about that without rearchitecting.
The full suite of daemons is something like 13, including 3 WSGI
processes, the master daemon for mailman and about 7 or 8 runners.

But I'm pretty sure people have run Mailman 3 on a Raspberry Pi.  How
constrained an environment are you aiming for?

 >   - Improve latency of messages

What latency are you observing?  My last project was getting about
100,000 incoming per day across 20K lists, two incoming runners, 8
outgoing, 1 each for the other Mailman runners. Never saw more than
about 5 seconds dwell in the Mailman system, except when the Mailman
to outgoing Postfix SMTP connection started glitching.  We fixed that
by reconfiguring the Mailman host (in Dallas) to use an MX in the same
datacenter instead of one in Boston. (!!)  And the normal case with a
process where I'd do "ls queue/*" evey 5s was completely empty queues.
Stuff just didn't stay around long enough for ls to see it.

I see no reason to suppose you can do much better than that, but
again, tell me what you're seeing.  I'm not experienced in dealing
with Mailman at scale, and that host was quite beefy.  Still I have a
strong feeling that latency is mostly a communication with MTA issue,
not in Mailman 3 itself.

 >   - Implement OpenPGP support

What does that mean?

 > Here are the things I did so far:
 > 
 >   - I have Mailman running with runners in threads instead of
 > processes, but that is in a proof-of-concept stage at this
 > point and needs some cleaning up

I guess this is supposed to address the resource consumption (memory
footprint?) issue?

After working with Mailman 3 and Postfix, I've become fond of the HUPD
(HUPD of Uncontrolled Proliferation of Daemons) model of application
design, at least for email.  I feel *much* more comfortable messing
with individual daemons this way, knowing that I can't affect the
others.  I'm not going to object to providing the threaded version if
people want it, but I would object to wholesale conversion to that
model without a lot of production experience based on it.

 > (I understand that Mailman is a GNU project that wants copyright
 > assignments, and I have done that in the past for other GNU
 > projects, and would be happy to do that for Mailman, but at the
 > same time I feel like putting up *any* barrier to contributing is
 > unfortunate.)

My experience has been that about 2/3 of resistence has been to any
paperwork as such, only about 1/3 to assignment vs. some sort of
formal license ("contributor agreement", as the PSF calls it).

As far as Mailman is concerned, a lot of the core code has been
completely rewritten for Mailman 3.  However, I know that in
implementing Mailman 2 features not yet in Mailman 3 I've been at
least heavily influenced by Mailman 2 code.  Not sure that anybody
else has been particularly careful about "clean implementations",
although Barry has said that the core of Mailman 3 core is completely
rewritten from scratch.  In any case, the last time licensing was
discussed, the founder (John Viega) was not on board with a separation
from GNU and a permissive license, and Barry and I at least are pretty
sentimental about that.  For those reasons, I believe at at least this
generation of Mailman core devs is unlikely to move in that direction.

I will take a look at the work you mention, but it will be a couple of
weeks at least before I have useful comments.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to 

[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support

[Stephen J. Turnbull]

Дилян   Палаузов writes:

 > Do you know by accident, if the runner processes use identical
 > memory, and by
 > calling https://man7.org/linux/man-pages/man2/madvise.2.html the
 > kernel can somehow detect these identical memory and then use a
 > single instance for all identical regions of memory?

The memory size is a side effect of the way fork works (formally, it
creates a new process by duplicating the old one, what "really"
happens is up to the CPU's MMU and the kernel's memory management).  I
believe that as long as copy-on-write is enabled it only actually
copies pages with changes.  Use ps or top to check to see how much of
the virtually memory is actually in RAM, and how much is shared.

I'm pretty sure real memory usage was considered in the original
design to use processes for each runner rather than threads.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Review requested for some merge requests

[Stephen J. Turnbull]

Reply-To set to Mailman Developers.  Please check that your replies go
there.  Or you can comment on MRs directly if you have a GitLab login.

I have three merge requests pending:

[1] https://gitlab.com/mailman/mailman/-/merge_requests/1159
[2] https://gitlab.com/mailman/mailman/-/merge_requests/1160
[3] https://gitlab.com/mailman/mailman/-/merge_requests/1161

[1] fixes the logging initialization bug that prevented the `smtp` log
from rotating properly, and maybe some others.  I don't think it needs
user review because it's a pure bugfix, but if you're curious and take
a look, and you think it needs better documentation let me know.

[2] adds a configuration variable `lmtp_max_size`.  This is
user-visible, so review of the documentation in docs/mta.rst and
config/mailman.cfg for clarity and readability would be appreciated.

[3] adds postfix-pgsql support for use of the PostgreSQL database to
route Mailman traffic.  It's really most useful in an edge case
(migration of 10,000+ list Mailman 2 instances), but it's
theoretically attractive on the principle of "single source of truth",
and possibly for implementing a "microservices" style cloud instance
(it eliminates the need for Mailman and Postfix to share a file
system).  Besides opinions on the value of the general idea I have two
specific questions:
  - Postfix docs recommend not looking up domains in the database to
reduce load on the database server.  But on a dedicated Mailman
system, you're going to be looking up pretty much everything twice
anyway (once to check for local recipient and once to fetch the
transport), and my experience so far is that this isn't a big load
on Postgres, and it's not a drag on list traffic distribution.
Should I implement domain lookup in Postgres?  Should it be
optional if the lists are being looked up in Postgres?
  - The current patch completely disables generation of the
postfix_lmtp alias file, even for the `mailman aliases` command.
Should I implement an option to force generation of the files for
documentation (and maybe a backup in a very rare and contorted
situation where Mailman can see Postgres but Postfix can't)?

Thanks,
Steve


___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Fwd: (Custom) Plugins: Proper file path(s) for adding a handler?

[Stephen J. Turnbull]

Thomas Ward via Mailman-Developers writes:

 > So, sending it to the mailman-developers list, which is more
 > tech/devel oriented.  Primarily, my goal here is to figure out how
 > to add handlers via a plugin, but there's no documentation for it,
 > so I'm looking for more information.

I think the only person who knows how to do this is Abhilash.  I spent
a couple hours on it and failed.

 > Worst case, I can probably add my handlers and pipelines as direct
 > files in the venv inside Mailman's paths for the PyPI packages,

That's what I did.  Took five minutes (of course I'd already written
the feature's code for the plug-in approach).

For future reference, "all day" is not a long time to wait for
specialized help from a volunteer developer.

Steve___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: how to debug Hyperkitty Django Q tasks?

[Stephen J. Turnbull]

Danil Smirnov writes:

 > Thank you Steve for your comments, but I am still interested in
 > knowing how Mailman developers would debug issues with Hyperkitty
 > periodic tasks.

I'd start by checking if the relevant process (probably qcluster) is
actually running -- in my experience the most common reason for ZERO
logs is the process isn't running.

I'll tell you what I'd do after that *because* a developer should
track this stuff down, but if everything failed, purely as a matter of
debugging I'd just change the default in the update_index function to
verbosity=1, then change it back (or to 2, etc) once you have whatever
information that gives you.  That's what I recommend you do once you
have confirmed the process is running.

Then while waiting on the user channels I'd grep for 'search_indexes'
in import statements to see where search_index.update_index is called
to see how that 'verbosity' parameter gets set.  It may be ignored
(defaulted to 0) or hardcoded, but there may also be a option that can
be set in the LOGGING parameter of settings.py.

If that doesn't work, I'd trace back through the imports and scan
source to try to find alternative logging options for django_haystack,
since you're not actually that interested in the HyperKitty function
which does no real work.

The thing is that phrase "no real work".  Even the verbosity parameter
delegates the logging to third-party code.  The work here is being
done by third-party code in Django and django_haystack, plus whatever
indexer you're using (Whoosh or Xapian).  Probably all of these have
many options for logging.

 > If no logging is possible for those tasks, I'd consider this as an
 > issue to fix.

Sure, but it's going to be a "patches welcome" HyperKitty issue
because all HyperKitty can tell you is the command was called.
Haystack can tell you that too, and a lot more.  django_q (which
calls the HyperKitty tasks) probably can too.

OTOH, we should document third-party logging better.  Note, that
doesn't require Mailman developers, non-developer contributors can do
that too.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] how to debug Hyperkitty Django Q tasks?

[Stephen J. Turnbull]

Danil Smirnov writes:

 > The issue was initially noticed by a very high load of CPU when
 > running periodic tasks.

I may have seen this but no longer have access to the system where it
happened, and there were other issues so I have no confidence that
the high loads were due to the problem described below.  This was
during the initial import of a very large number (~5000) of archives,
and running mailmanweb update_index for the whole thing.  The problem
was that with qcluster running, the whole archive indexing and the
periodic indexing stepped on each other's locks.

We solved this by telling people archive search would be available
randomly over the next few days (ie, when the archiving process
completed that list's archive ;-) and stopping the periodic process
(just renaming the index task in qcluster's queue does this) before
running the index everything task..

I suggest doing this to get your archives close to up to date so the
periodic indexing task doesn't get overwhelmed and fail to complete
with its interval.  I have no confidence this will solve your problem
completely, but the indexing task may get in the way of solving it.

I also recommend getting in touch with the Django community as they're
more likely to know things like how to set logging verbosity in these
low-level components.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Error in `mailman-web conmpress`

[Stephen J. Turnbull]

Abhilash Raj writes:

 > Have they installed all the projects from source? I feel it might
 > be causing the issue if they are using the git head of Postorius
 > but not for django-mailman3.

That turned out to be the problem.

 > This was added recently in django-mailman3 and hasn't yet been
 > released, so using the released version can cause the above error.

I told Durlabh to use either the versions that Mr. Liu used last year
or the most recent released versions so he'd have a base we can expect
to work and play well together, but he was lazy and went ahead using
git HEAD of some of the packages on the argument that it makes syncing
to HEAD easier later. :-(

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Error in `mailman-web conmpress`

[Stephen J. Turnbull]

Hi,

I'm seeing this in our GSOC student's build of the suite.  I don't yet
know if it's related to his changes, just wondering if anyone has seen
it or has any ideas how to resolve it:

% mailman-web compress --settings settings --pythonpath `pwd`/var/etc
Compressing... Invalid template postorius/user/address_preferences.html: 
Invalid filter: 'add_form_select'
Invalid template postorius/user/subscription_preferences.html: Invalid filter: 
'add_form_select'
Invalid template postorius/lists/header_matches.html: Invalid filter: 
'add_form_select'
done
Compressed 2 block(s) from 106 template(s) for 1 context(s).

`add_form_select` is not a standard Django template filter, it's
defined and registered in django-mailman3.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Mailman => Mailmisery

[Stephen J. Turnbull]

thomasa...@gmail.com writes:

 > Mailman is so old-fashioned as name and man dominated in respect to
 > the gender discussions. It is time to rename it to Mail-Miss,
 > MailMom or Mail-Woman for a while?.

This is not going to happen: domain names, systemd services, standard
URLs, package names, all would need to be changed in 10s of thousands
of sites as they upgrade.  Not to mention literally millions of URLs
that point to messages in archives.  None of the above are under our
control, and the string is likely to pop up in all kinds of
unpredictable places.

If you want to be helpful, try your hand at a much easier problem: a
plan for migrating Mailman's git repos from the use of the now-
deprecated "master" branch to something less offensive ("main") or
more descriptive ("development", "release").  That's something that if
done correctly on our end, will propagate to many, perhaps most
downstream repos and forks automatically, and is easy to explain how
to complete the changeover.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Making Hyperkitty more friendly to Search Engines

[Stephen J. Turnbull]

Дилян Палаузов writes:

 > it would be good, if Hyperkitty integrates better with search
 > engines for public archives.

Patches welcome!

There are only about 3 core developers at the moment, and we are
mostly focused on fixing bugs and implementing Mailman 2 features that
are not yet in Mailman 3.  It is very likely that your suggestion will
not get action in the near future.  To keep it from getting lost,
please create (an) issue(s) on the tracker at
gitlab.com/mailman/hyperkitty/issues.

Since you have a bunch of ideas which can be implemented more or less
independently, I suggest that the most effective way to organize them
is to create a "task" issue for each enhancement, and a "checklist"
issue that gives a link to each task issue and a brief description.

Steve


___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Fix for Mailman core #931 log rotation bug

[Stephen J. Turnbull]

Hi all,

I think I have a fix for the log rotation bug
(https://gitlab.com/mailman/mailman/-/issues/931).
I'd appreciate review of the theory of the fix, which is discussed in
detail in the issue.  I've done some live testing on a staging system,
and loggers appear to be reopening their logs as expected after about
15 hourly rotations.

I haven't had time to generate a test.  The environment where I'm
doing the work is really closed, but I'll do the sneakernetting and a
preliminary MR later for convenience in review.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: GSoC 2023

[Stephen J. Turnbull]

Hi Lalit,

(If you prefer to be called something else, a nickname or we should
use an honorific, please let us know.)

The basic procedure for GSoC is

1.  Read our GSoC Ideas page at
https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202023
The top of the page explains how GSoC works in our org.
2.  Think about projects that interest you (even if that means "just
know more about") or that seem appropriate to your existing skills
if you're looking to apply those rather than, or in addition to,
learning new ones.
3.  Get a GitLab account if you don't already have one, and check the
issue tracker at https://gitlab.com for easy or interesting issues
that you can work on (you need to demonstrate you know your way
around our change submission process),  The issue doesn't need to
be code, it can be a typo fix (including one you submit yourself);
the requirement is showing you've done our paperwork.

You don't have to submit a project from our ideas page, although
that's usually the most straightforward, and you shouldn't assume that
anything one the page is necessarily high priority for us.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] [MM3-users] GSOC 2023 mentors

[Stephen J. Turnbull]

Stephen J. Turnbull writes:

 > We now have mentors (Yay! thanks Abhilash and Danil) and I've
 > applied to PSF as a suborg.  More mentors are welcome (that allows
 > us to support more projects).

And Python has accepted us!

The ideas page is
https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202023
If you have suggestions for projects, it's probably best to discuss
them on mailman-developers@python.org (reply-to set) first because
Google has restrictions on some of the tasks that could be eligible
(for example, documentation projects have not been eligible in the
past--any documentation had to be relevant to code that was sufficient
to justify a project in itself).

However, if you have some experience with Summer of Code and have
write access to the wiki, you can add ideas directly.

In both cases it's useful to getting them implemented to volunteer as
mentor or backup mentor ;-) but that's not required.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] [MM3-users] GSOC 2023 mentors

[Stephen J. Turnbull]

Stephen J. Turnbull writes:

 > We still have no mentors for GSOC 2023.  Last chance to sign up
 > (with the Python Software Foundation org) is Feb 25.

We now have mentors (Yay! thanks Abhilash and Danil) and I've applied
to PSF as a suborg.  More mentors are welcome (that allows us to
support more projects).

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GSoC 2023

[Stephen J. Turnbull]

Hi all

Applications for GSoC 2023 are open and I would like to apply.
Program details: https://summerofcode.withgoogle.com/programs/2023
You may need to log in with a Google account.

We could go under the PSF umbrella again (they've applied already), or
we could go it alone.  The advantage to the PSF is we can possibly
borrow a mentor from them as we did in 2022 and they'll do some of the
org admin stuff, the advantage to a separate Mailman org is money and
tickets to the Mentor Summit if they do that.  There may be other
stuff I haven't thought about.  If we decide to apply ourselves but
get cold feet, I suppose we can withdraw without penalty and go with
PSF if we do so a reasonable time before deadline (Feb 8).

I need some backing from core devs (at least one backup mentor, which
is light work if I'm healthy), but I plan to post to the lists to see
if any of the usual suspects would like to be mentors, too.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Preventing abuse of postorius

[Stephen J. Turnbull]

Phil Budne writes:

 > Almost every day I find outgoing emails from postorius stuck in my
 > outbound mail queue. It seems like the anonymous_subscribe route is
 > being used to send mail to unwitting targets.

It's almost like that's what it's for, isn't it?  The problem, of
course, is that for our purposes ability to read mail at an email
address is a near-perfect way to identify subscribers.  So for a new
user, getting that identification involves confirmation via a round-
trip email conversation.

The problem here is that anonymous subscription via web has no
relation to an email address, and confirmation requires sending an
email to the address requested.  As Mark points out, removing the
anonymous subscription page is a way to prevent this attack.  But
that's kind of unpleas

Of course it's also trivial to spoof an address in email.  So I see
three ways to allow an as-yet unknown user to subscribe with relative
security:

(1) Authenticate web users through something like OAuth2 or SAML at
their email provider. 

(a) This could be generalized somewhat to allow trusted idPs to
authenticate any email addresses that have been associated with
that ID, but you'd have to make sure that only trustworthy idPs
that only provide authenticated emails are on the list.

(2) Disable the ability to anonymously subscribe through email, and
only allow the "From" address to subscribe, and only if it passes
DMARC-like From alignment.

(3) Transitive trust of existing authenticated users, who could sign a
request.  Surely this would have to be rate-limited both in number
of attestations allowed to an existing user, and via a delay in
allowing attestations from newly authenticated users (which could
be measured in weeks or in posts), but it might work.

I see Mark has already posted a proof-of-concept for disabling
anonymous subscription via web.  I think I will add these tasks to our
Summer of Code list, but if you want to take a crack at any of them,
be my guest.

 > I'm a Python programmer, but not a Djangoist.

If that's a potential offer of code, I love it!  (1) would probably be
most useful when done via Django socialauth mechanisms (but this isn't
really "Django programming"), although there are alternatives like
Shibboleth for SAML idPs that I think have Python bindings.  But I
think (2) is probably within reach of any Python programmer since the
toolbox (SPF, DKIM, DMARC) is already available in any complete
Mailman 3 installation.  I'd be happy to point you at the relevant
tools and existing Mailman client code for them.  (3) also also pretty
generic, but I don't know anything about availability of email signing
tools for "web of trust"-based PKI.  Pretty sure they're out there,
though.

Steve


___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: [ext] Re: before next release: disable backscatter indefault installation

[Stephen J. Turnbull]

Ralf Hildebrandt via Mailman-Developers writes:

 > This is probably due to an error in local_recipient_maps in Postfix.
 > https://www.postfix.org/LOCAL_RECIPIENT_README.html

Reading the documentation for Exim4 and Postfix these days, I have to
wonder if there isn't room for a new MTA with a sane configuration
language.  Feels like regressing to sendmail.cf levels of obfuscation.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Code formatting tools for Mailman project

[Stephen J. Turnbull]

Abhilash Raj writes:
 > I don't see the tools actually supporting formatting only-diffs.

I don't know if git diff supports everything GNU diff does but
--ignore-all-space would likely help quite a bit.

It shouldn't be that hard to write a blue-diff command for git:

1.  create a temporary branches at the relevant commits
2.  for each temporary branch
2.1 checkout the branch
2.2 run blue on the relevant .py files and commit
3.  diff the temparary branches and save the output
4.  delete the temporary branches

I don't know if this would help in practice, but the theory seems
good.

Scripting that should be easy, and I can test it on the Postorius
code.  I'm not going to get to it before December, though, so if you
want to do it first :-D

 > It _can_ be done by piping all the updated files to `blue` command,
 > but then reverting style changes in parts of file where there
 > wasn't any real change made would be more work than just fixing the
 > flake8 errors manually.

Is that necessary?  Since the desired result is reformatted files
without impairing the ability to diff, why would we revert the style
changes?

 > I am trying to think if we can make that process better, after making a 
 > one-time code formatting change, through tooling or alternative commands 
 > to grok the diffs.

Additional tools to parse the diffs might be useful.  Another
possibility would be to diff the AST. :-O

 > Are you usually looking for just anything that looks odd or typically 
 > any kind of difference that looks suspicious or just trying to learn the 
 > differences?

Both of those are common.  Typically it's something where I can't come
up with a bisectable test, eg, a GUI regression.  Then it's wait for
rebuild, fire up the app, mess with the GUI, repeat ... so I don't
actually bisect, once I have a bracket I just take that diff.

 > Can just comparing the commits/commit messages?

git diff --stat often helps isolate to a file.

 > or Merge Requests merged since the feature branch would help since
 > we mandate all changes go through MR workflow?

I don't see how that helps if you're looking at a dozen commits
touching that file on main.

 > I don't know if I can speak much about comparing raw diffs, because I 
 > very rarely end up doing that.

I try to avoid it, but about even in the large code bases I know best
(XEmacs, Lisp and C) half the time I end up looking at diffs.

 > I am usually looking at commit level data and looking for MRs that
 > last changed the point I am interested in with `git-blame` for some
 > contextual data around the MR. Merge commits have the MR no to
 > easily track where the change came from.

I rarely can pinpoint the relevant code without a diff, though.  Most
of the time the bad data is several frames up the stack, and was
generated far from the visible error.

Also, you and I are likely to be working on code we just wrote or
reviewed.  We need to consider the drive-by contributor who is not
going to know our code well.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Code formatting tools for Mailman project

[Stephen J. Turnbull]

Abhilash Raj writes:

 > > I (speaking only for myself) have no objection going forward for new
 > > files and major refactorings.  But if you do this all at once, I fear
 > > that most interesting diffs for the next year or so will be full of
 > > reformatting cruft, 
 > 
 > I am not sure I understand this part. Do you mean to say we'd be getting 
 > too may PRs with "formatting-only" changes?

No, I mean the diffs we do to localize errors.  The day after you
merge the reformatting PR, anybody who wants to identify a regression
since Postorius 1.3.7 is going to get as many lines of " -> ' as they
do of code changes.  Half the time what appears to be a regression
from future 1.3.8 to HEAD is going to turn out to be older than the
reformat merge, same problem just less frequent.  It gets less
frequent over time, but it will take quite a while before the reformat
merge consistently contributes less than 10% of changes, I expect.

 > Currently, my thought is to do a one-time format for the entire codebase 
 > and then add "blue --check" as a part of the "qa" CI gate. Then, for 
 > each subsequent MR that follows, people would need to run "tox -e 
 > format" before they commit and create MR. The hope there is also

Yes, this is a great idea for new files, or files you're refactoring
to death anyway.  But it has a potentially large downside of turning
diffs that should be one-line changes into rafts of fixquote changes.

 > My motivation here is to not have to push "oops, make flake8 gods happy" 
 > commits on top of my PRs and not have to worry about it when writing code.
 > 
 > While it isn't too huge of an issue, it is still manual effort that I 
 > feel can be automated to reduce some work and save time.

Sure, but doing it globally also makes some work.  The question is
does it make more work (and annoyance) for everybody who does diffs
than is experienced by the developer who needs to make a flake8
commit?  I do a lot more diffs on Mailman than I do commits.

 > > What is the purpose of this?  Anybody who runs it on a code base
 > > before the blue'd code gets merged is going to generate hard to read
 > > crufty PRs, no?  
 > 
 > How so? If the existing code base is already auto-formatted, then the 
 > PRs would be just regular diffs, formatted with the same tool.

Maybe tox -e format isn't a problem because you would add that tox
target in the same commit (or later) you do the reformat, but if this
is going to be policy, some people will aggressively reformat their
code to that standard before the package is reformatted.  Every once
in a while you still see MRs in Python where somebody PEP8s a whole
stdlib module where the contribution is a 2-line doc improvement.
Those get rejected of course.  We should do that too.

 > I've tried to solve at least the "git-blame" issue by adding the rev of 
 > the commit with the "mass-refactor" into a file in the main repo that 
 > you can feed to "git blame --ignore-rev-file .git-blame-ignore-revs".
 > 
 > For editors, there is a way to set git config for the blame-ignore-file 
 > so it works across everything that uses git.

That's very helpful for some operations.  How to make it discoverable
is something of a headache though -- I didn't know there was a
blame-ignore-file!

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: ARC user options

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > It would also be possible to link DB tables,

No, it's not.  It's all one row (IIRC).

 > or to define triggers that replicate insert/ update/ delete on a
 > number of tables/ fields.  

Which is exactly the complexity I don't want in Mailman if we can
avoid it.  Keep it to flat tables in approximately normal form.

 > The question is how much more insight than average list keeping
 > would be needed to do it.

"Too much." :-)  The simple "message per subscriber + per-subscription
'munge' flag" is just so much better.  For example, there are at least
three conceivable values: no munge, munge all, munge p=reject (and I
think Mailman actually implements munge p>=quarantine as well!)  This
gets very tedious in the twin-list implementation.

 > Would this approach make sense with Mailman 2?

The umbrella + twin lists approach is perfectly possible with Mailman
2, but the admin has to implement it themselves.  We are not going to
implement it or release it.

 > If the site policy is to accept posts from subscribers, it needs to
 > inspect the union of sub-lists subscriber sets.  How could that be
 > accomplished?

There's a "sibling list" feature for exactly this purpose.

 > > 4.  List-A-munge gets From munging for all posts, List-A and
 > >  List-A-nomunge never get From munging.  (In theory List-A-munge
 > >  could do munging only for p=reject posters, but always doing it
 > >  probably makes it easier for subscribers to maintain their
 > >  filters.)
 > 
 > How difficult is that to set up?

cost of 1 list X 3.

 > IMHO, it becomes overly complicated.

So don't do it.  Others will, however -- isn't that what
"decentralization" is all about? -- and some of them are quite good at
it.  Why not take advantage of that to make at least some mail flows
cleaner and more useful?

 > DMARC was thought so that From: bank.example can hardly be faked.

Yes, and that's still true.

 > Allowing fuzzy overrides is much like getting back to content
 > analysis.

Fuzzy overrides are not "allowed".  They *happened*.  Gmail did it
from the get-go.  RFC 8617 is recognition that it does happen, and a
protocol that purports to improve the accuracy of overrides.

 > I'd mark as trusted only a few domains, based on personal knowledge.

Have you analyzed your mail flows to see if there seem to be frequent
messages with multiple DKIM breaks?  AFAICS, in practice *you* as an
individual will need to trust your mailing lists because that's the
only place signatures are going to be invalidated, and you can demand
everybody else has to pass DMARC.  This is no different from before,
except replay attacks via mailing lists are going to be harder.  For
large sites with many users with diverse mail flows, the benefits of
both ARC and reputation systems are much larger.

 > It is not the cost.  To have a global knowledge of the Internet you need to 
 > have a user base that is statistically relevant with respect to the global 
 > population.  That is, you have to be Google, or Microsoft, or
 > Yahoo!, ...

That's not true.  Bayesian filters work well for almost everyone.

 > > If anything, it's the opposite for the mailing list community, because it 
 > > makes it easier for an independent list host construct and maintain its 
 > > reputation, and should get it better treatment from those with reputation 
 > > systems.
 > 
 > Yes.  However, I think that a list that experiments non-munging will be 
 > whitelisted sooner by small, personal sites who trust it than by large orgs 
 > computing its reputation.

They're *already* whitelisted by the small personal sites who trust
it.  The critical question is how fast are the large orgs going to
learn to trust small ARC participants, because it's exactly those
large orgs that are the root of all evil^Wer, most nondelivery
problems that we small sites experience.

ARC and DMARC are *not* targered at *us*, though if the large orgs use
them effectively we will benefit.  They're for large sites with
hundreds of thousands (and sometimes billions) of users who are
targeted by ransomeware hackers and national espionage agencies.

 > > This would require the MLM site to maintain a separate site.
 > 
 > It can be a dummy subdomain, a few lines in a zone file.  I'll change that 
 > line 
 > to "from a (sub)domain having p=reject", to have it more apparent.

Yeah, I was in a hurry.  Thing is, there are a lot of inexperienced
folks out there who would just send mail from "bi...@whitehouse.gov"
or something like that. :-)

 > Hm... a list SHOULD reject posts arriving with an ARC chain, valid or not. 
 > Shouldn't it?  I see no reasons to post indirectly (except for internal 
 > list-to-list flows which don't need ARC seals).

This is *Internet mail*.  "NO REASON" is its slogan!  But here's my
personal use case: I use a Japanese telco as my home ISP but use my
server at my employer as smarthost.  My employer (research university)
doesn't know or care, but they do 

[Mailman-Developers] Re: ARC user options

[Stephen J. Turnbull]

First let me make clear that (1) I do have influence on Mailman's
position here but (2) I am not authoritative and (3) Mailman has no
position yet.  I'm discussing this and that and we'll see where my
position and eventually Mailman's come out.  So anything I say may be
wrong (always check my logic ;-) and I may change my mind. :-)

Alessandro Vesely writes:

 > It is the MLM as a whole which has to conform, if it wishes to participate. 
 > Not the mailing list software.

If you mean the decision is list by list, conformance doesn't mean
much -- the subscribers still need to learn the rules list by list,
most of them won't know what "RFC " conformance means, and other
sites interacting with such a site will need to check the conformance
of lists individually.

On the other hand, if you mean site-wide, if it were implemented in
Mailman (and other software), conformance would be much more likely
and much more likely to be site-wide.

 > I push ARC as the authentication method because that was the major objection 
 > to 
 > using Author: (the "simple" method in the old version.)

Yes, I agree, authentication is important, and ARC provides validation
of the right data for some purposes.  I'm not sure it "does what you
want", but I do "want what it does".

 > > Maintaining synchronization of configurations of two lists will be tedious 
 > > for the admin, or involve relatively complicated coding if we arrange to 
 > > automatically mirror configuration changes.
 > 
 > Couldn't symlink most stuff?

I don't think there's anything to symlink.  In Mailman 3 all of this
configuration information is in an RDBMS like PostgreSQL, and routing
of posts and modification of messages (both bodies and headers).

 > I'm not clear how that would work.  Would you expand?

1.  lis...@example.com has two subscribers:
list-a-mu...@example.com
list-a-nomu...@example.com
List-A-[no]munge accepts subscriptions according to site and list
policy.
2.  List-A is configured not to allow other subscribers under any
circumstances.  List-A-[no]munge accept subscribers under the site
and list policy.
3.  List-A-[no]munge refuse all posts, and advertise List-A as the
destination for posts.  List-A accepts posts according to site and
list policy.
4.  List-A-munge gets From munging for all posts, List-A and
List-A-nomunge never get From munging.  (In theory List-A-munge
could do munging only for p=reject posters, but always doing it
probably makes it easier for subscribers to maintain their
filters.)

 > > https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09 
 > > would be the natural home but it's expired, so it doesn't do any harm 
 > > to have it in your draft.
 > 
 > What I dislike of that document is its considering the availability of a 
 > global 
 > reputation system as a widespread feature of all mail servers,

90% of the email users on the Internet are served by organizations
that can afford comprehensive and reasonably accurate reputation
databases and update algorithms.  (Whether they do bother with
accuracy is another question.)  So I think it's reasonable to ask "how
does a reputation database affect this feature" several times.  For
the rest of us, there are less sophisticated but still useful shared
reputation databases (ie, the RBLs), and local databases such as
SpamBayes can be useful.

 > while only the known giants actually have one.  In that respect,
 > ARC is a centripetal protocol, which is why I've been opposing it
 > until this attempt.

Everything is centripetal, because the only way we really know how to
scale networks while maintaining discoverability is hierarchically.
All reasonably decentralized networks have a (usually very expensive)
centralized system at their foundation.  I don't see ARC as being
particularly biased toward centralization, just because powerful
reputation systems are expensive.  If anything, it's the opposite for
the mailing list community, because it makes it easier for an
independent list host construct and maintain its reputation, and
should get it better treatment from those with reputation systems.

 > Should I add that it's out of scope to speculate how users can convince 
 > their 
 > mailbox provider to trust/ whitelist a given MLM?

I think that's always out of scope.  It doesn't hurt to add it, but
technically its out of scope for an RFC.

 > 3.  The no-munging method
[...]
 > *  Have an umbrella list with two subscribers, the twin lists.  The
 >twin lists would be configured to refuse subscriptions and posts
 >from non-members.

Whether to refuse posts from non-members is independent of the
no-munging protocol.

 > Before allowing subscription to a non-munging list, a MLM MAY test
 > that a recipient effectively receives its messages by sending a test
 > message with a broken signature from a domain having p=reject.

This would require the MLM site to maintain a separate site.
Otherwise 

[Mailman-Developers] Re: ARC user options

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > Internet-Draft MLM TransformationsSeptember 2022
 > 
 > 
 > 3.  The ARC method

This twin-list proposal doesn't depend specifically on ARC.  Any
subscriber who thinks they know better than the DMARC protocol can
choose to ignore the DMARC policy, and they would benefit from the
twin-list proposal.  For example, if the un-authenticated received
chain indicates that the MLM received the post directly from an
author not in the MLM's domain, and I trust the MLM, and its DKIM
signature is valid, I would accept that mail and ignore DMARC.  Of
course if the MLM host provides a sealed AAR field, that's much
better, but the one-hop-to-MLM scenario is very strong evidence of
authenticity if the MLM is very trustworthy.

 > For each list with From: munging enabled, a participating MLM MUST

To be honest, I don't think that Mailman would be willing to conform
to this.  I haven't seen the rest of your draft, but this section is
really more of an Informative/BCP RFC than Standards Track.  That is,
this is an option that list creators can already exercise with the
cooperation of the list owners, so there's no need to impose
requirements on MLMs.

If this option gets traction (ie, some vocal fans, I don't ask for a
huge number), then I think Mailman would be willing, and would prefer,
to go all the way to recipient-controlled From-munging as you
originally suggested.  Maintaining synchronization of configurations
of two lists will be tedious for the admin, or involve relatively
complicated coding if we arrange to automatically mirror configuration
changes.  A per-subscriber option for From munging would be simpler to
develop and maintain.

 > configure a twin list with From: munging disabled.  Both lists have
 > the same posting address, but separate subscriber lists.

I don't think having the same same posting address is likely to work.
Most MLMs probably won't implement at all, because list creators can
do it for themselves by having an umbrella list with two subscribers,
the twin lists.  The twin lists would be configured to refuse
subscriptions and posts from non-members.  On the other hand, doing
all this cleanly is more complicated than implementing recipient-
controlled From munging.

 >   Subscribers
 > who think that their mailbox provider runs a suitably configured
 > DMARC filter can subscribe to the twin list.  Users subscribed to
 > both lists receive double messages until they unsubscribe or suspend
 > delivery from one of them.

Recipient-controlled From-munging would also prevent unnecessary
double messages.  It's just cleaner.

This next paragraph seems unrelated to recipient controls on From
munging.  It seems pretty obvious, and is ARC-specific.
https://datatracker.ietf.org/doc/html/draft-ietf-dmarc-arc-usage-09
would be the natural home but it's expired, so it doesn't do any harm
to have it in your draft.  Have you considered reviving that document?

 > DMARC local policy overrides is one of the use cases that [RFC8617]
 > provides for ARC.  It requires that a DMARC filter be configured to
 > accept the authentication assessments provided by a verified ARC
 > chain when all of the domains involved are marked as trusted.  In
 > that case, the filter overrides DMARC policy and acts as if the
 > current Authentication-Results: were the ARC-Authentication-Results:
 > (AAR:) of the first ARC set (i=1).  Normally, a MLM SHOULD apply
 > DMARC policy on message arrival, so the first AAR: is expected to
 > have dmarc=pass.

This paragraph also seems unrelated to recipient controls on From
munging, and is not ARC-specific.

 > MLMs which in turn trust third party domains, such that they override
 > DMARC failures in posted messages, MUST

This "MUST" isn't going to work.  The MLM software can't ensure this,
and MLM operators will do as they please.

 > communicate the list of trusted domains to subscribers

Currently most sites (MTAs) operate content filters and blacklists,
but not whitelists, and the MLMs trust their sites.  I don't see why
that needs to change with ARC.  Either way, I'm pretty sure those most
sites would be unwilling to publish those lists and keep them up-to-
date, and the MLMs don't have them.

Note that it's quite rare for the MLM to trust anybody under current
conditions.  Most mail flows to MLMs are direct, and for MLMs that
host open subscription lists, they can be from anywhere.  So MLMs
expect their sites to provide content filtering and sender
authentication, not trust (reputation) algorithms, and the MLMs don't
participate in maintaining any of these facilities.

 > when they announce the creation of the
 > twin list, and on subsequent modifications.  How users can query the
 > list of domains trusted by their mailbox providers is beyond the
 > 

[Mailman-Developers] Re: ARC user options

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > The tale goes that large mailbox providers want ARC as a tool to
 > filter mail streams from lists that don't do a good filtering
 > themselves.  ARC, they say, allows to attribute reputation
 > correctly.  However, I don't think they can tell who a message is
 > from, after it has been munged.

I don't see why they would care who the message is from.  They know
who added the ARC seal who claim to have validated the original From.
If any of them are highly trusted, or the whole chain is unbroken and
none is known untrustworthy, that should be good enough.

 > In that respect, From: munging hampers ARC.

Sure, a little, especially if the recipient doesn't understand the
theory.  On the other hand, there are only a few MLMs that munge, they
should be able to extract the original address in most cases.

 > A MLM would continue From: munging unless a receiver is able to tell it to 
 > not do it.

Sure, but as you say you need to trust the users, in this case, the
receivers.  If it's all I know about them, I wouldn't trust a user who
uses a "p=reject" provider to be clueful enough to make that decision
safely.

 > I see that archived messages are not munged.  How come?  Isn't the
 > archive a regular subscriber?

In general, no, they're not.  Messages are sent to archive services
before the munging process takes place.  There are services like
mail-archive.com where you interface with the archiver as a regular
subscriber, but Pipermail (Mailman 2) and HyperKitty (Mailman 3) both
short-circuit the pipeline.

 > At worst, one could set up two lists, fed by the same stream, one
 > with munging enabled and the other not, letting users subscribe to
 > the one they prefer.

To be honest, while I'm at best 50% willing to implement the user
option, I could easily be persuaded by a few experiments with the dual
list proposal.  This could be implemented almost transparently for the
subscriber (except at subscription time) by using an umbrella list.

 > A list can set the Author: header field by copying From:.

This isn't a problem.  I would be perfectly happy to accept a patch to
handle Author today as long as it's RFC-ly correct, which looks
straightforward.

 > MUAs won't notice Author: fields any time soon.

Then they're useless.
1.  They need to display them, or most users won't be able to use them
in any way.
2.  It's important that they be included in automatically formatted
replies, and the munged From ignored.
3.  You'd hope From would either not be displayed or the content of
the Author field substituted into From.

 > > Translated into Scum-of-the-Earth-Spammer:
 > > 
 > >  You can use this header to send "referred by someone in victim's
 > >  contact list" (that you stole from Yahoo) spam and it will bypass
 > >  DMARC v1 because you can use an aligned From.  All-Hail-Author!
 > 
 > They're doing that with From:, and it works fine.  It is very hard
 > for MUAs to tell spoofed From:'s,

True, but DMARC should give essentially zero false negatives.  If it
says from is aligned, and the sending domain is reliable, it's not
spoofed.

 > Then, some people hold that even writing THIS IS PHISHING loud and
 > clear won't prevent users from opening and clicking that link.
 > Darwin will tell...

See the recent Twitter thread by @FatManTerra, who performed a very
cruel experiment where he advertised an obvious scam "for the purpose
of making people scammed by Terra whole" and amassed > $100,000 in
BitCoin in 24 hours.  (He returned it all, but it's unethical and must
make his victims feel like idiots.  I don't disagree with their
self-assessment, but it wasn't nice to rub their noses in it.)

 > Implementing and deploying ARC is not enough, you also need to
 > trust the ARC signer/ sealer.

I see it the other way around: once people have implemented ARC, you
have the opportunity to serve your users by trusting the ARC sealers.
We didn't really have that before, because of the ease of creating
fake mediators.

 > ARC won't be effective until it has been deployed at more than 60%
 > of SMTP servers and that's not a problem. :-)
 > https://www.rhyolite.com/anti-spam/you-might-be.html#senior-IETF-member-5

I doubt 60% of servers will implement within the decade, but I guess a
lot more than 60% of legit mail is handled somewhere by an ARC-
conforming server.  Google has long been a powerful advocate of ARC.
I think there's hope.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: ARC user options

[Stephen J. Turnbull]

Alessandro Vesely writes:
 > On Sun 04/Sep/2022 13:38:39 +0200 Stephen J. Turnbull wrote:

 > > Not sure what you mean by "sealing".  Do you mean they're not 
 > > implementing the rest of the protocol?
 > 
 > They add a complete ARC set.  Actually, they add three ARC sets to each 
 > message, one at reception (with a full ARC-Authentication-Results), followed 
 > by 
 > intermediate and final sets.

? That's not how this works.  You add (ARC-)Authentication-Results
immediately after the message is received from a remote, and the
ARC-Message-Signature and ARC-Seal immediately before transmitting it
to a remote.  I guess if you do the full set including full
authentication at each stage, it doesn't harm the protocol, but it
does impose a fair amount of unnecessary work on each recipient.

 > Thank you for that much needed clarification.  I heard someone
 > saying that the IETF was waiting to implement Mailman 3 to use ARC
 > in mailing list...

That seems unlikely.  Unless you're running a single-host system, it's
much preferable to do it on border MTAs, rather than in Mailman.  I
suspect it's more that they plan to do a big revamp of infrastructure
including Mailman 3 and they would do it at that time.

 > BTW, there is also a Perl implementation of ARC included in
 > Mail::DKIM.

Thanks, I'll keep that in mind.

 > Exactly.  I asked bind-users if anyone verifying ARC saw any difference 
 > after 
 > trusting isc.org.  Besides adding ARC sets, bind-users do From: munging, 
 > obviously.  Nobody saw any difference.

As far as I know ARC adds nothing if you're also doing From munging.
The point of ARC is entirely to get rid of From munging.  Once you've
munged From, your DKIM will be valid and you have DMARC from
alignment.

 > The point is how does Mailman know whether a recipient's MX trusts this 
 > particular list.

It doesn't.  Recipients can do any damned thing they want.  In the
case of AT and Verizon, pretty much everything they do is damned.
They frequently block all traffic from well-behaved lists because
there's a spammer in the same netblock, or just randomly.

 > And what does it do when it knows.  Some people babble something
 > about DNS records, which looks difficult.  Another possibility
 > could be an SMTP extension, difficult to implement as it involves
 > multiple levels.

I really don't see the bad actors in this space (by which I mean the
ISP-based freemail providers) doing either, since they don't even send
"we hate you spammer!" DSNs, they just black hole your list traffic
and any attempt to complain about it.  If services sent honest DSNs,
we could discount bogus 550s and not count them as bounces.

 > An easy way would be to ask the subscriber whether to do From:
 > munging or not.  Then, a user can disable From: munging for the
 > messages destined to her. That's easy for those who run their own
 > MTA.  People using Gmail, say, would have to figure out, presumably
 > by trial and error.

I'm dubious.  People are going to get their subscriptions disabled by
experimenting.  I don't know how many people still want
non-personalized lists, but implementing that would require a bit of
effort since two messages would need to be prepared depending on
whether don't-munge-for-me was set.

 > If such an option is not given, a mailing list could add the
 > Author: header field defined by RFC9057.  Receivers could restore
 > From: after DMARC filtering.

This can't work.  DMARC v2 will quickly be forced to check Author
alignment.  From the RFC:

In that regard, it would be reasonable for an MUA that would
normally organize, filter, or display information based on the
From: field to give the Author: header field preference.

Translated into Scum-of-the-Earth-Spammer:

You can use this header to send "referred by someone in victim's
contact list" (that you stole from Yahoo) spam and it will bypass
DMARC v1 because you can use an aligned From.  All-Hail-Author!

 > That assuming that someone is willing to do something to avoid
 > munged From:'s, which I'm beginning to doubt.

If ISPs cared at all about their customers, they'd implement ARC and
be fine.  It completely solves the problem by putting it on mailing
lists and other mediators to filter spam before sealing.  The problem,
as always, is recipient domains that are unwilling to publish
consistent policies or respond to non-delivery issues.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] ARC user options

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > There is a thread about ARC sealing in bind-users[*].

Not sure what you mean by "sealing".  Do you mean they're not
implementing the rest of the protocol?

 > They're applying ARC signatures, although they run Mailman 2.
 > It doesn't seem difficult to implement.

It's not.  But

1.  It's a bad idea to do it in Mailman.
2.  It was implemented in Mailman 3 three or four years ago as a proof
of concept during the development of ARC.
3.  There is a milter available for Postfix and Sendmail from the
Trusted Domain Project https://github.com/trusteddomainproject/OpenARC
as is the basic implementation which I presume is adaptable to
Exim, qmail, and other MTAs.

This is the preferred approach, as matter of conformance because
it should be implemented by the edge MTA(s), and as a practical
matter because Mailman *can't* do SPF since it is never an edge
MTA.  There is also a pure Python implementation on PyPI, I
believe (this is the basis for the Mailman 3 plugin, or maybe it
was dkimpy).

 > It requires trusting the users, though.

I don't think so, not any more than any other sign-and-send protocol.
What it requires is implementation by recipient domains who trust your
host, because if they don't it's 2014 all over again for your
subscribers if you have any DMARC p=reject posters.

 > Would Mailman implement something like it?

Yes for Mailman 3, it's already done (but you are recommended to
configure it in the MTA).  No for Mailman 2, it's EOL.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: GSoC for working professionals

[Stephen J. Turnbull]

Christopher writes:

 > The idea page mentioned a pubsub model, which to my understanding
 > it's to use an event queue where Core is the publisher, and
 > Postorious is the subscriber. Does this mean building a pubsub
 > model using some Python libraries?

I believe so, but I'm not familiar with them.  Again I have to defer
to Abhilash.  You might be able to catch him on #mailman @
libera.chat.  He's in PST time zone (-0700) and usually keeps a 9-5
workday, I think.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GSoC for working professionals

[Stephen J. Turnbull]

Christopher writes:

 > I am not a student, but I am working as a software engineer full
 > time (10-6 on weekdays). I would like to start contributing to open
 > source projects and stumbled upon GSoC which I thought it's a great
 > opportunity to get started. If I am able to churn out 14.5 hours of
 > work per week (by working on the weekends) across 12 weeks (summing
 > up to 175 hours), do I stand a chance to be accepted as a GSoC
 > participant?

You stand a chance, yes.  I don't know if we have specific preference
for students, I'll have to discuss it with Abhilash, but we're not
going to rule you out because you're working already.  At present
you're the only person who's expressed interest, too. :-)  (That
doesn't mean nobody's downloading code and looking for easy patches,
though.)

In a later post you mention the "REST Callbacks" task.  I think that
was Abhilash's idea, I'll see if I can get him to flesh it out a
little.  But the basic idea is exactly as stated in the ideas page:

   Request   Request
+--+  1  +---+  2  +--+
|  |>|   |>|  |
| User | | Postorius | | Core |
|  |<|   |<|  |
+--+  4  +---+  3  +--+
  Response   Response

So this means that, for example, Postorius can't present a real-time
dashboard for moderators that shows whether a list has held posts that
the moderator needs to check and approve or reject.  The system needs
to start with a request at 1, pass it on to 2, then core responds at
3, and finally Postorius passes it back to the user at 4.  What we'd
like to be able to do is register a callback with core so that when it
receives and holds a post, it tells Postorius without needing a
request, and then Postorius would update the dashboard.

The dashboard (Postorius <--> User interaction) is not part of this
project.  At present we don't have such a real-time interaction
between Postorius and the user.  But for that to be most useful,
rather than have Postorius continuously polling core with "are there
held message? are there held messages? are there held messages? ...",
we want to use such a callback mechanism.  The REST API callbacks are
a first step in that direction.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: GSoC for working professionals

[Stephen J. Turnbull]

Please note that the person who wrote this is not a core developer in
Mailman or participant in GSoC on the mentoring side.  That may change
if they volunteer, of course (you don't have to be a core dev to
mentor, although the final merge to the distribution mainline needs to
be by a core dev).

The official Mailman GSoC ideas page is at
https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202022
Proposals there have been somewhat vetted for "size" and "difficulty".

Other than that, I don't have time right now to either endorse or
deprecate any of the suggestions in this post.  If you find them
interesting, feel free to discuss them.

Дилян Палаузов writes:
 > Hello Christopher,
 > 
 > the eligibility criteria by the sponsor are at
 > https://developers.google.com/open-source/gsoc/faq#what_are_the_eligibility_requirements_for_participation
 > They do not seem to exclude your use case.
 > 
 > If you are willing to deliver results, I do not think that anybody is
 > going to prohibit you from participating in GSOC.
 > 
 > https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202022 contains a
 > list of Mailman3-Ideas for GSOC.
 > 
 > In the meantime you can setup
 > mailman+mailmanclient+postorius+hyperkitty+mailman-
 > hyperkitty+django_mailman3 on your system.  There are different way to
 > get the setup.  You can also submit some trivial improvement, like
 > grammatical corrections of the documentation.
 > 
 > I personally installed mailman few weeks ago and have not deployed it
 > yet in production.
 > 
 > 
 > The MLM I use currently has a notion of Sub-List: all subscribers of a
 > mailing list A, are automatically subscribers of mailing list B, if B
 > is a sub-list of A.  Moreover, the subject-prefix all subscribers get
 > is the one of the mailing list A, when a mail is sent to A (and
 > received by B-subscribers).  One of the concerns migrating to mailman3
 > for me is the lack of this feature (or at least I have not found it
 > yet).
 > 
 > 
 > Some email providers offer “Feedback loop”.  That is, when a mail is
 > sent from your server to their server and a user clicks that mail
 > explicitly as spam, your server gets a notification (in “Abuse
 > Reporting Format”) and can unsubscribe the user from a mailing list,
 > when the ARF message contains the user, and the mailing list name.
 > 
 > For the closed-source MLM I use I have implemented the above
 > functionality and I would like to get it working for Mailman.  That
 > would be, upon receiving an ARF message, sending a POST-request to the
 > REST API, containing:
 > - the subscriber address
 > - the returned message
 > - the mailing list name
 > 
 > Then Mailman shall unsubscribe the address from the mailing list, and
 > send a (templated) message to the subscriber, telling her which email
 > message she has marked manually as spam, why she was unsubscribed and
 > how can she subscribe again).
 > 
 > 
 > Some email providers are more stubborn in sending the ARF-messages, as
 > they delete the address of the subscriber, but keep the original
 > Message-Id header.
 > 
 > So when an ARF-message is received from such providers, it shall be
 > possible to decode from the Message-Id the real recipient, and trigger
 > the un-subscription process for the decoded email address.  The
 > encoding of the ML-recipient in the Message-Id header is to my
 > knowledge currently not possible.
 > 
 > So one idea for GSOC is to extend mailman to handle ARF-messages (or
 > parts of it), which arrive over the REST-interface.
 > 
 > The rationale for the REST-interface is, that all ARF-messages for a
 > server are send to a single email address and the software handling
 > that address can decide, if it is a mailman-originating message and
 > forward it to mailman, otherwise handle it differently. (In fact, the
 > software could also redirect the message over LMTP to a special
 > address, but getting a message over LMTP of REST is almost the same
 > here).
 > 
 > Greetings
 >   Дилян
 > 
 > 
 > 
 > On Sun, 2022-03-20 at 15:20 +0800, Christopher wrote:
 > > Hi everyone
 > > 
 > > Nice to meet you all! I am interested in GSoC 2022 and I discovered
 > > that
 > > Mailman might be one of the interesting projects I would like to
 > > contribute
 > > to. However, I am not a student, but I am working as a software
 > > engineer
 > > full time (10-6 on weekdays). I would like to start contributing to
 > > open
 > > source projects and stumbled upon GSoC which I thought it's a great
 > > opportunity to get started. If I am able to churn out 14.5 hours of
 > > work
 > > per week (by working on the weekends) across 12 weeks (summing up to
 > > 175
 > > hours), do I stand a chance to be accepted as a GSoC participant?
 > > 
 > > Thanks,
 > > Christopher Chong
 > > ___
 > > Mailman-Developers mailing list -- mailman-developers@python.org
 > > To unsubscribe send an email to mailman-developers-le...@python.org
 > > 

[Mailman-Developers] Re: --- in DMARC From: rewritings

[Stephen J. Turnbull]

Дилян Палаузов writes:

 > Since other MLMs have the option to include the full original-From:
 > address in the display-part of the From: header, mailman shall have
 > this option, too.

I'm thinking about it, but the question is "How?"

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: --- in DMARC From: rewritings

[Stephen J. Turnbull]

Mark Sapiro writes:
 > On 3/17/22 10:19, Stephen J. Turnbull wrote:
 > > Дилян Палаузов writes:
 > > 
 > >   > realname = re.sub(r'@([^ .]+\.)+[^ .]+$', '---', realname)
 > > 
 > > The obvious hack is to change '---' to '', which you can do locally.
 > > 
 > > I'm not sure why we use '---', since it's not informative.  I will
 > > have to see if one of the other devs knows, or maybe there's something
 > > in the history.

 > In particular, we don't include the original From: @domain because
 > of goal #2 [in dmarc.py]. We use --- to imply that something is
 > missing. Perhaps ellipsis would have been a better choice, but we
 > didn't want anything that even hinted at a domain.

OK, that makes some sense.  Here's goal #2:

# 2) the original From: address should not be in a comment or display
#name in the new From: because it is claimed that multiple domains
#in any fields in From: are indicative of spamminess.  This means
#it should be in Reply-To: or Cc:.

I don't recall there being documentation of this claim.  I certainly
believe it happens at least occasionally (I know several mail admins
who will implement any filter that might eliminate 1 or more spams in
the next decade ;-).  But is it really a major problem?

I'm coming around to the idea of a general format language for
configuring various Mailman-generated texts with a bunch of standard
codes (like strftime).  There are too many "minority" problems like
the OP's (and I suspect "multiple addresses are spam" for that matter)
for me to be comfortable ignoring the set, but on the other hand the
number of options we'd have to provide to satisfy 1/10th of them would
be insane, and unpopular with 99% of the folks considering changing
the default.

I am considering adding this as a project to GSoC.  Comments welcome.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] --- in DMARC From: rewritings

[Stephen J. Turnbull]

Дилян Палаузов writes:

 > realname = re.sub(r'@([^ .]+\.)+[^ .]+$', '---', realname)

The obvious hack is to change '---' to '', which you can do locally.

I'm not sure why we use '---', since it's not informative.  I will
have to see if one of the other devs knows, or maybe there's something
in the history.

 > compare to the IETF mailing lists, which would send:
 > From:where a and b are from a@b, d is from c@d

That's elegant, but I suspect that it would be confusing to anyone who
isn't an email hacker.

 > In my real-world case the From: is a...@example.org, the ML name is
 > a...@l.example.org (localname is the same, the domains are
 > different).  The result is:
 > 
 > From: aaa--- via Aaa 
 > 
 > So there is twice trippe-A and @example.org disappeared.  Mentioning
 > twice AAA in the display name in this case really has to added value.

Of course there's added value.  The first "aaa" is a necessary hack to
identify the author, while the second is the name of the list.  That
they are the same is just an accident (unless the list owner is "aaa",
in which case they should just add their display name).  You could
argue that since the mailing list's display name is usually just a
capitalized version of the local part of the posting address, it's
unnecessary, but that turns out to be untrue since some common MUAs
suppress addresses in the display, leaving only the display name.

 > and in the future offer as options:
 >   From:  User Name 
 >   From:  “User Name”  (userem...@address.com) <
 > listn...@lists.illinois.edu >
 >   From:  “User Name”  (userem...@address.com via listName Mailing List)
 > 
 >   From:  “User Name via listName” 

None of these work in the case of interest since they assume the
display name is available, and I don't see enough value in additional
options to do this myself.  FWIW, I don't think any of the other core
developers would be interested in doing it, either.  You could submit
a patch, but I can't promise it would be accepted, unless one of the
core devs here expresses an interest in it.

I think it's definitely possible to make a case for changing the
default here, but if we're going to offer options, I think we would
probably allow the admin to specify a format string rather than
offering multiple fixed formats.  And if we're going to change at all,
be prepared for a long bikeshedding conversation because (1) the
current approach works for everybody, although it is occasionally
ugly, and (2) we need to be careful that the result is readable and
informative in almost all MUAs.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GNU Mailman in Google Summer of Code 2022!

[Stephen J. Turnbull]

Hi all,

We are participating in Google Summer of Code 2022 (GSoC) under the
Python Software Foundation (PSF).  We did apply as a separate
organization, but didn't get in.  Many thanks to the PSF's GSoC
organization for picking us up at the last minute!

You can find our ideas page at
https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202022, and
item on the PSF ideas page under https://python-gsoc.org/ideas.html
(which isn't so interesting about Mailman, but you can check out who
else is under the PSF banner there).

Discussion and development of project ideas will take place on
mailman-developers@python.org.

If you have a developed idea not already listed that you're willing to
discuss and advocate, and you're willing to grab the template (last
item on our ideas page on list.org) and fill it out, please post it to
mailman-developers@python.org.  (This a deliberate hoop-jumping
exercise to filter in "serious" ideas.)  You can put more weight on it
by expressing interest in mentoring the project, and even more by
actually volunteering! ;-)

If you have a half-baked idea, and/or you want to get input from other
Mailman users, of course you should feel free to discuss it on
mailman-us...@mailman3.org.  We *might* pick it up from there, even if
it's just a drive-by post with little support from the list members.
But no promises about that, most of our attention will go to
mailman-developers.  If you feel some idea should get more love,
polish it a little and post that to mailman-developers (don't
cross-post, please).

Again, we'd love to have more mentors, including backup mentors.  We
currently only have two mentors.  Although we're both core developers,
and I'm on an academic schedule and can devote plenty of time to the
student during the summer, not so for Abhilash.

Volunteering at this stage is not a promise to do more than look at
student proposals.  You would only be asked to look at the proposals,
and volunteer as mentor for that project if you have interest and some
level of knowledge to contribute to it.  Even if you're a site admin
with little Python programming experience, vetting interns'
brainstorms about list admin-visible features and holding their hands
until the "pros from Dover" arrive is quite helpful in making the
student feel at home in our community.  Many students are shy about
blogging and communicating about their projects to the community --
just talking to them about their project and suggesting that they blog
what they told you is helpful.  Coding skills and knowledge of some
part of the Mailman 3 code base are most welcome, of course, but you
can make your mentor role what you want it to be.

Regards,
Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Unclear usage of `member = ISubscriptionManager(mlist).register(…)`

[Stephen J. Turnbull]

I hope I'm not underestimating your understanding.  If so, no offense
intended.  I don't think Mark's answer entirely addressed your
question, so here goes.

Дилян Палаузов writes:

 > I am trying to understand the source code.  My reading is, that
 > interfaces/subscriptions.py:class ISubscriptionManager.register always
 > returns a 3-tuple and the last element of the tuple is a member object,
 > or None.   This coincides with the implementation in
 > app/subscriptions.py.SubscriptionManager.register .
 > cli_synmembers.py:add_members() contains:
 > 
 > member = [...]
[...]
 > and commands/cli_addmembers.py:add_members() contains
 > 
 > member = [...]

 > So my question is: when regirstar.register returns a tuple, the last
 > argument of which is a members,

It's not a members (plural).  The last element is a member
(singular), as the code accessing it indicates.

 > why do the above snippets store the
 > result in a variable called `member`?

If you look a little bit earlier in cli_addmembers.py:add_members
(this one is plural!), you see

for line in in_fp:
# line parsing stuff and then
try:
member = [...]# the line quoted from your message above

What is in_fp?  It's a file object returned from open().  The
structure of that file is a sequence of lines, where each line
contains either a display name and an address enclosed in <>, or an
address, to be tested for membership, and if not a member to be
registered (both of those are encapsulated in registrar.register, the
result of the test is communicated by raising AlreadySubscribedError
or not).

So cli_addmembers is able to handle more than one subscription at a
time, but it passes them to register one at a time.  That's why the
function's name is plural, but the local variable name is singular.

 > How are the set preferences set to member handled further, as my
 > reading is that the variable just disappears, immediately after
 > member.preferences is set?

The one-line answer is "it's SQLAlchemy 'magic'." :-)

Note the decorator "@transactional* at the top.  That says that this
function manipulates the external database in sqlite3, postgresql, or
mysql, and if it fails in any way, all of it should be rolled back and
not entered into the database.  How this connection to a RDBMS works
is too complicated to explain it detail here, but the basic idea is
that the member object returned from registrar.register is endowed
with properties that call SQLAlchemy to add or update the member
records in the external database.  This makes it possible for you, the
programmer, to manipulate objects like member as ordinary Python
objects, but changes you make to them are reflected persistently in a
database.

OK, a little more detail since you'd need to study SQLAlchemy:
basically what an object-relational manager like SQLAlchemy does is to
provide code that accesses and caches data from the RDMBS when an
object's attributes are accessed, and more code that issues database
updates when they are changed.  @transactional wraps the function it
decorates in code that first creates an RDMBS transaction, then calls
the function.  This means that the database code does not update the
external database immediately, but queues the updates in the
transaction.  Finally, if the function exits normally, @transactional
regains control and executes the transaction, making all updates to
the database.

All this means that classes defined in a model/*.py have instances
that can be treated as persisting even after the program exits.  This
is actually quite natural if you consider how Python variables are not
variables in the sense of a data container in memory, but just name-
object bindings.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] How to recreate Issue #973 in mailman-core

[Stephen J. Turnbull]

Suyash Singh writes:

Welcome, Suyash!

Sorry this mail has been sitting in my drafts for a day or so, but I
don't think anybody else has answered yet.  Let me just send it and
start the conversation.

 > I have started a development environment and am starting to read the
 > documentation. I picked up issue #973 because it seemed like a easy issue
 > to fix.

Not clear what you're reading.  The dev setup guide starts here, but
the site structure is somewhat complicated (disorganized):
https://docs.mailman3.org/en/latest/devsetup.html

 > Link to Issue: https://gitlab.com/mailman/mailman/-/issues/973

Note that at least one other developer (whose handle I don't recognize
so probably new) is working on that issue.

 > 1) I am unable to recreate the problem. As I can't find the API url and how
 > do I access them in my development environment. I tried the using the
 > command "mailman shell" which is referred to in the documentation for
 > debugging. and after holding a message and handling it. I get the following
 > error:
 > >>> handle_message(mlist, 1, Action.discard)
 > >>> handle_message(mlist, 1, Action.discard)
 > Traceback (most recent call last):
 >   File "", line 1, in 
 >   File
 > "/home/suyashsingh/Desktop/mailman/mailman/src/mailman/app/moderator.py",
 > line 113, in handle_message
 > key, msgdata = requestdb.get_request(id)
 > TypeError: cannot unpack non-iterable NoneType object

I think you probably don't have Mailman running, but I can't be sure
because of the limited information you've given about how you set up
your dev environment.

 > 2) How can I start http://localhost:9001/ on my local machine as currently
 > its not working. What is the port 9001 used for?

In a dev environment, port 9001 is used for the REST API served by
Mailman core.  You start Mailman by running "mailman start" in the
directory where you want Mailman's database stored (this assumes you
have configured your dev environment to use sqlite3 as the backend).

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Core new release?

[Stephen J. Turnbull]

Danil Smirnov writes:

 > Well, this was just a question and not a request or a claim.

For future reference, "any hope ... soon?" is typically a deliberately
offensive way to make a claim.  "Just ... question(s)" is also a good
way to get yourself blocked by native English speakers, because I
don't think I've ever seen it used in good faith by a native speaker.
(That's not a threat; if I meant it that way I'd just do it.)  Write
as you like, but be aware that many people will think poorly of you if
you use idioms like these, unless they assume you're not native in
English and make efforts to account for that.

 > I'm actually thinking about establishing my own Docker-based
 > releases from master branch, but in this case I would skip
 > participation in beta/pre-release testing due to having my own
 > schedule.

Up to you.  master does get quite thorough unit testing, usually fixes
when tests fail, and a certain amount of integration testing and
fixing leading up to a release -- that's why it takes a while to
release.  It might be a good basis for your releases.  But that
depends on your purposes and how much work you want to put in
yourself.

Steve



___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Core new release?

[Stephen J. Turnbull]

Danil Smirnov writes:

 > Any hope to release a new version soon folks?

Releases come when they come, you know that.  If you want to know what
Abhilash's schedule (if any) is, so that you can plan your own work,
say so.  It would help if you could avoid the snark.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: tox(ic).ini ;-)

[Stephen J. Turnbull]

Abhilash Raj writes:

 > True, we define the supported Django versions explicitly in tox.ini, one
 > per minor version. Sometimes, it gets out of date.

I don't have a problem with "sometimes".  But the current situation
looks unprofessional.

 > > Also, the envlist for Postorius is just unuseful.  Django-1.1.1?  LOL
 > 
 > 1.1.1 is probably a typo for 1.11 I think.

Sure, but still, Django 1?!

 > > What I think we should do is
 > > 
 > > 1.  Put a full list of Django versions that we have ever supported in
 > > the deps variable.
 > 
 > By ever supported, you mean even the ones we don't support anymore?

Yes.  We could put in a comment that they're not necessarily supported
by us, but there are always going to be people (RHEL...) who for
whatever reason want to be a couple versions behind what we support.
This would also go forward with versions that are still experimental.

The logic behind "ever supported" is that it's an easy rule to update
to (just take the union of the version specs in various files), and we
only do it once.  After that, we only add "djangoXX: Django >=x.y,
  > > 2.  Put the full list of Python-Django combinations that are supported
 > > (where Django is relevant).
 > 
 > Put that where? In envlist?

Yes.

 > > 3.  Standardize on lint for syntax checking and cov for coverage analysis.
 > 
 > Agreed!

Good!

 > > Then people with a single version of Python and a single version of
 > > Django (which I think is going to be typical of non-core-developers
 > > including a lot of people who send patches) can just run "tox".
 > 
 > So, you suggest we put a single environment in envlist, something like:
 >   
 >   envlist = "py39-django31"

No, that's not the way envlist works.  I'm saying we put

envlist = py{36,37,38,39}-Django{22,23,30,31}

(which expands to 16 combinations!) and somebody who only has Python
3.8 and Django 2.3 installed will only get that combination tested.
... Hm, on second thought I'm not sure if tox works that way for
anything but "pyXX" by default, but I'm pretty sure we can tell tox
only to test installed versions of Django, and not download them if
they're not already installed.

 > I have almost stopped running the full CI locally and just relying on CI
 > to run other version jobs if one passes locally. 

Well, you could always vendor your tox.ini. ;-)  I don't really object
to just putting one version into envlist, but if we get into a
situation where some people are working on compatibility the most
recent Python while others are using something a couple of releases
back, it could get annoying.

 > When you say "top page on Gitlab", do you mean something like:
 > 
 >https://gitlab.com/mailman 

This one.

 > The former doesn't have any customizations we can do, no "group"
 > level README available.

That kinda sucks.  It is possible to create subgroups (I guess
"archived" is a special-purpose subgroup available by default).  Not
sure it's worth the effort.  Too many things to keep synced!

 > But we can add this to Core's landing page within the README.rst.

That might be best.  Or maybe just rely on readthedocs.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: tox(ic).ini ;-)

[Stephen J. Turnbull]

Abhilash Raj writes:

 > So, envlist doesn't really represent the list of supported
 > versions.  [...]  envlist represent the exact list of env that will
 > be used if you run `tox`,

I understand how tox defines envlist, but I would expect that it would
basically tell you what you can expect to run tests with and pass.

 > but usually I just do `tox -e ` and this works even for
 > Python versions not in the envlist.

Me too; I've been short on space on my main workstation, so I'm
usually down to one full suite of Python and my packages at a time.

But that works for Python versions because Python is very special.  It
*doesn't* work for Django, in particular.

Also, the envlist for Postorius is just unuseful.  Django-1.1.1?  LOL

What I think we should do is

1.  Put a full list of Django versions that we have ever supported in
the deps variable.
2.  Put the full list of Python-Django combinations that are supported
(where Django is relevant).
3.  Standardize on lint for syntax checking and cov for coverage analysis.

Then people with a single version of Python and a single version of
Django (which I think is going to be typical of non-core-developers
including a lot of people who send patches) can just run "tox".

 > I have been considering putting just one latest Python version in
 > envlist so I can just do `tox` to run the test suite once and use
 > the explicit `-e` flag for CI, which we already do.

I think that a simple tox invocation should be reserved to the users.
I mostly run tox on the whole suite from a script, in the background
(ie, when I'm at lunch, a meeting, or a class).  If I'm doing unit
tests on an in-process branch, I generally run nose2 directly anyway.
(Obviously other people do things their way, and I'm completely open
to reason on the issue. ;-)

 > We should, yes. I am trying to to figure out if I can define the CI
 > definition in a single repo and have that used in all the Mailman
 > projects.  Haven't had time to figure that out yet :( Github
 > Actions provides a good way to share CI configs, maybe Gitlab has
 > some way too!

I'm way behind on that tech. :-(

 > This is a very fun one actually, I have been trying to think of how
 > to fix this. The version of Django-latest is > one supported (in
 > setup.py) by P.

Does Postorius still support Django 1.1.1?!

 > Yes, but there aren't any docs

Speaking of missing docs, the top page on GitLab is woefully lacking
in guidance on which of the subprojects you need for a functional
system, which are documentation, which are example configuration, and
which are experimental.

Regards,
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] tox(ic).ini ;-)

[Stephen J. Turnbull]

I'm fiddling with my Python installations and noticed this:

steve Mailman/pristine 1:19 % grep envlist */tox.ini
django-mailman3/tox.ini:envlist = py{37,38,39}-django{21,22,30,31},lint
hyperkitty/tox.ini:envlist = py{36,37,38,39}-django{22,30,31},docs,lint
mailman-hyperkitty/tox.ini:envlist = {py35,py36,py37}{,-coverage},lint
mailman-web/tox.ini:envlist = docs
mailman/tox.ini:envlist = {py35,py36,py37}-{nocov,cov,diffcov}{,-mysql,-pg},qa
mailmanclient/tox.ini:envlist = py{35,36,37,38},lint
postorius/tox.ini:envlist = py{35,36,37}-django{111,20,latest},pep8

We should do something about coordinating on supported versions, no?
Also, coordinating common factors (cov vs coverage, pep8 vs lint vs qa)?
Also, the djangolatest environment for postorius looks broken (but I
don't understand tox that well):

django-latest: https://github.com/django/django/archive/main.tar.gz

Also, shouldn't every subproject support docs?

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Core new release?

[Stephen J. Turnbull]

Danil Smirnov writes:

 > I know that it's just a month from the previous Mailman Core release, but
 > given the stats above should the next release be planned earlier
 > than usual?

I hope not, because I need Abhilash to pay attention to the Summer of
Code student selection for the next couple of days.  After that, he's
all yours. :-)  Until the coding starts in June, anyway!

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: [Mailman-cabal] Invitation to the CERT Vendor Meeting 2021 [General-7305] - mailman

[Stephen J. Turnbull]

Mark Sapiro writes:
 > On 4/21/21 11:25 AM, Stephen J. Turnbull wrote:

 > > Has anybody seen anything like this before?

 > My fault. There was an issue on mpo causing a shunted message
 > <https://gitlab.com/mailman/mailman/-/issues/881>, and I installed a
 > quick but incorrect patch

Sorry you have to do extra work, but no problem for me.  I almost
complained to CERT ;-) but then I actually thought about it :-P, so no
harm done (to me, anyway).

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] [Mailman-cabal] Invitation to the CERT Vendor Meeting 2021 [General-7305] - mailman

[Stephen J. Turnbull]

Hi,

There was recently a mail on the (closed) Mailman security list from
the CERT Coordination Center which had this strange CC field:

CC: '@mail.python.org, c...@mail.python.org, e...@mail.python.org,
r...@mail.python.org, t...@mail.python.org, o...@mail.python.org,
r...@mail.python.org, d...@mail.python.org, i...@mail.python.org,
n...@mail.python.org, a...@mail.python.org, t...@mail.python.org,
e...@mail.python.org, c...@mail.python.org, "", "."@mail.python.org,
g...@mail.python.org

I guess those are "conformant" email addresses, but they seem unlikely
to be mailboxes at python.org.  I'm not sure if these addresses were
added by the generating software at CERT, some MTA, or Mailman.

Has anybody seen anything like this before?

P.S. I hope I'm not spoiling this for any amateur sleuths who wanted
to figure it out for themselves, but yes, except for the apostrophe
and the empty address, those are the letters in "CERT Coordination
Center ", in order, with dupes eliminated.

P.P.S. It's about a conference, not a CVE.  Hakuna matata!

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: [MM3-users] Long lines in mail composed by HyperKitty

[Stephen J. Turnbull]

Abhilash Raj writes:

 > Is this something new that you’ve noticed?

I've been noticing long-lines in format=fixed (ie, the default
setting) for a while, but hadn't paid attention to the sending
clients.  For some reason I was rooting around in the headers of this
particular message, and saw HyperKitty and that kinda tickled my RFC
wonk reflex. :-)

 > I think previously, Hyperkitty used to wrap lines at 90 characters
 > when displaying them in UI

UI shouldn't have anything to do with the message as formatted for the
wire, though.  I'm not seeing this in the UI, I'm seeing a message
*sent from the HyperKitty UI*, viewed in my usual Emacs-based MUA.

 > and then I think the replies would’ve been wrapped at 90 + “ >”
 > (2 chars) per line maximum. I may need to confirm this because we
 > don’t do anything else when sending the message or any recent
 > changes to that part of the code.
 > 
 > But recently, when adding support for rich text rendering, I think
 > I removed that 90 char wrap in the UI,

This shouldn't be related to my issue.

 > > I think that HyperKitty should format mail per RFC 3676 format=flowed
 > > https://tools.ietf.org/html/rfc3676#section-4.  However, I don't use
 > > "modern" (aka crappy HTML-oriented) clients, so I don't know whether
 > > they handle format=flowed properly.
 > 
 > Support for format=flowed is good in the web client (Fastmail) and Mac
 > client that I’ve been using.

Good!

 > Just to confirm this, but the way to implement that would be to add
 > format=flowed to the generated email’s content-type header and then 
 > add a CRLF after LINE_LENGTH octects, right?

No, basically the idea is to take a message formatted for fixed-length
lines, and allow the receiving client to reflow paragraphs as it wants
to.

The method is to add "format=flowed" to the generated MIME body's
content-type header field, and then append an ASCII space to the end
of each line that is part of a flowable paragraph.  Each paragraph
ends with a fixed line (no trailing space).

Preformatted tables and ASCII art are supported by simply not adding
the space.  Obviously that is not an easy thing to do for a plain
text message body source, but should be easy for Markdown source.

The rules for handling quoted material are non-trivial and not very
smart.  Eg, the quoting style where the writer's initials appear
before the ">" as in "sjt>" is not supported, and will be very ugly if
flowed.  See the RFC for that.
https://tools.ietf.org/html/rfc3676#section-4

I would expect that email.message supports generating format=flowed
message bodies, at least the simple case of a series of flowed
paragraphs, but I haven't checked.

 > We use EmailMessage[1] from Django[2], which is itself a wrapper over Message
 > class form standard library. I don’t know if the BytesGenerator supports
 > some sort of policy when serializing the body, but if not, I guess we can
 > handle breaking lines with CRLF before we pass it to Django.

That's helpful, thanks!  Yes, I suspect that we should do our own
rendering.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Long lines in mail composed by HyperKitty

[Stephen J. Turnbull]

Hi,

[I prefer discussion be directed to mailman-developers, so Reply-To is
set.  But if you're not subscribed to -developers, following up here
is OK, and I'll eventually summarize to -developers.  Just don't
followup to both.]

I just noticed that among the agents that send non-conformant long
lines[1] is HyperKitty.  Nowadays violating the 80 octet
limit is common and MUAs mostly handle it, but the 1000 octet limit is
enforced by many MTAs[2]

I think that HyperKitty should format mail per RFC 3676 format=flowed
https://tools.ietf.org/html/rfc3676#section-4.  However, I don't use
"modern" (aka crappy HTML-oriented) clients, so I don't know whether
they handle format=flowed properly.

Discussion greatly desired.

Steve



Footnotes: 
[1]   RFC 5321: MTAs MAY impose a limit >= 1000 octets including
CRLF.  RFC 5322: MUST be <= 1000 octets including CRLF, SHOULD be
<= 80 octets including CRLF.

[2]  In practice, most (?) MTAs just break the line at 998 octets and
insert CRLF rather than reject the message.  I assume they would break
happily in the middle of a multi-byte character, i.e., any non-ASCII
in a UTF-8 text.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] X-Mailman-Original-DKIM-Signature

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > I discovered this just today, after a list I'm subscribed to enabled it.
 > 
 > I have a filter which tries to validate original signatures by
 > removing footers and subject tags.  Removing "X-Mailman-Original-"
 > is going to be the next addition.

You mean removing that substring from the field names?  Should allow
more successful validations, I guess.

 > I guess the purpose of this option is to spare a dkim=fail result
 > in the recipient's header.  However, for documentation purposes[*],
 > I'd be comforted by some document, discussion, or even a crispy
 > comment about the intended usefulness of masking existing
 > signatures.  Google found nothing.  Any pointer?

The unvalidated and unsigned X-Mailman-Original-OAR field seems like a
bad idea to me -- there's a reason why the ARC protocol is so finicky.
Anyway, it's superseded by ARC.  The others are harmless, I guess, and
useful to you.

The purpose of removing signatures that Mailman's decorations will
break has been discussed occasionally on these lists.  The problem
that it tries to solve is admins who think they're smarter than the
IETF and rate messages with failed signatures as *more* spammy than
messages with no signature, and end up rejecting list traffic for that
reason.  I guess Jim's patch to move them to X-Mailman-Original-*
fields is intended to preserve the signature for your use.

I thought only Guido van Rossum had a time machine!


Footnotes: 
[1]  I can think of a couple possibilities, but they've all been
superseded by IETF standardized fields.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] [GSoC] Some Questions in List Configuration Tool

[Stephen J. Turnbull]

Steven Chen writes:

 > I am currently working on the proposal of List Configuration Tool in
 > GSoC idea list.

Glad to hear it!

 > I wonder what kind of configurations should be exported, and what
 > should not.

 > Q1. We can get nearly all the configurations for a list from the
 > core's REST API
 > (https://mailman.readthedocs.io/en/release-3.1/src/mailman/rest/docs/listconf.html),
 > but some of the data should not be exported, such as `created_at`,
 > `last_post_at`, and etc. Hence, we may need a filter to extract out
 > the meaningful configurations to export out.

If you are migrating a list to a new host, you *do* want to export
those details.

At this level, I think it's probably more meaningful to separate the
attributes into "read/write" (RW) and "read-only" (RO).  The tool's UI
should allow you to see everything, export everything, and change the
RW attributes in place.  RO attributes would include things like
last_post_at, and maybe list_id (oops, that page doesn't mention the
RFC 2919 List-Id attribute ... that should be part of the config!)
Perhaps the list's addresses should not be RW by default but there
could be a "hot stove" mode where you can change them.

Also, there are partially identifying attributes like
'acceptable_aliases', which might be useful as initial values in some
cases (for example, I have a tree of umbrella lists for my advisees
depending on enrollment status and expected graduation year -- the
all_students top umbrella list should be an alias for all the others).

 > I wonder whether we have a more convenient way to get all
 > meaningful configurations,

No.  There is the "collection" endpoint ".../config", and the
individual resource endpoints like ".../config/display_name".  These
are written with the 'PUT' and 'PATCH' HTTP methods, respectively.
That's it.

 > and what kind of configuration fields are expected to be exported
 > from the user's perspective?

Look up "list styles".  This is a partial configuration that can be
applied to a new or existing list.  I think that there are "stylable"
options such as 'advertised', there are list-identifying attributes
like 'display_name', 

 > Q2. Besides all the configurations obtained from the listconf
 > mentioned in Q1, we have more configurations for a list, such as
 > member list. In the Idea Page
 > (https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202021), it
 > says that the membership roster should not be included by default.
 > Since we already have the `list_mass_subscribe` method for importing a
 > lot of subscriptions at once, so should the list configuration tool
 > include the functionality to handle membership automatically or just
 > does not include the memberships in the exported configuration?

I don't think this is necessary.  Postorius already has a
"good-enough" interface for this.

 > For Q1, I do have some thinking for some important configuration
 > fields that should not be exported:
 > owner_address: should be filtered out, since only the list owner can
 > import the configuration template,

But this attribute is independent of the admin who does the
importing.  Also, you need to serve the host migration use case.

 > the owner_address should be set when the owner apply the
 > configuration template to the list instead of import from the
 > previous configuration.  mail_host, fqdn_listname: should be
 > filtered out, since lists are created under the domains, domains
 > should be created before the lists are created. Hence, only the
 > name before the domain should be included in the exported
 > configuration, and the full fqdn_listname can be generated when
 > creating the list.

Consider the virtual host use case, such as readthedocs.  Suppose they
migrate doc-...@mailman.readthedocs.org to mailman.readthedocs.io.  Then
they need at least the subdomain "mailman."

The styling use case should be more flexible.  Perhaps a checklist of
all RW attributes.

 > I do need more suggestions on what to choose in the configuration that
 > can be exported. Any suggestion is welcome.

You might want to ask the users (specifically, mailman-us...@mailman3.org).

I would not worry too much about this, however.  At this stage, you
can have a tentative list of the kinds of attributes and which
attributes are which kind.  It's more important to think about use
cases you want to serve, at least migrations and list styles, and
maybe the tool can replace the current list configuration UI in
Postorius.  Also, is it just a backend for Postorius?  Can it be
written in Django?  Or should it be written in Python, so it can be
accessed from Postorius, the command line, and perhaps even EMWD's
alternative Affinity management interface (that might be difficult
since Affinity is written in PHP)?

I forgot to mention this previously, but you should make your account
on summerofcode.withgoogle.com *immediately*, and start your
proposal there.  Make sure all needed fields are filled out.  You can
edit up until the 

[Mailman-Developers] Interest to participate in GSoC under mailman

[Stephen J. Turnbull]

Shubhank Saxena writes:

 > My name is Shubhank(https://gitlab.com/shubhank-saxena).

Welcome, Shubhank!

 > I have now started to write a proposal in the same direction and I was
 > wondering how can I start the discussion regarding the Mailman Core
 > projects that are given on the webpage (
 > https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202021). I am
 > interested in the project "Support for REST Callbacks in Core".

That starts the discussion. :-)  Tell us about why you're interested
in that task, what you plan to do (to the extent you know -- some
people know a lot, some people know almost nothing, I can't say that
coming in with knowledge doesn't give the former a headstart, but for
those in the "know little" situation, plenty of students have caught
up in the past, you can do it too!)  If you have questions about
what's expected, what part of the docs and code to start reading, etc,
asking those questions is a good place to start.

 > Also, is the IRC channel active?

Well, it's open.  At present, the core developers/mentors are in UTC-7
and UTC+9, so getting together online at the same time is non-trivial.
It will be more active now that GCSoC applications are open (sorry, my
client seems to be unhappy after system upgrades, so I'm not online at
the moment).  You can find me as yaseppochi and Abhilash as maxking in
#mailman.

 > I did not see any conversations happening so I was wondering if the
 > mode of communication has shifted strictly to the mailing list?

It depends on what we're doing.  Up until now everybody was mostly
working solo, Abhilash on Mailman 3, Mark on Mailman 2 and user
support, me on user support and GSoC prep.  So "asynchronous"
communication by mailing list made sense.  It also provides a less
ephemeral record than IRC (yes logs, but AFAICS most people don't keep
logs all that much).

Regards,
Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Question about GSoC SPAM

[Stephen J. Turnbull]

For the sake of any other students with similar concerns, here are a
few clarifications.  I have not cleared this with Abhilash, but I've
been reasonably accurate at channeling leadership so I'll apply EAFP
here. ;-)

Abhilash Raj writes:
 > > On Mar 29, 2021, at 8:07 AM, Steven Chen  wrote:

 > > This is Steven1677(https://gitlab.com/Steven1677) on GitLab. I want to
 > > contribute to the mailman3 project as a GSoC student.
 > 
 > Welcome Steven!

+1!

 > You don’t have to share anything on the public list that you are not
 > comfortable sharing, including your personal information.

Here are the technical details.  Only the mentors and org admins for
the organization you are applying to (all of them, not limited to
those you are interacting with[1]) can see your application on
Google's site.  (Of course the GSoC administrators @Google can, too.)
This means that it is safe to publish the URL to your proposal on this
(or any public) list.  Only the applicant and those who are logged in
as members of the organization have access.

I prefer that you *not* do so in general, but instead copy those parts
of your proposal that you would like to discuss in the email itself.
It's not really polite to the rest of the list to have discussions of
texts they can't read.  It's reasonable to include the just the URL in
cases where you've made "uninteresting" changes like typo fixes,
scheduling details, and want checking of the @GSoC version from your
potential mentors for our convenience.  (Other mentors may have
different opinions, but in the end we all will respect the students'
preferences.)

Whether you post your full proposal or not is up to you.  Many
students do publish the whole technical plan (including schedule) at
some point in the process, others don't.  It's up to you but remember
there are a lot of smart people, as well as representatives of
constituencies the mentors have little experience with (for example,
several executives of commercial hosting services that support
Mailman) who hang out here.

In public posts, you may redact anything you consider personal,
including schedules as well as PII.

 > AFAIK, it is only Google that needs that information to verify if you
 > are a student and (maybe!) your mentors to have your cell phone
 > number just in case they need to get in touch with you after your
 > selection for some reason. I don’t think that second part is necessary
 > either, but your mentors will have access to the application with
 > your personal info.

I think it's a good idea to have alternative means of contact.  I
heard of one case in the past where a student took a scheduled break
of one week for a relative's wedding in a remote area without
Internet, got sick and spent another ten days in a hospital.  It
played havoc with the schedule but the mentors did manage to recover
the project, the student came back, completed, and got paid.  Without
telephone contact, they would have been considered AWOL by the mentors
and I doubt things would have worked out.

Presumably the "normal" alternative to email would be IRC or video
chats, of course, but in emergencies you might have phone service in
areas without real access to the Internet.

Steve


Footnotes: 
[1]  Mailman is a small organization.  Be aware that there are large
"umbrella" organizations like PSF and GNOME that support dozens of
suborgs.  I believe that all the mentors and suborg admins under PSF
can see all the proposals -- that could run to a couple hundred
people.  That wouldn't bother me, but others might feel differently.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Allow to post in developers list

[Stephen J. Turnbull]

Ankit Punia writes:

 > I want to contribute to mailman. So I request you to allow me to post in
 > developers list so I can communicate with other contributors and
 > understand to product better.

Welcome Ankit!

Just so you know, you're already welcome to post.  We do impose
moderation on first posts because it keeps most spam off the list, at
the cost of a few hours' inconvenience to first-time posters.

If you haven't already joined the list, you can do so here:

 > To subscribe send an email to mailman-developers-j...@python.org
 > or visit
 > https://mail.python.org/mailman3/lists/mailman-developers.python.org/

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GNU Mailman has been accepted in Summer of Code!

[Stephen J. Turnbull]

Just to let everyone know.

[Cc'd to Mailman 3 Users, but please direct followups to the Mailman
Developers list only!]

GSoC "who we are" page is here:
https://summerofcode.withgoogle.com/organizations/4903353420611584/

Ideas page is here:
https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202021

Student applications open on March 30, and continue until April 13
18:00 UTC.

Comments, ideas, etc welcome on Mailman Developers (preferred) or
directly to me or Abhilash.  Please do NOT edit the ideas page on the
wiki without discussing with me, Abhilash, or Mark, as Google has
strict rules about what ideas are relevant to their program.  If you
have been active in GSoC and do understand the rules, the discussion
will be "pro forma", but at least we do want to know what we might be
asked about as potential mentors.

We can add more mentors, definitely, you're welcome.  If you can
reliably spend about 5 hours per week on it for the summer, we'll sign
you up as a regular mentor.  If you'd like to interact regularly with
students, but can't promise 5 hours a week, we can negotiate a
semi-formal status (but you won't get the privileges of a full mentor,
like the T-shirt).  Otherwise, we'll try to keep discussions on the
list, everybody welcome.

Regards,
Steve


___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Unable to create a thread in local dev environment

[Stephen J. Turnbull]

Prashant Pandey writes:

 > When I am creating a thread using "Start a new thread in hyperkitty", after
 > typing the subject and message it says "The message has been sent
 > successfully."
 > but after this, it does not show in the list discussion.

In my experience, this just works, so there's not enough information
here to do more than guess at the problem.  If you get the "success"
message, then it was presumably sent to and received by Mailman core.

Is it in the moderation queue?  If you just set up the server, you may
be considered a "new user" and your message held.

Try refreshing your browser window.

Is there anything in the logs?

Is the archive qrunner running?

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] HyperKitty seems to produce non-conforming long lines

[Stephen J. Turnbull]

For example,
https://mail.python.org/archives/list/python-id...@python.org/message/NAH2RRPDQ3F3AO3UBJEL3BTSDAQ3CQXO/

As I received it, some lines actually exceeded the SMTP hard limit of
998 characters (excluding the separating CRLF).

There are several alternative approaches; I think the best is to leave
the logical lines alone, and break the physical lines with format=flowed.

Issue #341 filed.
https://gitlab.com/mailman/hyperkitty/-/issues/341

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Hyperkitty's ability to build a thread

[Stephen J. Turnbull]

Mark Sapiro writes:

 > There is an article on threading at
 >  and an RFC
 > . These describe algorithms
 > which are fairly complex, but if someone wanted to try to implement them
 > in HyperKitty, we would certainly consider the implementation.

Ouch.  I didn't realize we didn't use Jamie's algorithm.  It's not
that hard to implement[1], and it's robust and extremely efficient[2],
modulo the cost of accessing message-id, in-reply-to, and references.

A robust, tested, and documented implementation sounds like a GSoC
project to me.  And a PyPI package, though that would be somewhat
harder.

Footnotes: 
[1]  It took me about a day to get it mostly working in Elisp, and
most of the difficulty and the remaining issues were due to working
around bugs in the MUA that caused uncaught exceptions in the MUA.

[2]  It's multipass, but it's worst-case and average-case linear.
Worst-case is linear because the line-length restriction keeps the
length of references down to about 15 at most.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Hyperkitty's ability to build a thread

[Stephen J. Turnbull]

Danil Smirnov writes:

 > As per my small investigation, a subscriber Robert N. Evans seems to have
 > "In-Reply-To" headers stripped from the messages that probably causes the
 > thread to break.
 > 
 > I wonder if Hyperkitty is able to leverage some other method to combine the
 > thread correctly in this case?

It's simply not possible to guarantee correct threading if neither
References nor In-Reply-To are present.

It is possible to place a message in an approximately appropriate
place by threading the threadable messages, grouping that message with
threads with "the same" subject, and inserting it (and any
descendants) after some message with an earlier date, but this is
inherently ambiguous as that message could be a reply to *any* such
message with an earlier date.

This would work well if there is a single linear thread.  But it is
unlikely to work at all well if several posters replied to a single
message in the recent past so that there are multiple subthreads
active at a given time.

Gmail has a big advantage, since they're reading your mail, indexing
it, and creating a fine-grained statistical profile.  That database
can probably be leveraged for better threading.  Or if your posters
consistently top-post, it's probably not too hard to match quoted
content against the top-level content of an earlier post -- if you
have both the development and the computational resources of Google.
(Come to think of it, for Gmail this would probably allow them to
compress their storage by 50%.)  Or maybe they just got lucky.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GSoC application

[Stephen J. Turnbull]

Hi, everybody

I would like to have Mailman participate in GSoC as a regular
organization this year (as I mentioned earlier I'd like to go to the
GSoC Mentor Summit if they have one -- I'm not *presuming* I'd be one
of the ones to go, but 2 out of 3 or 4 seems like a better bet than
one of the org admins at PSF doesn't go and then 1 out of 100+ PSF
mentors ;-).

The deadline for organization applications is Feb 19, and I'll
probably get started on it tomorrow or Sunday.

I think Abhilash already said he is in, so that's enough for me to get
started on the application.  Any objections from Mailman core?
Anybody in or out of core interested in mentoring?  If you're
interested but not sure what it involves, feel free to ask me.

Obviously some Python programming ability and familiarity with the
Mailman 3 codebase are important, but you don't have to be a Python
or Mailman core hacker to have a role, eg as a backup mentor, or
helping set up a development environment.

Steve

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Footer settings question: how common is dash-dash-space?

[Stephen J. Turnbull]

Richard Damon writes:

 > It is one of the 'Markdown' codes for a horizontal rule

But 72 of them?  Don't most people use about 5?

 > will occasionally end up with a line like that when documenting an
 > electrical signal that is always low (at least for a given
 > example).

OK, that's very plausible, and an excellent example.  It will almost
never happen to me (and that instance was in 1997 or so ;-), but for
you it could be an ongoing, if occasional, annoyance.  Not cool.

 > that would make the definition not match the existing practice, so
 > list settings would need to be edited.

True, for those folks who have edited their footers.  For the rest,
they get it with the upgrade, as the IETF list did.  We can mitigate
that, with some risk of guessing wrong.  But it's not a huge
consequence if we do guess wrong, I think.

 > Also, the question comes does the separator require and exact
 > number of underscores

Yes.  I think 72 is the sweet spot, or maybe 70 with a trailing space.

 > (and the frustrations of having the wrong number)

I doubt anybody would notice. :-)  Again, we can mitigate.  We separate
the footer from the separator.  I think we already do that in Mailman
3.

If you have time, please keep trying to poke holes in my ideas.
Whether to revert in Mailman 2 is entirely up to Mark, but Mailman 3
is still up for discussion.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Footer settings question: how common is dash-dash-space?

[Stephen J. Turnbull]

Thomas Hochstein writes:
 > "Stephen J. Turnbull" wrote:

 > Because [reusing the .sig separator for the footer] makes sense.

I understand that they are syntactically identical.

 > The message footer is identical to a signature in every respect:

Not true.  The mail standards are very careful to distinguish between
*originator* elements, such as Reply-To, and those that may be used by
third parties, such as Received and List-Id.  This could go either way.

 > > It's not obvious to me that that was a good idea.  Maybe it's better
 > > to distinguish the list's "signature" from a poster's signature by
 > > using a different separator.
 > 
 > Why?

Because the semantics are different.  One is for use by the author,
the other for use by the list.  I would not want .signatures stripped
or suppressed because they are often interesting or amusing, but do
want most footers suppressed (and would rarely care if they were
stripped) because my MUA knows how to get list information (such as
archives) out of the message header.

I'm not suggesting that my peculiarities are a good reason to override
the sense of most users.  It's an example of how the semantics are
different, and might usefully be given different syntax.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Footer settings question: how common is dash-dash-space?

[Stephen J. Turnbull]

Richard Damon writes:
 > On 1/30/21 9:40 PM, Sam Kuper wrote:

 > > All I am advocating is that:

 > > 2.  Footer separators should be distinguishable from likely body
 > > text, and should not be excessively long.

 > The problem is that the current Footer separator fails point 2 if
 > 'likely' is interpreted as to by likely enough for a program to
 > algorithmically trim a post based on it. In particular it looks like
 > markdown code.

Really?  I can't recall ever seeing a long line of underscores in the
wild -- except in my (accidentally) rms-baiting .sig (see below).

I don't have a strong opinion on this, myself (that's why I
asked).  But it's quite clear from RFC 3676 that there are tools that
handle .signatures specially, and I'm not sure that they're all
strippers.  For example, they could be tools to extract contact
information, in which case the naive implementation (go to EOF, back
up to .sig separator, parse) would fail or even corrupt data if list
footers use the "-- " convention.  The fact that the RFC goes to the
trouble of special-casing the .sig separator (it's specifically NOT
format=flowed even if that is specified, and not subject to stuffing,
DelSp, etc.) strongly suggests that there are a lot of naive tools out
there.

 > The key aspect is that before MUA developers can really try to implement
 > it as built in, the code needs to be reserved well enough that you won't
 > find it occuring in the wild.

A slightly more precise spec than Sam's such as exactly 72 underscores
followed by exactly one SPC (or even HT) should be unlkely to occur
even in markdown.

 > I would say I am not being defeatist, but practical. The key factor
 > here is that to implement your goal, you will need to get an RFC
 > established

Not the way IETF works.  Almost certainly[1] this convention would be
considered "not wire protocol" and therefore not in scope for IETF,
except perhaps for an update to 3676 giving the footer separator the
same treatment as the .sig separator (and that assumes my suggestion
were adopted).  If the distinction is desired by users, having Mailman
document it, and IWBNI other MLMs adopted it, that would be enough.
No RFC needed, really.

 > And, it that use case important enough that it not only 'pays' for the
 > work in creating a new RFC and getting it implemented, but also for the
 > loss of footer trimming that happens because some people are still using
 > MUAs that do the signature removal but haven't added the footer
 > removal.

I don't consider that a big loss if we switch back.  Given the
prevalence of top-posting nowadays, I doubt anybody cares about .sig
stripping except us Boomers (and Mark, who is honorary Greatest
Generation no matter his age :-).  Especially since "-- " has only
been the footer separator since June 2017.  I doubt many users will
have noticed (either way).

Steve


Footnotes: 
[1]  In IETF mail standardization, all we can be certain of is
uncertainty itself.

-- 



What are those straight lines?  "XEmacs rules."
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Footer settings question: how common is dash-dash-space?

[Stephen J. Turnbull]

Mark Sapiro writes:
 > On 1/29/21 11:48 PM, Stephen J. Turnbull wrote:
 > > Mark Sapiro writes:

 > >  > According to <https://www.ietf.org/mailman/listinfo/last-call> that list
 > >  > is on Mailman 2.1.29. The default message footer separator was changed
 > >  > from "___" to "-- " as of
 > >  > Mailman 2.1.24

 > > Why did we do that? 

 > Because of <https://bugs.launchpad.net/mailman/+bug/266269>. You can see
 > from that thread that I didn't want to do it, but I was apparently
 > convinced :(

Thought it might be something like that.  Feel free to ping me at
@netiquette-busters if you want any help standards-nerding bad ideas
into oblivion. :-)

I still don't know if it's a good idea or a bad idea, BTW. ;-)

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Footer settings question: how common is dash-dash-space?

[Stephen J. Turnbull]

Mark Sapiro writes:

 > According to  that list
 > is on Mailman 2.1.29. The default message footer separator was changed
 > from "___" to "-- " as of
 > Mailman 2.1.24

Why did we do that?  RFC 3676 doesn't standardize the practice, as far
as I can tell.  It simply mentions the convention and advises how to
detect it properly (no requirements language for generators except
that a line containing only "-- " must not end a paragraph when
format=flowed is active).

It's not obvious to me that that was a good idea.  Maybe it's better
to distinguish the list's "signature" from a poster's signature by
using a different separator.  In particular, if Ale wants to keep
poster signatures, he has to keep track of which lists don't use
footers.  Maybe he doesn't, and the question is moot for this thread.
But there is a potential for annoyance there.

If it makes sense, shouldn't we do it for Mailman 3 too?

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Issues gathering list members

[Stephen J. Turnbull]

Emanuel,

Mailman-Developers is not the best place to send this.  Such problems
are rarely bugs, and the active developers all read Mailman-Users
frequently.  You should send inquiries about installation and
configuration problems to mailman-us...@mailman3.org, because it's
quite possible that other users (== site and list admins, not
subscribers!) have encountered your problem.  You're much more likely
to get a *timely* and useful answer there, simply because there are
more knowledgeable folks there -- including us.

costavitor...@gmail.com writes:

 > I'm testing sending emails by telnet to a mailing list.

Test-mail-by-telnet is useful to check if you can connect to Mailman
on a port.  Otherwise it is a risky idea unless you are a serious
standards geek and a perfectionist, and even then it's a PITA.
Mailman does quite a few checks on mail for distribution, and may
*shunt* mail that does not conform to RFCs 821 and 822, may *discard*
mail that has no body after MIME processing, and will *hold, discard,
or reject* mail that matches various other rules.

I strongly recommend that you use a regular mail client, or a
preformatted test mail and inject it with an MTA.  Even a very simple
one (Berkeley mail or mailx are quick to start) will do (as long as
you stick to pure ASCII mail).  That removes one variable from the
diagnosis process.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: GSoC 2021 ideas

[Stephen J. Turnbull]

Terri Oda writes:

 > I think it will be more than CSS, because we moved a number of
 > options around in an attempt to improve the grouping, so you might
 > need to change the django page templates

Are CSS and templates considered "code"?  ISTR this being quite
controversial in the past, since they're declarative rather than
algorithmic.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] GSoC 2021 org applications

[Stephen J. Turnbull]

Abhilash Raj writes:
 > On Mon, Jan 4, 2021, at 12:49 PM, Terri Oda wrote:
 > > Happy new year everyone!
 > 
 > Happy new year to you too Terri!

And Happy New Year to everyone, as best we can.

 > > I'm starting to prep Python for GSoC 2021, and I wanted to issue
 > > the usual invitation that we'd be happy to have Mailman as a
 > > sub-org if anyone wants to do GSoC this year.  (Mailman's been
 > > fine as a separate org, but I have to do the paperwork for Python
 > > anyhow so I figured I'd offer if there's interest but no one
 > > wants to cover the admin side of things.)

Thank you very much!  Depending on how many students we get, I'm
probably available as an emergency/backup mentor for Python.

 > Thanks for the head start, I do want us to participate this year
 > and am planning to mentor at least one student.

I am also planning to participate.  I wonder if there will be an
in-person Mentor Summit; if so, I'd like to attend and do some
networking.  I'm only a couple of years from retirement from the
university and likely conversion to full-time software developer
(including Mailman, of course, but I have a bunch of ideas and
postponed projects).  If we go the separate application route, I'm
willing to do the org admin part.

I have to get grades in by Jan 18th, and after that I'll be in good
shape to work several hours/week on org and/or idea proposals.

Terri, in the past you've allowed rejected orgs to apply as Python
suborgs.  Did that work out ok?  Do you expect to have that policy
again this year?

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Stephen J. Turnbull]

Danil Smirnov writes:

 > Sorry for the confusion - I should write "would allow" instead of "allows"
 > as using socketmap_table would be a great feature but it obviously requires
 > a new endpoint in Mailman core which does not exist yet.

Ah, OK, so what exists now is the Postfix docker (which you provided a
reference).

 > Otherwise everything looks fine and with those two issues
 > resolved/mitigated there is no problem to deploy Mailman 3 in Kubernetes, I
 > can prepare an example Helm chart for it.

That would be great!

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman, postfix and docker

[Stephen J. Turnbull]

Danil Smirnov writes:

 > But how about that idea to offload Postfix-Mailman communication from a
 > filesystem to a network using Postfix socketmap_tables feature which I like
 > a lot as it allows me to get rid of using any shared volumes in the stack
 > completely - it's quite convenient in multi-host setups like Docker Swarm
 > or Kubernetes...

Would you be willing to write up a description of your configuration
that we can add to the wiki and/or documentation?  Communication in
multihost setups, especially those intended to isolate hosts exposed
to the internet from the intranet, is a sore point for us since
Mailman 3 is really designed as a single-host, three application
system.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Mass subscriptions as a DoS attack

[Stephen J. Turnbull]

Alessandro Vesely writes:

 > Trolls can wreak havoc by subscribing to one or more high volume
 > mailing lists on behalf of a target one.
 > 
 > Are there mechanisms to prevent that?

Don't antagonize trolls. :-/

The subscription to the high-volume list itself can't be prevented by
the victim list, because it's the high-volume list that accepts the
subscription, then sends to the victim list.  However, most lists
require confirmation, which is send by email to the list, looks like
administrivia and will most likely get filtered by the victim list's
Mailman.  I guess we could set the bar higher by requiring the
confirmation token be returned by email in a DMARC-From-aligned
message (and if From alignment fails, require moderator approval for
the subscription).  This is still a setting on the high-volume list
side, though.

On the victim side, I'm not 100% sure that the confirmation message
would be filtered as administrivia.  If necessary, we could beef up
that filter, and add RFC 2369 and 2919 header fields to the filter to
catch the actual "DoS" posts.  The RFC field filter wouldn't do
anything if the high-volume list doesn't use those fields, of course.

But even if the bad actor manages to get the victim subscribed to the
high-volume list, most lists nowadays require list membership to post.
The bad actor could try to reverse-subscribe the high-volume list to
the victim list (but now the bad actor runs into the barriers
described above again), so if you're worried about this a lot, you can
set the subscription policy at the victim list to confirm and approve.
If you're worried about this a little, you can set the victim list's
new member policy to 'moderate', which catches the garden variety
butthead as well as the sophisticated troll.

I haven't heard of an individual being attacked in this way for many
years, and I've never heard of a list attacked this way.  The
occasional attack now uses the subscription process itself, by getting
thousands of lists to send confirmation tokens to the victim.  I
suspect actually getting the victim subscribed is just not worth the
bad actor's effort to hack or social engineer the confirmation
process.  I figure XKCD 538 applies -- instead trying to hack multiple
lists, they'll just use a big pipe-wrench (eg, hire a botnet).

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Verifying broken DKIM signatures

[Stephen J. Turnbull]

Alessandro Vesely writes:

First, what Mailman are you talking about?  Only Mailman 3 is likely
to get these improvements, as Mailman 2 is end-of-life.  However,
Mailman 2 installations are likely to be around in large numbers for
several years, and if Mailman 2 is any evidence, likely few Mailman 3
installations would use these features unless forced to by a disaster
like the Yahoo/AOL sudden switch to DMARC p=reject.

 > Yet, it is possible to undo the transformation that Mailman put in
 > place, thereby validating the original DKIM signature.

It would always be possible to undo all transformations by supplying
the original email as part of a multipart/alternative, or perhaps a
new multipart subtype, maybe with some kind of device to make reading
the message/rfc822 original difficult in standard MUAs.  Then the
problem is much simpler: the validator need only validate the whole
and the original, and the receiver would have all information
necessary to decide whether the alleged Mailman version is based on
normal processing or is a malicious fake.  This would require some
changes to Mailman to implement, and to Postorius for a configuration
UI.  (In the case of Microsoft MUAs, if Mailman is configured to strip
HTML, the result might be less than 10% bigger than the original! ;-)

It is sometimes possible to reverse transformations with only the
information in the post after Mailman processing.  However, some very
desirable changes are destructive (eg, anonymized lists, conversion of
HTML to plain text, removal of prohibitive attachments).  Some
non-destructive changes (headers and footers) are highly customizable.
So the question is what are the transformations that users want to
reverse, and whether that's really possible.

This kind of transformation reversal probably requires no changes to
Mailman, just an addition of a Handler which could be written
independently and "dropped in" (with a configuration change to the
default pipeline).  The necessary information about transformations
that are configured would be available from Mailman in the usual way
(existing Handlers need that information).

 > Mailman carries out some irreversible changes, such as rewriting
 > To: or Cc: changing the order of the mailboxes,

Does this happen outside of DMARC mitigation?  Can you show examples?

 > or rewriting Content-Transfer-Encoding: irrespective of quotation
s > marks and case (for example "7bit" even if the original, signed
 > field was spelled as "7Bit").

I'm not aware of such behavior *unless other modifications were done*.
In that case, Mailman is specifying the C-T-E it uses, it is not
rewriting the original C-T-E.

 > I guess this behavior is coded deeply in Python libraries,

I don't think so.  As far as I know, the email module in Python 3
provides some support for parsing header fields but I don't know why
this would change order or spelling of field contents.  I would guess
that to the extent that it happens it has to do with Mailman-level
processing (for example, collecting addresses from the same domain so
they can be presented as multiple RCPT TO with a single DATA).  I can
say for sure that some care was taken to ensure that the order of
header fields, including multiple instances of the same field is
carefully preserved.

 > but would like to know developers' opinions.  Is that something
 > that could be fixed?

First, the issues with headers could be improved, though not entirely
fixed, in DKIM itself by further canonicalizing structured headers
before signing or verifying.

I'm not saying that this is the right way forward, but it should be
considered.

 > The second question is about producing a hint to the verifier telling which 
 > transformation(s) have been applied to the message.  That would come as an 
 > additional header field, for example:
 > 
 >  DKIM-Transform: footer

This could be done easily, but it would be at best a hint.  Among
other things, it might be desirable to identify the agent that
performs the transformation, as well as the algorithm and perhaps the
host and/or the list.  Mailman adds footers in different ways,
specifically appending text and adding a MIME part.  Third party
patches are available that dig into HTML structure (at least for
Mailman 2).  There are lists that feed into lists, and apply their own
transformations.

 > or as an extra tag in a DKIM signature, for example:
 > 
 >  DKIM-Signature: v=1; (...) tf=footer; (...)

Not possible without a lot of effort and specific cooperation from
MTAs.  Mailman doesn't DKIM sign messages, really doesn't want to
(there are Python modules for this, but use and configuration would be
our responsibility so we'd like to have specialists do it), and
probably shouldn't (we're not specialists) -- that should be left to
the border MTA of the administrative domain.

 > That hint could spare the verifier one pass over the message.  Is
 > it something  that could be implemented?  If not, I'd try guessing,
 > according 

[Mailman-Developers] Re: Anonymous mailing lists

[Stephen J. Turnbull]

Mark Sapiro writes:

 > Steve is not saying the option is not provided. It is provided in both
 > Mailman 2.1 and Mailman 3. He is giving reasons why you might not want
 > to use it.

Also, I think Mailman 3 provides Pretty Good Anonymity[tm], but we
welcome further suggestions.  No promises about inclusion or timing if
accepted, of course, but suggestions are welcome and will be taken
seriously vs. resources available.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Anonymous mailing lists

[Stephen J. Turnbull]

ves...@tana.it writes:

 > Someone started talking about the risk of having their names and
 > email addresses archived in a publicly accessible mailing list.  So
 > I thought I'd ask.  In short, the proposal provided for completely
 > removing such data, to protect privacy.

Mark provides the practical stuff.  Here's some theory.

"Completely" is a really hard problem given that some people obfuscate
their addresses in .signatures and in text, and the difficulty of
recognizing personal names.

There's also the fact that an anonymous list is a flag that you've got
people who want to be anonymous.  A big bullseye for troublemakers
(and the FBI, the MSS, and the GRU).

 > As an additional bonus, that would also "solve" any DMARC problem.

Sure, but the cost is extremely high.  I like to know who's posting.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] RFC: Config complexity [was: I have issue where it does not send ...]

[Stephen J. Turnbull]

Moved from mailman-us...@mailman3.org.[1]

Mark Sapiro writes:

 > For what it's worth, The actual mail list management, i.e.
 > receiving and delivering posts, only involves Mailman core and your
 > mailman suite config file is only for Postorius and HyperKitty and
 > doesn't affect Mailman core.

Aaargghh!  We should not be doing this to users.  If we're going to
have a "mailman suite" project, there should be a single file for
configuration of *all* installed components, at least for "vanilla"
installations.  (It's bad enough that in many installations one has to
configure the webserver three times, once in a front-end like Apache
or nginx, once in a wsgi dispatcher like uWSGI, and once in Django.)

I understand that it may take a ton of work and backward
incompatibility to implement this, or even make substantial progress,
so (until I have time to do it myself) I'm not seriously suggesting
it at this time.

It occurred to me that we could have a short comment at the top of
every config file that lists all the config files and what they
control, with one marked "(this file)".  Or something.  WDOT?

Steve


Footnotes: 
[1]  
https://lists.mailman3.org/archives/list/mailman-us...@mailman3.org/message/YMSZRS6OSJSS35EV2WLABHOQBD3BSVHM/
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Mailman 2.1.30 archives challenges

[Stephen J. Turnbull]

Mark Sapiro writes:

 > Because its a workaround for a Debian (or some distro) Python packaging
 > issue which should be fixed by the packager, and because I only thought
 > of the above fix yesterday when generating the above reply.
 > 
 > Also, it seems quite rare.

I wonder if this is something that is related to Unicode and locales;
only happens if your server has certain locales (LC_CTYPE).
It does seem like a distro bug to me, too, though.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: How to know the URLs for CURL while getting or modifying the data of mailman 3

[Stephen J. Turnbull]

Shashi,

This whole thread is off-topic for the developers list.  This list is
for development of Mailman, not for teaching people to use it.  Read
the whole recommended documentation before you ask busy developers to
tutor you in very basic things.  There's a mailman-us...@mailman3.org
list for users to help each other.  All of the Mailman developers who
might be willing to answer your questions are there as well, but there
are also many experienced users, some of whom have experimented with
the REST API who will also help.  Please reply there for further
help.  (Reply-To is set.)

That said, there are plenty of URLs in the API documentation.  Here's
a hint: search for "dump_json".  That function takes an URL.  You may
want https: instead of http: in your URLs, and you may need to change
the port from 9001, depending on the configuration of your REST
server.  Try it with simple GET requests for information first, such
as for domains or lists.  You can also do these requests with an
ordinary web browser, though I don't know what it will do with the
JSON-formatted reply.

If you don't have domains and lists configured, I suppose the
informational requests will come back as empty JSON objects.  It
should be safe to make them.  You can either use Postorius to
bootstrap a few example domains, lists, and users, or (the hard way)
figure out how to POST appropriate requests.  I don't recommend trying
to update the backend DBMS by hand; the schema is quite complicated.
I don't know how to use curl to update Mailman's databases.  That
requires a POST request which I have never done with curl.  The POST
request needs to be valid JSON.

The more I think about this, the less I think your idea of using curl
is worthwhile, because formatting JSON requests by hand is going to be
a real annoyance and a great way to mess up your database with
spurious information.  You're really going to want programmatic
support for this.  But that's already all available in Python in the
Mailman system.

It's up to you, but I feel a responsibility to warn you that using
curl is not going to be simpler than learning enough Python to do
these things.
 
Regards,
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] How to know the URLs for CURL while getting or modifying the data of mailman 3

[Stephen J. Turnbull]

Shashikanth Komandoor writes:

 >  But I don't know the URLs to be used for the purpose I want to
 > do some thing like I want to add user to list or remove a user from the
 > list.

I gave you the URL to the relevant docs.  RT Free M.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Question about REST API for Mailman 3

[Stephen J. Turnbull]

Shashikanth Komandoor writes:

 >   I am developing an application on which just clicking on submit
 > with some input values loaded should invoke the REST API of Mailman 3
 > running on another server and do some activities like adding of user(s) to
 > a list or removal of user(s) from a list or so.
 > 
 >   Is it possible to do so ? If so, how ?

It's possible.  The REST API itself is documented here:

https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/rest.html

I haven't tried the examples with curl, but they probably should
work; I don't think you need any special HTTP headers or anything like
that.

The "another server" part is possible, but a bad idea in production
because the REST API is 100% not authenticated.  Security is
maintained by running any applications that access the REST API on the
same host.  If you really want to do this, it's explained in this
thread:

https://lists.mailman3.org/archives/list/mailman-us...@mailman3.org/thread/M24EAWD7VK6PHDJVWI76XIXBI225FRF6/

Note that the context of this thread is a development environment and
the idea is to restrict access to a small LAN.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: italian translations

[Stephen J. Turnbull]

Daniele Pizzolli writes:

 > That's a waste of effort.  Please add/merge it soon.  Also any feedback
 > will be appreciated.

Sorry about this.  I think Abhilash (Cc'd) is the only person who
knows about the process for adding translations, and is also doing the
release engineering.  He lives and works in the San Francisco Bay
Area, which as you probably know has been a hot spot for COVID-19
infections.  I doubt he has much spare time for Mailman at the
moment.  I know I don't.

He has been more active on the mailing lists lately, so I hope there
will be more release news soon.  Please be patient.

Can you help me understand about why there is confusion about what's
been done by whom?  Are there multiple independent systems for doing
the translations?  Do you just do it differently because that's the
way you work?  I would think that we should have a system so that
translators can log on and see if anything needs to be done for their
language or if anybody has worked on it lately.  Does the system lack
that kind of work logging?

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Question on Subscription Moderation

[Stephen J. Turnbull]

Brian Carpenter writes:
 > On 4/7/20 10:21 AM, Stephen J. Turnbull wrote:
 > > Hi all,
 > >
 > > I wonder if mass subscribe should be exempted from the ban list check,
 > > what do you all think?  It would be convenient for me, as students
 > > often change or add personal addresses, and I have to do an unban,
 > > subscribe, reban dance.  (See my use case below.)
 > 
 > Please don't do that. If an ISP is on a Feedback Loop and receives spam 
 > reports from list members, then that ISP has to remove those list 
 > members if the list owner won't. Adding those spam reporters to the ban 
 > list can prevent the list owner from resubscribing those troublesome 
 > members who will just be reporting their list posts as spam all over again.

OK, it needs to be more nuanced than that.  Perhaps it's worth
distinguishing between addresses that have been specifically banned
and those caught by regexps, or perhaps I missed an option that allows
shutting down self-subscriptions completely rather than via the ban
list.

I think Mark responded to all your other comments, and I hope the
patch in 3.3.1 works as expected.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Question on Subscription Moderation

[Stephen J. Turnbull]

Hi all,

I wonder if mass subscribe should be exempted from the ban list check,
what do you all think?  It would be convenient for me, as students
often change or add personal addresses, and I have to do an unban,
subscribe, reban dance.  (See my use case below.)

Richard Damon writes:

 > I presume this is working like MM2, where the ban list is checked
 > at the subscription request point,

That's correct.  I'm a little surprised that this works, actually,
since the mass subscribe function is *also* prevented from subscribing
banned addresses.  Apparently the ban check is just a little bit after
where mass subscribe comes in, but earlier in the chain than the
moderation hold and dance.  Which makes sense, since to point of a ban
is to save the moderator work!

 > yes, you can have subscription requests and subscriptions from a
 > banned email address, those addresses just can't start the process.

It has to work this way for the subscriptions.  For example, my
advisee lists got hit by subscription spam a while back.  Since I know
who the students are and mass subscribe them every year in April
(beginning of our academic year), I just hit the spammers with
DISCARD ^. ;-)  (The smiley is not part of the regexp. :-)  If that
caused existing subscriptions to be invalidated, it would be a big
problem.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Rest API not returning members in alphabetical order

[Stephen J. Turnbull]

Mark Sapiro writes:
 > On 4/2/20 1:05 PM, brian_carpen...@emwd.com wrote:

 > > https://mariadb.com/kb/en/why-is-order-by-in-a-from-subquery-ignored/
 > > is saying it is not a bug so I doubt SQLAlchemy will think it is.
 > 
 > My take is a bit different. MariaDB is saying this is not a MariaDB bug.
 > They would say that it is a SQLAlchemy bug in that in this case, the
 > SQLAlchemy generated query should put the ORDER BY clause on the outer
 > SELECT and not on the sub-query.

Yup, my take is the same.

@mailman-developers I will do some research to find out if SQLAlchemy
is aware of this already, or maybe has documented it as a limitation
with appropriate ways to deal with it in user code.  But it's
definitely reportable against SQLAlchemy if MariaDB is right about the
SQL standard (trust but verify on that one too :-).

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Rest API not returning members in alphabetical order

[Stephen J. Turnbull]

Brian Carpenter writes:

 > However is there anything you guys can do to mitigate that issue
 > with Mariadb?

Bottom line up front: In the meantime I'd say if you want to use
MariaDB you're going to have to sort the results yourself on the
client side.  Sorry about that.

Goran's analysis is correct AFAICS, and implies that SQLAlchemy is not
working correctly with Mariadb.  It might be possible to change our
code to avoid the problem, but really it's SQLAlchemy's job to do that
(or to reject our code or warn that it may not have the expected
effect) in my opinion.  Will think more carefully, RTFM, and then
check SQLAlchemy tracker and file a bug or enhancement request if it
seems appropriate.

Since it seems the SQL generated by SQLAlchemy is nonconformant to
standard SQL, any of our backends might silently do this at any
upgrade.  I think *we* *should* do something about it, and I'll file a
Mailman bug tomorrow (if somebody doesn't beat me to it, hint!)  But
I'm an Internet standards geek, not a database querymonger, and I
don't know how soon I'll have time to book up on the subtleties of SQL
and SQLAlchemy.  Maybe Abhilash has the skills and can find the time,
but I don't recall Mark being an SQL hacker.  So, no promises of a
quick fix.

Steve

P.S. I'm sorry to be such a downer on the likelihood of a quick fix,
but I'm enjoying this. :-)  Always good to be learning new stuff!
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Rest API not returning members in alphabetical order

[Stephen J. Turnbull]

Goran Terzic writes:

 > Thank you everyone for looking into this.
 > 
 > Looks like the problem is that  SQLAlchemy is generating query with
 > 'order by' part inside nested subquery, which is not supported by
 > mariadb.

Thank you for finding that, I never would have!

 > https://mariadb.com/kb/en/why-is-order-by-in-a-from-subquery-ignored/
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Rest API not returning members in alphabetical order

[Stephen J. Turnbull]

Brian Carpenter writes:
 > On 4/1/20 9:22 AM, Stephen J. Turnbull wrote:

 > > Do you have the same version of the same DBMS backend in both?
 > > Same version of SQLAlchemy?
 > 
 > Alphabetically in regards to email address.

OK.  Thanks for confirming.

What REST endpoint are you using?  http://SERVER:PORT/3.1/members?
Or http://SERVER:PORT/3.1/lists/LIST-POST/roster/member?
(SERVER, PORT, and LIST-POST are specific to your installation.
SERVER:PORT is likely localhost:9001, and LIST-POST is the posting
address of the list being queried.)

 > We are using MySQL on the development server instead of Postgresql. We 
 > tried different versions of SQLAlchemy as well. Same results.

Do you mean you have MySQL on one server, and PostgreSQL on another,
and only one is misbehaving?  If so, which one?

At this point, I can't see anything wrong with the Mailman code.
Maybe somebody else will.
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Rest API not returning members in alphabetical order

[Stephen J. Turnbull]

goran.ter...@gmail.com writes:

 > I got list of members in order of creation.
 > 
 > I tried same thing on another server, and I got members ordered
 > alphabetically.

What do you mean by "alphabetically"?  "Alphabetically on display
name" is a plausible interpretation, but in fact Mailman 3 orders
alphabetically on email address.

 > Any ideas what might be causing this behavior?

Ordering of the result set is done by the DBMS backend, so the problem
is either that the ORM code (SQLAlchemy) is not communicating the
request to the DBMS, or the DBMS is buggy.  That is, .find_members is
implemented in models/subscription.py, where it requests the backend
to order by list-id, email address, and user role in that order of
precedence.  Do you have the same version of the same DBMS backend in
both?  Same version of SQLAlchemy?

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] The Viewing of Private Archives (Hyperkitty)

[Stephen J. Turnbull]

brian_carpen...@emwd.com writes:

 > [T]he following message in Hyperkitty should be reworded as it is
 > confusing:
 > 
 > "This mailing list is private. You must be subscribed to view the archives."

Issue https://gitlab.com/mailman/hyperkitty/-/issues/286 filed, with a
suggestion (but I myself think it's inadequate, please discuss).
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Screen Shots of Affinity

[Stephen J. Turnbull]

brian_carpen...@emwd.com writes:

 > Made it through the first week of development of Affinity. A decent
 > amount of progress was made today. Here is a preliminary screenshot
 > of the login page for Affinity:
 > 
 > https://static.emwd.com/uploads/2020/03/affinity_login-1030x728.png

Thanks for the update!

Looking forward to shots of the screens with config controls on them. :-)

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: New Admin Interface for Mailman 3

[Stephen J. Turnbull]

Abhilash Raj writes:

 > I've been quite out-of-sync with opensource work in past few months
 > and my responses are often delayed. Do add me to To:, Cc: if it is
 > something urgent that you'd like me response on though.

I'll be watching, and I'll be sure to pass on all the hard parts^W^W
stuff you need to be aware of until Brian and his developer(s) know
who's best for what in the source.

Thanks for the correction "list templates" -> "list styles".  Of
course that's what I meant.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: [Mailman-Users] multiple explicit reply addresses?

[Stephen J. Turnbull]

Note change from Mailman [2] Users to Mailman Developers.

Mark Sapiro writes on mailman-us...@python.org:
 > On 3/3/20 2:25 AM, Bernie Cosell wrote:

 > > It seems very explicit that the explicit reply address is
 > > singular.. would making it e...@x.com, e...@y.com work?
 > 
 > It won't work. The code uses email.utils.parseaddr() which accepts
 > various display-name and address formats, but only one address.
 > 'e...@x.com, e...@y.com' will result in only 'e...@x.com' in the Reply-To:.

I think we should check if this is the same in Mailman 3, and change
it if it is.  If we have an attribute whose main function is to
populate a header field, it should accept any valid value for that
field (POLA).  Unless there's a Very Good Reason[tm], in which case
"Errors should never pass silently." :-)

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] New Admin Interface for Mailman 3

[Stephen J. Turnbull]

brian_carpen...@emwd.com writes:

 > After much thought, I have decided to break away from the use of
 > Postorius/Hyperkitty. Tomorrow a PHP developer will begin work on
 > brand new admin interface on my behalf. Here are my goals:

I do have some comments. :-)

Before any details, I'll say the core developers applaud this effort.
Though we'd like to think Postorius and HyperKitty are great as far as
they go, we've always believed that there are many uses cases for
Mailman they can't serve well.  That includes but is not limited to
close integration with integrated platforms such as cPanel and
ticketing systems.

 > 2. To provide an easy installation approach to the setup and
 >portability of the admin interface.

This is likely to limit the use cases.  I encourage you to think
carefully about that, and to prioritize the cases you know very well,
such as the needs of hosting service administrators.  I won't be
surprised if you make a lot of ordinary users (subscribers) happy too,
but don't get overambitious.

 > 3. To reveal everything that Mailman core has via the new admin
 >interface

I myself am not sure why, but Barry has stated that it's hard to do
this, at least in the face of current pace of development of Mailman 3
core.  I love Barry and deeply respect his judgment, but I find the
idea itself offensive :-), so dealing with it on the core side is on
my own TODO list.  Let's cooperate!

 >and perhaps to bring in additional features such as the ability
 >to migrate a Mailman 2 list into a Mailman 3 list from a browser
 >interface.

This is already possible in Postorius and HyperKitty for
"well-behaved" Mailman 2 lists.  Unfortunately I don't have a
definition of "well-behaved" (yet, and I don't think it's easy to
do). :-/  OTOH, from our point of view, this is a site administrator
decision, so doing the conversion at Mailman 3 installation time and
the site administrator's convenience is the logical process.

Do you have a specific use case for this?  Do you have users (list
administrators or virtual domain administrators) who want to try
Mailman 3 lists but continue some of their lists with Mailman 2?  (I
still think this is unlikely to be something core development would
prioritize, but I'd like to have real data about it.)

 > 4. To bring in the use of stats that a list admin would like to see
 >such as overall number of list members, which list is generating
 >the most bounces, subscription/unsubscription rates, etc.

These are surely going to require core-side features.  Speaking for
myself, they are desirable features, and I'm pretty sure the core
developers will agree.

These features (and more) are already implemented in the Systers forks
of Mailman.  I can introduce some of their developers (if interested,
let's talk off-list, I don't have names and contacts off-hand).

 > 5. To provide a list creation wizard for list admins that will
 >automatically create a list that has preset settings in place
 >based upon the input during the list creation wizard.

Again, this is going to require cooperation from core.  We have a
feature called "list templates" for Mailman 3 in core, which we intend
to expose fully in Postorius and populate with common templates.  Ask
Abhilash, who knows it best I think.

 > The name of the new admin interface: Affinity

"Affinity for Mailman [3]".  I like it!

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: [Mailman-Users] The last release from the GNU Mailman project

[Stephen J. Turnbull]

@mailman-users
I'm going to get into a lot of design here, so I'm moving the thread to
mailman-developers@python.org.  Reply-To set; please respect.  Brian
kept in Reply-To as a courtesy, don't know if he's subscribed over there.

@mailman-developers Brian is planning to develop an alternative web UI
for admin and archives.  I don't see this as a disincentive to
Postorius or HyperKitty development (proposed implementation language
is PHP, so we'll still need something we can support that we can
bundle), and alternatives have always been part of the vision.  I for
one plan to cooperate, and hope others feel the same.  Thread starts
here on mailman-users:

https://www.mail-archive.com/mailman-users@python.org/msg72530.html

@Brian
If you aren't subscribed to mailman-developers, and don't want to
subscribe,, I'll try to keep you in the loop.

Brian Carpenter writes:

 > I agree and understand. The forum side is not being considered at
 > this time until we get the admin interface nailed down. Right now I
 > am looking at Discourse as a motivation for developing the forum
 > side.  I also think for mailing lists to survive in the future,
 > integrating both approaches to communications is what modern users
 > are looking for.

The Python developers looked at Discourse, I think there's actually a
fair amount of activity.  "I think" because I don't participate in
their forum server, not even entirely sure it's Discourse (checked, it
is).  I don't miss it at all.  I don't think the Discourse users miss
mailing lists at all.  There seems to be near zero crosstalk, even
less crosstalk between Discourse and the lists than there is between
the lists and the issue tracker.

I don't know what would happen with better integration.  Discourse
integration of email, uh, "is poor" IMHO, which in my opinion is an
indication that integration is somewhere between very hard and
impossible -- the original author of Discourse seems to be a brilliant
designer and programmer, with plenty of sympathy for user needs.  If
he didn't do it well, it's surely not at all easy.  A lot of people
feel as you do that "both" is the right answer, and there certainly
was a lot of demand for "both" in Python when Discourse was set up
there.

 > I just think the current interfaces and the decision to go with
 > Django has hurt Mailman 3 rather than help it.

That's assuming that the likely alternative was a non-Django framework
rather than "ssh to the host and fire up python mailman.client".  It
was "ssh to the host and fire up python mailman.client", though. ;-)

 > I also mirror Jim's question of who is the "we" in this
 > conversation.

In practice, it was an "I": Barry Warsaw started rewriting the core
around a decade ago.  Then when a beta-ready version became imminent a
few years later, a couple more "I"s, IIRC Terri Oda and Florian Fuchs
wrote much of Postorius (which Barry named), and the Fedora folks did
HyperKitty because they wanted a forum-like archiver it for the Fedora
lists.  For the last few years, both Postorius and HyperKitty have
been maintained and developed by the "Mailman project team", but those
are the folks primarily responsible for the original design decisions
AIUI.

That's how this works.  People see a need, they start hacking on it
without fanfare because they're not committed to it, once they have an
idea of how much work a commitment involves and they're OK with that
they commit and announce, which often has a chilling effect on
independent alternatives, and tends to cut out users who don't know
they need pay attention if they want input to something that will be
available years later (if at all).  I don't know what to do about the
users; Barry did talk about Mailman 3 on-list occasionally, mostly in
response to issues raised about Mailman 2.

 > Why wasn't I invited? :-D

As always in open source, everybody in general is invited, (almost)
nobody gets a personal invitation.[1]  It's unfortunate that the way
things work folks like you and Jim don't find it so easy to pop over
to mailman-developers to find out about these things in advance, but
we also don't want to burden mailman-users with nitty-gritty details
that that may never be relevant to them.

 > What prevents Mailman 3 from replacing Mailman 2 completely?

Mailman 2 ain't broke, so I don't advise people who are happy with
their installations to try to fix it.  Not even by installing Mailman
3. ;-)

 > Is it because of the interfaces for Mailman 3 totally left Mailman
 > 2 behind or was it the decision to use Django?

Mailman 3 cannot be a drop-in replacement for Mailman 2 because by
design Mailman 3 core has no comprehensive administrative or user
access, except via shell access to the Mailman server.  Otherwise, the
only user access is subscribe/unsubscribe by mail, and I don't think
there's any administrative access by mail (maybe moderation can be
enabled? but it would be disabled by default because mail is tres
insecure by default).

 > Cannot Mailman 3 

[Mailman-Developers] Improving the speed of mailman import21

[Stephen J. Turnbull]

Abhilash Raj writes:

 > 90% of the time is spent trying to encrypt user passwords, for each
 > of the imported member. Well, duh, encryption is an expensive
 > operation and when you do that once per-imported member, it is
 > definitely going to be slow.

Why are we storing unencrypted passwords at all?  Passwords are pretty
low-security in any case, but this is asking for trouble.

 > Although, another interesting fact is the user passwords are kind
 > of useless in Mailman 3. In Mailman 2 you had to setup a password
 > or one was auto-generated for you per-list and you needed that to
 > login to the web ui. However, in Mailman 3, the passwords (in
 > Core's database) aren't used for logging in since Web Frontend
 > stores the authentication tokens (social auth or passwords). In
 > fact, the users who sign up first time on Mailman 3 probably don't
 > ever have a password set in Mailman Core's database.

I'll trust you on that.  Although it suggests the question, if nobody
has a password, why does it take so much time to encrypt no passwords?

 > So, I commented out the code that actually imports the
 > password(src/mailman/utilities/importer.py#L663-664)

I'm happy with this.  This is a major breaking change *if* anyone is
using core passwords which they probably aren't, but it deserves
flashing lights and sirens in the release announcements.

Steve

-- 
Associate Professor  Division of Policy and Planning Science
http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information
Email: turnb...@sk.tsukuba.ac.jp   University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] [Mailman-cabal] Re: removal from Debian

[Stephen J. Turnbull]

Moving to -developers, I see no reason why this should be private.

On Sat, Sep 28, 2019, at 2:33 PM, bendik...@vfemail.net wrote:

 > libmatheval will be removed from Debian 21-10-2019
 > Also causes the removal of (transitive) reverse dependencies:
 > mailman3 and mailman-suite 

Abhilash writes:

 > What is the reason for this removal? Is it because the project is
 > unmaintained or something else?

The most recent version was released in 2013.  I would guess there are
no maintainers.

 > Mark Sapiro writes:

 > I would like to know exactly what in the Mailman 3 stack depends on
 > libmatheval because whatever it is should be broken in these installs.

According to apt, uswgi-core 2.0.18-* requires libmatheval1.  Perhaps
it's demand-loaded?

Steve

-- 
Associate Professor  Division of Policy and Planning Science
http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information
Email: turnb...@sk.tsukuba.ac.jp   University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Can we disable archiving by default

[Stephen J. Turnbull]

sandeep kumar writes:

 > I tried many options but I am not able to do so.

It helps if you tell us what you tried and didn't work.

As far as timing goes, cron is your friend.  It should be possible to
curl or the mailman command interface to access the REST interface to
enable and disable archives.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Thanks to Community and Follow Up after GSoC

[Stephen J. Turnbull]

Aaryan,

Welcome back!  Sorry to hear about your arm, but will be praying for
your recovery.  It sounds like you've managed your school tasks, and
I'm happy to hear that!  Looking forward to working with you again,
but take your time, and don't neglect your schoolwork!

Steve

-- 
Associate Professor  Division of Policy and Planning Science
http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information
Email: turnb...@sk.tsukuba.ac.jp   University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Implementing `bounce_you_are_disabled_warnings_interval` in `Send_Warnings` function.

[Stephen J. Turnbull]

Sorry for the delay, family issues and $WORK stuff.

Aaryan Bhagat writes:

 > Am I missing something here?

No, I don't think so.  *We* (Mailman) may be able to improve something
here, but it's not in scope of your GSoC work as far as I can see.
Your opinions (and if you choose, post-GSoC work) on these questions
will be appreciated since it's in the same area you're getting a lot
of experience in.

Steve

-- 
Associate Professor  Division of Policy and Planning Science
http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information
Email: turnb...@sk.tsukuba.ac.jp   University of Tsukuba
Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Migrating mailman 2 to mailman 3.1

[Stephen J. Turnbull]

sandeep kumar writes:

 > Now that we are ready to go into production. We are currently
 > running mailman 2 in our production. Now the biggest challenge is
 > we have to migrate around 3000 lists from mailman 2 to mailman 3.
 > Is there any documentation

There's a small amount, but we're not satisfied with it.  We have an
on-going Season of Docs project to create migration documentation.

 > or any inputs from the team would be appreciated and will be very
 > helpful.

Abhilash and Mark have substantial experience with migrating lists,
and operating Mailman 2 in parallel with Mailman 3 (ie, gradual
migration, not the same list on both servers!)  However, they've been
doing this at scale of dozens of lists, maybe a dozen at a time.

I hope that some of our really large-scale users will have something
to say, but the cases I'm aware of are at your stage more or less --
no post-mortems yet that I know of.

 > Iam trying to access docs.mailman3.org but it is not opening don't
 > why.?

It appears that recent Firefox is very picky about the TLS versions it
will connect to, and somehow mailman3.org doesn't support that.
Abhilash?

In the meantime, try other browsers which may not be so picky about
crypto algorithms.  Also you can try mailman.readthedocs.io.

Steve
___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9

[Mailman-Developers] Re: Implementing `bounce_you_are_disabled_warnings_interval` in `Send_Warnings` function.

[Stephen J. Turnbull]

Aaryan Bhagat writes:

 > I am sorry but I do not understand your concern. I just mentioned
 > the cases where we actually have to make changes to the database
 > after each processing of `BounceEvents`. I do not expect `Mailman`
 > to show any undesirable behaviour here.

You wrote some posts that indicated you were specifically worried
about the timing of the warning mails.  When the conversation
continued without much explanation of your concerns, I assumed that
you were still thinking about the concurrency issues that arise in an
asynchronous system such as email.

___
Mailman-Developers mailing list -- mailman-developers@python.org
To unsubscribe send an email to mailman-developers-le...@python.org
https://mail.python.org/mailman3/lists/mailman-developers.python.org/
Mailman FAQ: https://wiki.list.org/x/AgA3

Security Policy: https://wiki.list.org/x/QIA9