Bug#968244: [Pkg-mailman-hackers] Bug#968244: mailman3-web: Crash with: /usr/share/mailman3-web/manage.py runjobs daily

[Athanasius]

On Sat, Jan 23, 2021 at 09:47:10PM +0100, Jonas Meurer wrote:
> Could you please try out whether this bug has been fixed with the latest
> updates to mailman3 and hyperkitty? A lot of changes happened to the
> 'import21' code, so I'm hopeful that this bug might have been fixed in the
> meantime. Unfortunately I lack the time to build a reproducer myself.

  I also filed an upstream bug about this.  We tracked down the issue as
being related to sending email addresses over an API without ensuring
they are URL encoded.  A '#' (hash) character in them causes the
receiving API server to truncate at that point, treating the '#' as a
URL anchor which only has signifigance on the browser side (when there's
actually a browser involved).

  The last I heard about that bug was 4 months ago:



Pulling the latest git master of that code, head
177c123074ece0e64fc317cd78eea1d68fec1b88 I see no changes to the
relevant code.  Indeed `git log -- src/mailmanclient/client.py` only
shows one change since '4 months ago', which is simply a copyright
end-year bump across all files.

-- 
- Athanasius = Athanasius(at)miggy.org / https://miggy.org/
  GPG/PGP Key: https://miggy.org/gpg-key
   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME


signature.asc
Description: PGP signature

Bug#968244: [Pkg-mailman-hackers] Bug#968244: mailman3-web: Crash with: /usr/share/mailman3-web/manage.py runjobs daily

[Jonas Meurer]

Control: tag -1 moreinfo

Hello Athanasius,

Am 11.08.20 um 16:45 schrieb Athanasius:

After installing mailman3 I used migrated old Mailman 2.1 lists in using
'import21' and then their archives using 'python manage.py
hyperkitty_import -l ...'.

All seems to have gone OK, but the daily mailman3-web job in
/etc/cron.d/mailman3-web:

@daily www-data [ -f /usr/bin/django-admin ] && flock -n 
/var/run/mailman3-web/cron.daily /usr/share/mailman3-web/manage.py runjobs daily

is throwing errors:

Traceback (most recent call last):
   File 
"/usr/lib/python3/dist-packages/django_extensions/management/commands/runjobs.py",
 line 36, in runjobs
 job().execute()
   File "/usr/lib/python3/dist-packages/hyperkitty/jobs/sync_mailman.py", line 
35, in execute
 sync_with_mailman()
   File "/usr/lib/python3/dist-packages/hyperkitty/lib/mailman.py", line 145, 
in sync_with_mailman
 sender.set_mailman_id()
   File "/usr/lib/python3/dist-packages/hyperkitty/models/sender.py", line 54, 
in set_mailman_id
 mm_user = client.get_user(self.address)
   File "/usr/lib/python3/dist-packages/mailmanclient/client.py", line 322, in 
get_user
 return User(self._connection, content['self_link'], content)
KeyError: 'self_link'
ERROR OCCURED IN DAILY JOB: sync_mailman (APP: hyperkitty)
START TRACEBACK:
END TRACEBACK

Adding a little debug output:

Address: #.pleasegivegenerou...@hotmail.com
Response: {'date': 'Tue, 11 Aug 2020 14:35:42 GMT', 'server': 'WSGIServer/0.2 
CPython/3.7.3', 'content-length': '712742', 'content-type': 'application/json; 
charset=UTF-8', 'status': '200', 'content-location': 
'http://river-nat:8001/3.0/users/#.pleasegivegenerou...@hotmail.com'}

And the 'content' at that point is *huge* (693KiB), indeed without a
'self_link' key at its top level, although it contains an 'entries'
array in which each member has a 'self_link' key.

This appears to be code to sync up hyperkitty's idea of senders with
actual members of the list, so is probably relatively harmless, but I
don't know if *subsequent* more important code is being prevented from
running.


Could you please try out whether this bug has been fixed with the latest 
updates to mailman3 and hyperkitty? A lot of changes happened to the 
'import21' code, so I'm hopeful that this bug might have been fixed in 
the meantime. Unfortunately I lack the time to build a reproducer myself.


Cheers
 jonas




OpenPGP_signature
Description: OpenPGP digital signature