Hi Mark, Thanks again for your answer.
First of all, all of what I write below happened on a test list server, so no real harm was done. I'm just curious how to recover from this, and how to get to what I need. :-) On Wed, 11 Sept 2024 at 21:16, Mark Sapiro <m...@msapiro.net> wrote: > On 9/11/24 00:23, Marco van Tol wrote: > > > > We're interested in the development of a feature in mailman3 where we can > > configure it to automatically expire/remove threads in/from the archive > > older than x number of days. > > The script at https://www.msapiro.net/scripts/prune_arch3 could be > easily modified to do this and then be run periodically by cron. > I made a modified version of the script, and ran it. (attachment: script-1.py) The script went on its way for a bit, and then blew up. (attachment: error-run-1.txt) Now the archive for the list is broken in the sense that when I search for "*", and order it by "earliest first", it will show a server error. The mailmanweb.log also gives errors when this happens. (attachment: mailman-web.log) I tried to run `./manage.py update_index` which did not fix the issue for the archive. I then ran `./manage.py rebuild_index` which did fix the issue for the archive. Following this I ran the script again, and it showed similar output, including an error message after a while, but on every run it would delete new messages. I could keep running it until all the messages that I needed to be gone were gone, and then do a final `rebuild_index` to get the server back in shape. The mailmanweb.log would show the occasional message like this while I was doing this: WARNING 2024-10-16 12:31:52,043 41 hyperkitty.tasks Cannot rebuild the thread cache: thread 28 does not exist. These did not re-appear after I rebuilt the index. Is there some call I need to make to refresh the Email.objects list between runs? Do I just blindly rerun `Email.objects.filter(<args>)` after every call to msg.delete()? And secondary: can I avoid the call to rebuild_index? The actual production server has a massive count of messages. Thanks! Marco van Tol RIPE NCC
f04331d81855:/opt/mailman-web-data$ python tol-test-3.py 2006-01-01
276
chair.ripe.net 2005-05-04 12:23:11+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 56
12:04:15 [Q] INFO Enqueued [default] 57
chair.ripe.net 2005-04-28 09:38:04+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 58
12:04:15 [Q] INFO Enqueued [default] 59
chair.ripe.net 2005-04-26 20:02:54+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 60
12:04:15 [Q] INFO Enqueued [default] 61
chair.ripe.net 2005-04-26 23:37:03+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 62
chair.ripe.net 2005-04-27 00:01:24+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 63
chair.ripe.net 2005-04-27 10:51:32+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 64
chair.ripe.net 2005-04-27 07:44:01+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 65
12:04:15 [Q] INFO Enqueued [default] 66
chair.ripe.net 2005-05-04 12:32:51+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 67
12:04:15 [Q] INFO Enqueued [default] 68
chair.ripe.net 2005-04-30 19:16:23+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 69
chair.ripe.net 2005-05-02 13:25:15+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 70
chair.ripe.net 2005-05-04 12:51:34+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 71
chair.ripe.net 2005-05-04 12:16:24+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 72
chair.ripe.net 2005-09-23 07:33:52+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 73
chair.ripe.net 2005-05-05 09:30:44+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 74
chair.ripe.net 2005-05-05 12:05:55+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 75
chair.ripe.net 2005-05-20 14:22:36+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 76
12:04:15 [Q] INFO Enqueued [default] 77
chair.ripe.net 2005-05-20 15:04:23+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 78
chair.ripe.net 2005-05-24 23:16:32+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 79
chair.ripe.net 2005-07-28 01:09:13+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 80
chair.ripe.net 2005-07-25 13:43:37+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 81
12:04:15 [Q] INFO Enqueued [default] 82
chair.ripe.net 2005-07-27 07:35:36+00:00 <redacted>
12:04:15 [Q] INFO Enqueued [default] 83
chair.ripe.net 2005-08-25 12:24:57+00:00 <redacted>
12:04:16 [Q] INFO Enqueued [default] 84
12:04:16 [Q] INFO Enqueued [default] 85
chair.ripe.net 2005-08-26 22:50:05+00:00 <redacted>
Traceback (most recent call last):
File
"/usr/lib/python3.12/site-packages/django/db/models/fields/related_descriptors.py",
line 218, in __get__
rel_obj = self.field.get_cached_value(instance)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/fields/mixins.py",
line 15, in get_cached_value
return instance._state.fields_cache[cache_name]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'parent'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/mailman-web-data/tol-test-3.py", line 31, in <module>
msg.delete()
File "/usr/lib/python3.12/site-packages/django/db/models/base.py", line 1132,
in delete
return collector.delete()
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/deletion.py", line
463, in delete
signals.pre_delete.send(
File "/usr/lib/python3.12/site-packages/django/dispatch/dispatcher.py", line
177, in send
(receiver, receiver(signal=self, sender=sender, **named))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/hyperkitty/signals.py", line 61, in
Email_on_pre_delete
kwargs["instance"].on_pre_delete()
File "/usr/lib/python3.12/site-packages/hyperkitty/models/email.py", line
256, in on_pre_delete
if self.parent is None:
^^^^^^^^^^^
File
"/usr/lib/python3.12/site-packages/django/db/models/fields/related_descriptors.py",
line 236, in __get__
rel_obj = self.get_object(instance)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/lib/python3.12/site-packages/django/db/models/fields/related_descriptors.py",
line 199, in get_object
return qs.get(self.field.get_reverse_related_filter(instance))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 637,
in get
raise self.model.DoesNotExist(
hyperkitty.models.email.Email.DoesNotExist: Email matching query does not exist.
import os
import sys
import django
from datetime import datetime, timezone
sys.path.insert(0, '/opt/mailman-web')
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
django.setup()
from hyperkitty.models.email import Email
def usage():
print(f'Usage: {sys.argv[0]} yyyy-mm-dd', file=sys.stderr)
sys.exit(1)
if len(sys.argv) != 2:
usage()
try:
year, month, day = sys.argv[1].split('-')
except ValueError:
usage()
try:
cutoff = datetime(int(year), int(month), int(day), tzinfo=timezone.utc)
except (TypeError, ValueError):
usage()
count = 0
print(Email.objects.filter(mailinglist=4, date__lt=cutoff).count())
for msg in Email.objects.filter(mailinglist=4, date__lt=cutoff):
print(msg.mailinglist, msg.date, msg.sender)
msg.delete()
count += 1
print(f"Deleted {count} messages")
mailman-web.log
Description: Binary data
_______________________________________________ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
