Wow, what a cool problem... I think that v 2.1 (once it is out of beta) is for you. You'll be able to move that database into a PostGresSQL database or any database of your choice.
I do hope that you are running a SCSI disk subsystem (or better). That will help immensely with resource problems. Here is an obscure thought - what if you actually set aside some memory for a ram drive - say about 64 Mb and then backed up the directory ~mailman/lists/<listname>/.. to the RAM drive, then moved the physical mapping of that directory and mapped the RAM drive there. Your database and its backups/rebuilds would all take place in RAM. They would be very fast so you would have less simultaneous accesses to the database. Every hour copy the database from RAM back to your HD somewhere. Alternately, you could make your main list an umbrella list and then pull folks out of the main list and into a sublist. Mailing addresses starting with an A would go to one list, the B's to another, the Z's to another list and any numeric emails to yet another list. (you would have to add each sub-list to the main list, just like a user). Folks would still subscribe to the main/umbrella list. They would stay there till that night, when your cron ran a script to pull them out and re-subscribe them to their proper sublist. That way all your subscribes are taking place on a nearly empty list... and you control when all the heavy lifting occurs (the movement of folks into the larger sub-lists). You will have to generate your own listinfo page (not really all that hard - just write a simple html page that does what you want and stick that in place of the current Listinfo page - using the web admin). Your list info page could ask folks to enter their email address and then you would use some back-end logic to isolate the first letter and transfer them to the proper Listinfo page. From there, they would be okay! Good Luck. Keep us in the loop and let us know what your do! Jon ----- Original Message ----- From: "Juergen Geinitz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, April 18, 2002 8:18 AM Subject: [Mailman-Users] Trouble with LARGE mailing list > Hello, > I have a mailing list here running for one of our customers > which is getting extremely large and therefore slow. > The list is a newsletter list with one post a week - that doesn't > harm me. But I hav about 500 new subscribes a day which makes up to now > about 150000 members - and thats my problem: > the config.db is > -rw-rw---- 1 www-data www-data 15587680 Apr 18 14:13 config.db > > and therefore is blowing my main memory. Lokk at this output of "top" > 31084 www-data 18 0 81M R 48.7 10.8 0:07 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31262 www-data 20 0 42M R 48.1 5.5 0:03 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31260 geinitz 13 0 2456 R 0.9 0.3 0:00 top > 30983 www-data 9 0 73M do_select S 0.3 9.7 0:11 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31102 www-data 9 0 42M do_select S 0.3 5.6 0:04 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31120 www-data 10 0 73M do_select S 0.3 9.7 0:11 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31219 www-data 9 0 42M do_select S 0.3 5.6 0:04 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31240 www-data 9 0 42M do_select S 0.3 5.6 0:04 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 11 root 9 0 0 mosix_mem SW 0.1 0.0 10:07 memsorter > 30682 www-data 9 0 984 do_select S 0.1 0.1 0:05 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31220 www-data 9 0 42M do_select S 0.1 5.6 0:03 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31222 www-data 9 0 42M do_select S 0.1 5.6 0:03 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > 31242 www-data 9 0 42M do_select S 0.1 5.6 0:03 > /usr/bin/python -S /usr/lib/mailman-aldi/scripts/driver subscribe > > so every subscribe is between 40 an 80 M in core. At the moment the > system is running an own apache > for this with a connect limit of 15 which gives 768M main memory used > plus 200M swap. When I remove > the apache limit, the system goes into swapping - more subscribe > requests arrive - less memory > an about 5 minutes later - BANG > > Any Idea? > Any hint? > > Regards > Juergen > -- > novalis media > Ein Bereich der Vereinigte Verlagsanstalten GmbH - > http://www.vva.de > http://www.novalis-media.de > Geschaeftsfuehrer Stefan Meutsch HRB 658 Duesseldorf > > > ------------------------------------------------------ > Mailman-Users mailing list > [EMAIL PROTECTED] > http://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py ------------------------------------------------------ Mailman-Users mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
