Re: [Mailman-Users] Trimming archives

[Mark Sapiro]

Darren G Pifer wrote:
>
>   In reviewing posts to this mailing list, I found that it is not
>possible to prune the archives.


I don't know what gave you that idea. Perhaps you saw my post at
http://mail.python.org/pipermail/mailman-users/2005-October/047180.html,
but that only said "There is no option within Mailman to prune old
messages from the archives." It meant an automatic process. There are
lots of ways to do it manually.


>However, we will need to do something
>shortly because the filesystem that we have mailman on is now 84% full,
>mostly due to the growing archives. We are using LVM under Linux and I
>can increase the space but I would like to see if we can compress any
>older files in the archive directory first. For instance, we have one
>mbox, ./archives/private/opr.mbox/opr.mbox that is 700MB is growing
>everyday. It is our operators list. Can I gzip it and create a new
>opr.mbox by touching it?


You can gzip it and Mailman will create a new one automatically for the
next message.

If you don't need the old archives online, you can do much more, but
there is a CAVEAT which may or may not be important. If you do the
following, the URLs of the remaining individual archive messages will
change and any saved links to these messages will break or return the
wrong message.

Here's the process.

Back up archives/private/opr.mbox/opr.mbox off line. Edit the file and
delete all but the recent entries that you want to keep. Then do

bin/arch --wipe opr

This will rebuild the entire archive with only the recent messages.

If preserving the message numbers and recent URLs is important, you
have a couple of choices. Instead of deleting the old messages from
the .mbox, you could edit them to replace the message body with
something like "Message archived off line." and then rebuild. Or you
could just gzip or backup off line the .mbox and let Mailman create a
new one starting with the next message. This way, the HTML archive
will still be complete, but if you ever need to rebuild it, you have a
more complex process. You will have to recreate the full .mbox at
least temporarily and rebuild the archive from that or you can do it
in pieces like

bin/arch --wipe opr archived_opr.mbox
bin/arch opr

which recreates the archive with the old messages and then adds the
current messages.


Also see Brad's reply in this thread
.

-- 
Mark Sapiro <[EMAIL PROTECTED]>   The highway is for gamblers,
San Francisco Bay Area, Californiabetter use your sense - B. Dylan

--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

Re: [Mailman-Users] Trimming archives

[Brad Knowles]

At 9:10 AM -0500 2005-10-31, Darren G Pifer wrote:

> In reviewing posts to this mailing list, I found that it is not
>  possible to prune the archives.

Not automatically, no.

>   However, we will need to do something
>  shortly because the filesystem that we have mailman on is now 84% full,
>  mostly due to the growing archives. We are using LVM under Linux and I
>  can increase the space but I would like to see if we can compress any
>  older files in the archive directory first.

There are cron jobs for Mailman that are detailed in the 
documentation which will periodically compress the old web-accessible 
text format archives, yes.  The HTML-format web archives themselves 
cannot be compressed, however.  And the original 7th edition 
mbox-format mailbox (from which all the public archives are derived) 
cannot be compressed as-is.

So, there's a limit to the amount of space that you would be able 
to recover.  Moreover, your choices are limited in terms of the 
compression algorithms you can choose, at least without making any 
modifications to the Mailman source code -- for the standard cron 
jobs, gzip is used and not bzip2.

>   For instance, we have one
>  mbox, ./archives/private/opr.mbox/opr.mbox that is 700MB is growing
>  everyday. It is our operators list. Can I gzip it and create a new
>  opr.mbox by touching it?

This is the unprocessed "source" archive mailbox, from which all 
public archives are generated.  This is a record of exactly what came 
in and in what order.

I would be inclined to rename it first, using an extension based 
on the date when the archive was to be compressed, then create the 
new one with the same ownership and permissions.  You could then 
compress the old file at your leisure.

Of course, if you ever have to rebuild your public archives, 
you'll need to make sure that you process the messages in the oldest 
compressed archive first, otherwise all the message numbers will get 
out-of-whack.

>Also, there are some text files
>  ./archives/private/opr/2005-October.txt that is 500+ MB.

Those are the processed text-format web-accessible archives which 
were created based on the source archive mailbox contents.  These are 
the archives which are typically compressed in the standard cron jobs.

>Can .txt files
>  compressed and still useable by the system?

Mailman doesn't make any attempt to use these files.  These files 
are produced by Mailman for use by humans.  Once they've been 
compressed, it's fine to leave them in compressed format, since they 
would only be downloaded by people accessing the archive system and 
who wish to see text-format archives as opposed to the HTML-format 
archives.

>  Is anybody doing anything to
>  archive older, unused mailman archives to tape? We use Tivoli (TSM) here
>  and wanted to know if anybody has something automated for this.

There are no standard automated procedures I know of to split and 
compress old 7th edition mbox-format source archives.  They are 
always assumed to be in pure 7th edition mbox text format, and if you 
want to split and compress them you will need to come up with your 
own procedures for doing that.

-- 
Brad Knowles, <[EMAIL PROTECTED]>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

 -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
 Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See  for more info.
--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

[Mailman-Users] Trimming archives

[Darren G Pifer]

Hello,

   In reviewing posts to this mailing list, I found that it is not
possible to prune the archives. However, we will need to do something
shortly because the filesystem that we have mailman on is now 84% full,
mostly due to the growing archives. We are using LVM under Linux and I
can increase the space but I would like to see if we can compress any
older files in the archive directory first. For instance, we have one
mbox, ./archives/private/opr.mbox/opr.mbox that is 700MB is growing
everyday. It is our operators list. Can I gzip it and create a new
opr.mbox by touching it? Also, there are some text files
./archives/private/opr/2005-October.txt that is 500+ MB. Can .txt files
compressed and still useable by the system? Is anybody doing anything to
archive older, unused mailman archives to tape? We use Tivoli (TSM) here
and wanted to know if anybody has something automated for this.

Darren
ODU



--
Mailman-Users mailing list
Mailman-Users@python.org
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: 
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Security Policy: 
http://www.python.org/cgi-bin/faqw-mm.py?req=show&file=faq01.027.htp

Re: [Mailman-Users] trimming archives

[Dave Stern - Former Rocket Scientist]

On Thu, 3 Apr 2003, Dave  Stern - Former Rocket Scientist wrote:

> How do you trim back archives? I presume
>
> find {$MAILMAN}/archives/private -name "*.gz" -exec rm "{}" \;
>
> would be a bad idea as the indices still think they exist (or would
> the cronjobs fix this)
>


Before I get a flurry of RTF-faq messages, perhaps I should explain in better
detail. Let's say I wanna remove ALL archived data prior to 2003. I could

find {$MAILMAN}/archives/private -name "2002-*" -exec rm -rf "{}" \;

which will take care of the txt an gzipped files and running
{$MAILMAN}/bin/arch `{$MAILMAN}/bin/list_lists`
would theoretically rearchive them but the problem is the index.html files
for each list still has a stanza for each month.

What I'm looking for is an automated way of cleaning things up. I don't
wanna have to manually edit the idex.html altho it sounds like I may have to
ie something like doing another find for index.html, exec sed where it
searches for "2002", go back a line to the beginning of the  stanza,
delete to EOF, and re-addYech


 =-=-=-=-=-=-=-=-=-=-=-=-  generated by /dev/dave -=-=-=-=-=-=-=-=-=-=-=-=-=-=
 David SternUniversity of Maryland
Institute for Advanced Computer Studies


--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

This message was sent to: [EMAIL PROTECTED]
Unsubscribe or change your options at
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

[Mailman-Users] trimming archives

[Dave Stern - Former Rocket Scientist]

How do you trim back archives? I presume

find {$MAILMAN}/archives/private -name "*.gz" -exec rm "{}" \;

would be a bad idea as the indices still think they exist (or would
the cronjobs fix this)

Thanks



 =-=-=-=-=-=-=-=-=-=-=-=-  generated by /dev/dave -=-=-=-=-=-=-=-=-=-=-=-=-=-=
 David SternUniversity of Maryland
Institute for Advanced Computer Studies


--
Mailman-Users mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://www.python.org/cgi-bin/faqw-mm.py
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/

This message was sent to: [EMAIL PROTECTED]
Unsubscribe or change your options at
http://mail.python.org/mailman/options/mailman-users/archive%40jab.org

Re: [Mailman-Users] trimming archives

[David]

Does this mean that every thing is built based on the mbox file, in my
case: /archives/private/forum.mbox/forum.mbox ?

If I edit the offending emails from that file, do I then run 

mailman@host ~$ rm -rf /home/mailman/archives/private/forum/*
mailman@host ~$ /bin/arch forum /archives/private/forum.mbox/forum.mbox

Sorry to be pedantic, but I don't want to lose the archive that I need to
keep by making a dumb mistake.

Many thanks... David


On Sun, 21 Jan 2001, Dan Mick wrote:

> 
> 
> David wrote:
> > 
> > I have just set up a new list, and have a number of test messages in the
> > archive that I would rather weren't there.
> > 
> > Is there an easy way to do this - ie, selectively delete threads or
> > messages?
> 
> Depends on what you mean by easy, but you can just edit the .mbox file,
> delete the archives, and run bin/arch to regenerate them.
> 
> --
> Mailman-Users maillist  -  [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/mailman-users
> 


--
Mailman-Users maillist  -  [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users

Re: [Mailman-Users] trimming archives

[Dan Mick]

David wrote:
> 
> I have just set up a new list, and have a number of test messages in the
> archive that I would rather weren't there.
> 
> Is there an easy way to do this - ie, selectively delete threads or
> messages?

Depends on what you mean by easy, but you can just edit the .mbox file,
delete the archives, and run bin/arch to regenerate them.

--
Mailman-Users maillist  -  [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users

[Mailman-Users] trimming archives

[David]

I have just set up a new list, and have a number of test messages in the
archive that I would rather weren't there.

Is there an easy way to do this - ie, selectively delete threads or
messages?

David.


--
Mailman-Users maillist  -  [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/mailman-users