Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Osamu Aoki]

control: tags -1 wontfix

On Tue, Aug 08, 2017 at 12:10:07PM -0600, Charles Cazabon wrote:
> 積丹尼 Dan Jacobson  wrote:
> > 
> > OK, now all you need to do is add to the FAQ:

Sigh ...

> > Q: Do the oldmail files just grow and grow?
> 
> Not only is that not a Frequently Asked Question, it is a Never Asked
> Question.

Well said.  Thanks, Charles.

> When I get users asking me this, I'll worry about it.

Excuse me not convincing Dan to stop.  He goes hyperbolic on some bug
reports.

Dan,

I think you mean well but your bug reports like this waste many people's
time.  Please think about it. I am marking this as a "wontfix" bug.

I am not closing this as a reminder to you.

Regards,

Osamu

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

積丹尼 Dan Jacobson  wrote:
> 
> OK, now all you need to do is add to the FAQ:
> 
> Q: Do the oldmail files just grow and grow?

Not only is that not a Frequently Asked Question, it is a Never Asked
Question.

When I get users asking me this, I'll worry about it.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[積丹尼 Dan Jacobson]

> "CC" == Charles Cazabon  writes:
CC> getmail doesn't remove them /immediately/.  It waits 30 days or so

Ah!, confirmed with:

for i in `ls -S ~/.getmail/oldmail*|sed q`; do
perl -F\\0 -anwle 'print $F[1]' $i|sort -n|sed q|LC_ALL=C xargs -i date -d 
@{}
done

OK, now all you need to do is add to the FAQ:

Q: Do the oldmail files just grow and grow?

A: No. Entries are removed 30 days after the items they refer to are
deleted from the server.

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

積丹尼 Dan Jacobson  wrote:
> > "CC" == Charles Cazabon  writes:
> > getmail will remove entries from them that it knows are no longer needed
> 
> That is great to know! However
[...]
> all say "deleted", but in the oldmail file one still sees [them]

getmail doesn't remove them /immediately/.  It waits 30 days or so, for
various reasons that I don't much feel like explaining in detail - if you're
interested, read the code and the getmail mailing list archives.

> etc. So maybe there really is a bug!

I don't think so.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[積丹尼 Dan Jacobson]

> "CC" == Charles Cazabon  writes:
CC> getmail will remove entries from them that it knows are no longer needed

That is great to know! However
2017-08-08 20:37:20 msg  1/34 (3122 bytes) msgid 1243652154/79861 from 
 delivered to MDA_external command procmail (), deleted
2017-08-08 20:37:21 msg  2/34 (3391 bytes) msgid 1243652154/79862 from 
 delivered to MDA_external command procmail (), deleted
2017-08-08 20:37:21 msg  3/34 (3409 bytes) msgid 1243652154/79863 from 
 delivered to MDA_external command procmail (), deleted
2017-08-08 20:37:22 msg  4/34 (3410 bytes) msgid 1243652154/79864 from 
 delivered to MDA_external command procmail (), deleted
all say "deleted", but in the oldmail file one still sees
1243652154/79867^@1502195833
1243652154/79864^@1502195833
1243652154/79865^@1502195833
1243652154/79862^@1502195833
etc. So maybe there really is a bug!
The mail is no longer upsteam, no longer downstream, so there is no
longer a need for a record, no?

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

積丹尼 Dan Jacobson  wrote:
> I don't plan to backup my oldmail files at all.

If you don't mind getmail re-retrieving all your old mail after you lose them,
then that's fine.  Otherwise, you need to back them up.

> And having the user restore a days old stale oldmail file from backups
> would perhaps be worse than no oldmail file at all.

Actually, you're completely wrong there.  With no oldmail file, getmail will
re-retrieve all old messages still on the server.  With a days-old backup of
your oldmail file, getmail will only retrieve those few days' worth of old
messages.

Having the backup is clearly better.

> Even if disks are bigger these days it is simply bad practice to
> cross ones fingers and let files grow and grow.

Actually, you're wrong there too.  The data in the oldmail files is not
"unused".  It's all internal state data that is still needed.  And given that
most users will only have tens to hundreds of kilobytes of oldmail data, and
that 200k of oldmail data is 1/640,000 of the size of a tiny 128GB SSD, the
storage space is NOT AN ISSUE.  Not at all.  Not even a little bit.

> I'm sure there must be some entries in a 10 years big oldmail file that
> could be deleted.

You're wrong there, too.  Many users leave their mail on the server after
retrieval; ALL of the contents of the oldmail file are needed on an ongoing
basis.  getmail will remove entries from them that it knows are no longer
needed; there is no manual maintenance of these files necessary at all.

You do not appear to know how getmail uses this data or how it works; your
suggestions are invalid.  I would also like to know what exactly you are
trying to accomplish - you proposed a solution ("I want to edit or trim my
oldmail files") without actually saying what problem you're trying to solve.
If you can explain what the problem you're trying to solve is, perhaps I can
point you to the correct solution, because editing the oldmail files is not
it.  You can contact me through the getmail users' mailing list for further
discussion of getmail.

Maintainer, please close this bug as invalid.  There is no issue here to
solve.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[積丹尼 Dan Jacobson]

Like one has different backup procedures for /etc vs. /var,
I don't plan to backup my oldmail files at all.

And having the user restore a days old stale oldmail file from backups
would perhaps be worse than no oldmail file at all.

http://pyropus.ca/software/getmail/faq.html#faq says

Do NOT delete or edit these files. You'll make getmail re-retrieve all
your old mail, or even prevent getmail from running. The files are tiny
by modern storage standards; you could have a million of these files and
still not have to worry about the disk space they take up for a thousand
years.

Even if disks are bigger these days it is simply bad practice to
cross ones fingers and let files grow and grow. E.g., that's why we use
logrotate, and browsers limit the history they save. Environmentally
conscious programs all clean up after themselves and I can't think of
any programs that don't.

And sure disks might be big but it is an embarrassment when there is
this one file bigger than any other that nobody dares to delete because
the FAQ says so. Also yes disks might be bigger but some people might be
backing up important /var files over cellular or satellite etc.

I'm sure there must be some entries in a 10 years big oldmail file that
could be deleted. Maybe Osamu could make a script to delete obsolete
entries.

Anyway, here's how I use getmail, several times a day:

Dreamhost ->(getmail)->~/Maildir, then
~/Maildir->(gnus)->~/[gnus directories](Then I read my mail in gnus.)

At this point, there is no mail left on Dreamhost, and no mail in
~/Maildir, so I am certainly back to the same state of Day 1 when I
first installed getmail, so I can certainly zap the oldmail files.

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

Correction:

Charles Cazabon  wrote:
> You can remove all the messages you want from your destination maildirs or
> mbox files with affecting getmail, [...]

That's a typo.  I meant "without affecting getmail".

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

積丹尼 Dan Jacobson  wrote:
> What if my house burned down and I must start from scratch?
> Mention how to continue if those files are lost.

You have two choices.

1. Restore the files from your backups.  You do take regular backups, right?
You know the saying "Files you don't have backed up are files you don't want
to keep"?

2. Reconfigure getmail and let it retrieve all your old mail again.  This will
recreate your oldmail files.

> I assume if one really wanted to get rid of them,

That's a nonsensical statement.  If you are running getmail and want to
continue running getmail, you *DON'T* want to get rid of them.  And there is
no reason, and definitely no need, to screw with them.

> a safe way would be
> 0. Disconnect the upstream mail server from the internet so no new mail
> is coming in.
> 1. get all your mail upstream -> downsteam. Now all is deleted from
> upstream.
> 2. Now delete the oldmail files.
> 3. Now reconnect the server.

Sorry, but you seem to have completely misunderstood how this works.  That
wouldn't do what you think it would - not even close.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

Osamu Aoki  wrote:
> "CC" == Charles Cazabon  writes:
> > 
> > In short: no, you can never safely trim them.  You can remove them only if
> > you never want to retrieve mail from the corresponding POP/IMAP account or
> > folder again.  The files are absolutely tiny by any modern standard, so
> > there will never, ever be a need to "trim" them.
> 
> Is this question on mbox file?

Perhaps this was the original confusion.  To clarify a few points, in case the
OP wants to know:

1. oldmail files are NOT mbox files or any other type of email file.  They do
not contain email messages.  They only contain internal state used by getmail
to remember whether messages in the mailbox being retrieved from have been
previously seen and when.

2. do not try to edit oldmail files.  There is no possible benefit, and you
*will* cause errors ranging from getmail re-retrieving all your old mail to
getmail failing to be able to retrieve from your POP/IMAP account at all.

3. getmail doesn't care what happens to mail *after* it has successfully
delivered it according to your getmailrc file(s).  You can remove all the
messages you want from your destination maildirs or mbox files with affecting
getmail, but with mbox files make sure you follow the proper locking rules and
ensure you leave it in a valid state or you can cause getmail to error out.

> But is this something FAQ deals.  On desktop user system, it's non-issue.  If
> it is server, it's a common sense.

Exactly.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Osamu Aoki]

Hi,

On Mon, Aug 07, 2017 at 11:46:03AM +0800, 積丹尼 Dan Jacobson wrote:
> What if my house burned down and I must start from scratch?
> Mention how to continue if those files are lost.
> 
> I assume if one really wanted to get rid of them, a safe way would be
> 0. Disconnect the upstream mail server from the internet so no new mail
> is coming in.
> 1. get all your mail upstream -> downsteam. Now all is deleted from
> upstream.
> 2. Now delete the oldmail files.
> 3. Now reconnect the server.
> 
> > "CC" == Charles Cazabon  writes:
> 
> CC> In short: no, you can never safely trim them.  You can remove them only 
> if you
> CC> never want to retrieve mail from the corresponding POP/IMAP account or 
> folder
> CC> again.  The files are absolutely tiny by any modern standard, so there 
> will
> CC> never, ever be a need to "trim" them.

Is this question on mbox file?  Stop mail daemon etc. (if it is high trafic
maybe stop the upstream one like mentioned if you can), and move mbox file away
and play whatever you want to do.  This is standard procedure for daemon
accessed log file type files.

But is this something FAQ deals.  On desktop user system, it's non-issue.  If
it is server, it's a common sense.

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[積丹尼 Dan Jacobson]

What if my house burned down and I must start from scratch?
Mention how to continue if those files are lost.

I assume if one really wanted to get rid of them, a safe way would be
0. Disconnect the upstream mail server from the internet so no new mail
is coming in.
1. get all your mail upstream -> downsteam. Now all is deleted from
upstream.
2. Now delete the oldmail files.
3. Now reconnect the server.

> "CC" == Charles Cazabon  writes:

CC> In short: no, you can never safely trim them.  You can remove them only if 
you
CC> never want to retrieve mail from the corresponding POP/IMAP account or 
folder
CC> again.  The files are absolutely tiny by any modern standard, so there will
CC> never, ever be a need to "trim" them.

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Charles Cazabon]

Dan Jacobson  wrote:
> X-Debbugs-Cc: charlesc-getmail-b...@pyropus.ca
> Package: getmail4
> Version: 4.53.0-1
> Severity: wishlist
> 
> The only place oldmail files are mentioned is
> file:///usr/share/doc/getmail4/faq.html#faq-about-upgrade
> 
> Please add another FAQ: "Do the oldmail files just grow and grow? When can I
> safely trim/remove them?"

I've added such a FAQ to online docs on my website.  It will be included in
the docs in the next release of getmail.

In short: no, you can never safely trim them.  You can remove them only if you
never want to retrieve mail from the corresponding POP/IMAP account or folder
again.  The files are absolutely tiny by any modern standard, so there will
never, ever be a need to "trim" them.

Charles
-- 
--
Charles Cazabon  
Software, consulting, and services available at http://pyropus.ca/
--

Bug#871228: Add FAQ: do the oldmail files just grow and grow?

[Dan Jacobson]

X-Debbugs-Cc: charlesc-getmail-b...@pyropus.ca
Package: getmail4
Version: 4.53.0-1
Severity: wishlist

The only place oldmail files are mentioned is
file:///usr/share/doc/getmail4/faq.html#faq-about-upgrade

Please add another FAQ: "Do the oldmail files just grow and grow? When can I 
safely trim/remove them?"