Re: strange response to message part fetch command

[Pete Maclean]

At 04:30 AM 8/25/2004, Pawel Salek wrote:
Hi,
I encountered a strange response to message part fetch command. It
looks like a bug in the server to me but I would like to get a second
opinion. Suggestions how to work around this problem are welcome too.
The problem is the server promises to send 23824 octects long literal
but sends 4040+ characters and happily announces the processing has
finished (sanitized log below). Any clues?
Could it be that the server is counting 23824 as the size of the body part 
prior to base64 encoding?  Encoding 23824 octets in base64 would typically 
result in slightly less than 4040 characters but it depends of course on 
the line length used.  It would be a bad bug but, in a sense, an 
understandable one.

Pete Maclean 

RE: shared mailbox permanent flags?

[Pete Maclean]

Larry, Thanks but I think you are missing something.  And I
apologize if I have not been clear.  I am not concerned about how to
basically handle system flags.  I handle them (all except \Recent),
handle them in exactly the way you suggest, and always have.
I did, however, do two things wrongly.  First, I did not handle
\Recent at all because I mistakenly understood that it was not obligatory
to do so.  Second, I set up my code to include in the SELECT/EXAMINE
FLAGS response only selected flags; in practice though all flags (except
appropriately \Recent) happened to be included.  So, effectively,
except for the need to handle \Recent, it was only my understanding that
was clearly broken not my server.  My concern is to be sure that,
given my new understanding, my handling of the "real" system
flags is entirely correct.  Basically it has to be correct and I
just want to satisfy myself that there are no subtle implications that I
have missed.
Pete Maclean
At 05:00 PM 8/19/2004, Larry Osterman wrote:
The simplest way of doing this
is to turn the system flags into bit fields and add a 6 or so bits per
message to your per-session in-memory list of messages (you do have an
in-memory list of messages, right?)  Then as the user does store and
fetch of the flags, you just flip the bits on and off in the per-session
in-memory list.
 
 
Larry Osterman 

From: Pete Maclean
[mailto:[EMAIL PROTECTED]]

Sent: Thursday, August 19, 2004 1:29 PM
To: Larry Osterman; [EMAIL PROTECTED]
Subject: RE: shared mailbox permanent flags?
 
Thanks, Larry.  I too forgot about the specialness of \Recent when I
wrote that message.  My misunderstanding about the other flags is a
different matter though.  For the entire four or so years that I
have been working with IMAP up until yesterday I was confidently
convinced that a server was not obliged to handle system flags that it
could not manage as PERMANENTFLAGS.  I was quite distressed to
realize my error.  Fortunately, in spite of this, my server does
handle such flags correctly (well, I hope correctly -- see below) with
the exception of \Recent.  And that I shall fix.
I am now struggling to make sure that I properly understand all the
implications of system-flag handling so that I can confirm that my code
is indeed correct.  For a start, a server must list all the system
flags (except \Recent) in a FLAGS response to every successful SELECT or
EXAMINE (even though that seems redundant).
RFC 3510 section 7.1 states that, "If the client attempts to STORE a
flag that is not in the PERMANENTFLAGS list, the server will either
ignore the change or store the state change for the remainder of the
current session only."  Now, I suppose that ignoring such a
change could be deemed a reasonable strategy because one could argue that
it is effectively equivalent to storing the change only to have another
party immediately do a STORE that reverses it.  However, if a server
adopts that strategy, it is equivalent to its not handling the flag at
all (except to the extent that it does not consider a STORE for the flag
to be an error).  So it must advertise that it supports the flag
even though it really does not.
Now, what if a server stores the state change for the remainder of the
current session only.  Does that mean it stores it *for* the current
session only?  I presume so because anything else could be a
nightmare.  This suggests that, if the same client creates a second
session and selects the same mailbox, it could see different flag states
from the first.  Granted, opening such a second session is probably
an ill-advised venture to begin with (although I have seen it happen) so
this inconsistency is hardly a big deal -- never mind that it is
something that a properly functioning client should be prepared
for.
By contrast to all this, my reading of the RFC is that, when the \Seen
flag is set implicitly (by virtue of an appropriate FETCH), it must be
indeed be set; there is no option.  Now, a server could choose not
to set it and hold forth the same argument as above (that the flag could
be instantly reset by a third party) but that argument breaks down
because, if the flag is effective only for the session involved, then in
practice nothing external could sensibly change it.  This all seems
a bit of a mess.
What I have said may sound somewhat critical and that may be because I am
rather peeved at my own misunderstanding.  My prime intention in
writing this message though is solely to make sure that I now do
understand this well.
Pete Maclean
At 12:01 PM 8/19/2004, Larry Osterman wrote:
Ah, you're right - I forgot about \Recent.  \Recent is
"special" since it's not a "real" flag.

From: Pete Maclean
[mailto:[EMAIL PROTECTED]]
Sent: Wed 8/18/2004 2:48 PM
To: Larry Osterman; petite_abeille; [EMAIL PROTECTED]
Subject: RE: shared mailbox permanent flags?
I think that either what you say is not quite correct or I am not
understanding it right.  RFC 3501 

RE: shared mailbox permanent flags?

[Pete Maclean]

Thanks, Larry.  I too forgot about the specialness of \Recent when I
wrote that message.  My misunderstanding about the other flags is a
different matter though.  For the entire four or so years that I
have been working with IMAP up until yesterday I was confidently
convinced that a server was not obliged to handle system flags that it
could not manage as PERMANENTFLAGS.  I was quite distressed to
realize my error.  Fortunately, in spite of this, my server does
handle such flags correctly (well, I hope correctly -- see below) with
the exception of \Recent.  And that I shall fix.
I am now struggling to make sure that I properly understand all the
implications of system-flag handling so that I can confirm that my code
is indeed correct.  For a start, a server must list all the system
flags (except \Recent) in a FLAGS response to every successful SELECT or
EXAMINE (even though that seems redundant).
RFC 3510 section 7.1 states that, "If the client attempts to STORE a
flag that is not in the PERMANENTFLAGS list, the server will either
ignore the change or store the state change for the remainder of the
current session only."  Now, I suppose that ignoring such a
change could be deemed a reasonable strategy because one could argue that
it is effectively equivalent to storing the change only to have another
party immediately do a STORE that reverses it.  However, if a server
adopts that strategy, it is equivalent to its not handling the flag at
all (except to the extent that it does not consider a STORE for the flag
to be an error).  So it must advertise that it supports the flag
even though it really does not.
Now, what if a server stores the state change for the remainder of the
current session only.  Does that mean it stores it *for* the current
session only?  I presume so because anything else could be a
nightmare.  This suggests that, if the same client creates a second
session and selects the same mailbox, it could see different flag states
from the first.  Granted, opening such a second session is probably
an ill-advised venture to begin with (although I have seen it happen) so
this inconsistency is hardly a big deal -- never mind that it is
something that a properly functioning client should be prepared
for.
By contrast to all this, my reading of the RFC is that, when the \Seen
flag is set implicitly (by virtue of an appropriate FETCH), it must be
indeed be set; there is no option.  Now, a server could choose not
to set it and hold forth the same argument as above (that the flag could
be instantly reset by a third party) but that argument breaks down
because, if the flag is effective only for the session involved, then in
practice nothing external could sensibly change it.  This all seems
a bit of a mess.
What I have said may sound somewhat critical and that may be because I am
rather peeved at my own misunderstanding.  My prime intention in
writing this message though is solely to make sure that I now do
understand this well.
Pete Maclean
At 12:01 PM 8/19/2004, Larry Osterman wrote:
Ah,
you're right - I forgot about \Recent.  \Recent is
"special" since it's not a "real" flag.


From: Pete Maclean
[mailto:[EMAIL PROTECTED]]
Sent: Wed 8/18/2004 2:48 PM
To: Larry Osterman; petite_abeille; [EMAIL PROTECTED]
Subject: RE: shared mailbox permanent flags?

I think that either what you say is not quite correct or I
am not
understanding it right.  RFC 3501 gives an example of a FLAGS
response that
does not include all the system/built-in flags:
Example:    S: * FLAGS (\Answered \Flagged \Deleted \Seen
\Draft)
In this case, \Recent is missing.  As I read the spec, if you do
not
include a particular system flag in this response, it means that you do
not
support it for the mailbox.  Incidentally, some versions of my
server do
not support \Recent at all.
Pete Maclean
At 05:09 PM 8/18/2004, Larry Osterman wrote:
>You MUST support ALL the built-in flags on every mailbox.  You
might not
>be able to persist those flag settings, but you MUST support them on
a
>per-session basis.
>
>-Original Message-
>From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
>On Behalf Of petite_abeille
>Sent: Wednesday, August 18, 2004 2:00 PM
>To: [EMAIL PROTECTED]
>Subject: Re: shared mailbox permanent flags?
>
>Hi Timo,
>
>On Aug 18, 2004, at 22:37, Timo Sirainen wrote:
>
> > FLAGS list shouldn't be empty. It should contain at least the
system
> > flags.
>
>Ok. So FLAGS should always returns \Answered \Deleted \Draft
\Flagged
>\Recent and \Seen? Even if they are not applicable for such 
mailbox?
>
> >> But this doesn't seem to help :/
> >
> > Yep. I doubt there's a way to make it work many (if any)
clients.
>
>Sigh :/
>
>PA.

RE: shared mailbox permanent flags?

[Pete Maclean]

Argh!  I have always had a serious misunderstanding about this then.  It 
seemed to me that the very fact that there is a response explicitly listing 
FLAGS applicable to the mailbox meant that a server was not required to 
support all of them.  Well, I stand corrected.  Thank you for the 
clarification, Mark.

Pete Maclean
At 05:54 PM 8/18/2004, Mark Crispin wrote:
On Wed, 18 Aug 2004, Pete Maclean wrote:
I think that either what you say is not quite correct or I am not 
understanding it right.  RFC 3501 gives an example of a FLAGS response 
that does not include all the system/built-in flags:
Example:S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
\Recent is not included in the untagged FLAG response because it can not 
be used with STORE.  It is, however, mandatory to implement.

Note that difference between the "flag" and "flag-fetch" rules in the 
Formal Syntax.

In this case, \Recent is missing.  As I read the spec, if you do not 
include a particular system flag in this response, it means that you do 
not support it for the mailbox.
Nope.  Refer to section 7.2.6:
  [...] The flag parenthesized list identifies the flags (at a
  minimum, the system-defined flags) that are applicable for this
  mailbox.
Incidentally, some versions of my server do not support \Recent at all.
A server may choose never to return a lit \Recent flag, but it must 
otherwise implement it (e.g. the RECENT argument to SEARCH and STATUS).

-- Mark --
http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

RE: shared mailbox permanent flags?

[Pete Maclean]

I think that either what you say is not quite correct or I am not 
understanding it right.  RFC 3501 gives an example of a FLAGS response that 
does not include all the system/built-in flags:

Example:S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
In this case, \Recent is missing.  As I read the spec, if you do not 
include a particular system flag in this response, it means that you do not 
support it for the mailbox.  Incidentally, some versions of my server do 
not support \Recent at all.

Pete Maclean
At 05:09 PM 8/18/2004, Larry Osterman wrote:
You MUST support ALL the built-in flags on every mailbox.  You might not
be able to persist those flag settings, but you MUST support them on a
per-session basis.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of petite_abeille
Sent: Wednesday, August 18, 2004 2:00 PM
To: [EMAIL PROTECTED]
Subject: Re: shared mailbox permanent flags?
Hi Timo,
On Aug 18, 2004, at 22:37, Timo Sirainen wrote:
> FLAGS list shouldn't be empty. It should contain at least the system
> flags.
Ok. So FLAGS should always returns \Answered \Deleted \Draft \Flagged
\Recent and \Seen? Even if they are not applicable for such  mailbox?
>> But this doesn't seem to help :/
>
> Yep. I doubt there's a way to make it work many (if any) clients.
Sigh :/
PA.

Re: FLAGS vs PERMANENTFLAGS

[Pete Maclean]

Stuart,
The SELECT response that you quote strikes me as unusual, perhaps even a 
bit anomalous, but not unreasonable.  The user might, for example, have 
permissions that, in principle, give him write access to the mailbox but 
have specific prohibitions for storing each message flag.  (That's a 
fanciful explanation -- I have never come across a server with per-flag 
permissions -- but a possible one.)

By my reading of the RFC, I must agree with you that, if a server stores a 
flag in a persistent manner that it has not advertised in a SELECT or 
EXAMINE response as a PERMANENTFLAG, then it is in violation.  If you can 
confirm that the server involved in this particular case is doing that 
then, I believe, you should complain strongly.

I have experience with transitorily connected wireless clients.  However, 
while that mode of operation is very likely to influence such 
considerations as what and how much data is downloaded and in what size 
pieces, I cannot see why it should affect the handling of flags in any way 
beyond that.  And it is not at all clear what you mean by "honouring" 
PERMANENTFLAGS.

Pete Maclean
At 02:56 PM 8/6/2004, Stuart Nicholson wrote:
Nicely put and I see your point. I'm dealing with a wireless client that
isn't using the normal IMAP 'connected' model. Our sessions typically last
less than 5 minutes which has obviously influenced our design. However I
would expect folders that can't be modified to have the READ-ONLY property
surely? We see things like this (taken from a user log):
0003 SELECT "INBOX"[0x0D][0x0A]
 * 40 EXISTS
 * 0 RECENT
 * OK [UIDVALIDITY 1012731894] UID validity status
 * OK [UIDNEXT 7880] Predicted next UID
 * FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
 * OK [PERMANENTFLAGS ()] Permanent flags
 * OK [UNSEEN 8] first unseen message in INBOX
0003 OK [READ-WRITE] SELECT completed
And the user complains that other clients allow FLAGS to be set and they are
set permanently (over sessions) which seems to violate the RFC, at least as
I understood it.
I'm beginning to think given our short session times and rigid session model
(i.e. we always sync the message flags at start of session), there's really
little point in us honouring PERMANENTFLAGS at all.
Thanks,
Stuart Nicholson
www.snappermail.com
- Original Message -
From: "Larry Osterman" <[EMAIL PROTECTED]>
To: "Stuart Nicholson" <[EMAIL PROTECTED]>;
<[EMAIL PROTECTED]>
Sent: Thursday, August 05, 2004 4:32 PM
Subject: RE: FLAGS vs PERMANENTFLAGS
That behavior (empty permanentflags but flags) is explicitly allowed.
And if you read the spec VERY carefully you'll find it's REQUIRED if the
user can't modify the folder.  The server maintains flags, but they
aren't persisted.
Think of it this way: FLAGS is the set of flags supported by the server.
PERMENANTFLAGS is the set of flags that will be persisted in the
server's database.  FLAGS is necessarily a superset of PERMENANTFLAGS,
but they are different and have different meanings.

Larry Osterman
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Stuart Nicholson
Sent: Friday, August 06, 2004 11:16 AM
To: [EMAIL PROTECTED]
Subject: FLAGS vs PERMANENTFLAGS
Greetings, my first post here so an brief intro: I'm the developer
building
IMAP client support for SnapperFish. We're responsible for 'SnapperMail'
(www.snappermail.com) a wireless email client for PalmOS based devices.
Question: How well honoured are FLAGS vs PERMANENTFLAGS in the IMAP
community? Is PERMANENTFLAGS overdesign that actually isn't used in the
wild
that often? Do other clients actually only honour FLAGS and disregard
PERMANENTFLAGS?
Is ask beause our IMAP client is currently coded closely to RFC3501 in
this
respect however we've received a number of reports from users of servers
that persistently return empty (i.e. "PERMANENTFLAGS ()") responses
while
still allowing the flags in FLAGS to be permanently set on messages in
the
mailbox. This seems in direct violation to the RFC however other IMAP
clients happily set flags on these particular servers.
Is it worth just relaxing our client on this point hence disregarding
PERMANENTFLAGS and simply relying on FLAGS and the READ-WRITE/READ-ONLY
folder property?
Thanks,
Stuart Nicholson
www.snappermail.com
--
-
 For information about this mailing list, and its archives, see:
 http://www.washington.edu/imap/imap-list.html
-

--
-
 For information about this mailing list, and its archives, see:
 http://www.washington.edu/imap/imap-list.html
-
--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: What is wrong with this server?

[Pete Maclean]

At 02:45 PM 7/12/2004, Sebastian Hagedorn wrote:
-- Pete Maclean <[EMAIL PROTECTED]> is rumored to have mumbled on 
Montag, 12. Juli 2004 12:38 Uhr -0400 regarding What is wrong with this 
server?:

That, in my view, is not the real problem though.  As a server developer
myself, I am inclined to be extremely forgiving of server bugs.  The
problem, as I see it, is that when a colleague of mine reported the
problem to the vendor, Merak, the response that came back refused to
acknowledge any error but apparently attempted to justify the behavior
citing RFC 1370 ("Applicability Statement for OSPF"), a document that has
nothing whatsoever to do with IMAP.
Maybe they meant RFC 1730?
Ah, yes, I bet they did.  Thank you.
Pete

What is wrong with this server?

[Pete Maclean]

* OK Merak 7.4.5 IMAP4rev1 Thu, 08 Jul 2004 11:24:27 +0300
a login test1 123456
a OK LOGIN Completed
b select inbox
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft \Recent)
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 0]
* OK [UIDVALIDITY 818058273] UIDs valid
* OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Draft \Deleted)]
b OK [READ-WRITE] SELECT Completed
U100 uid fetch 383 (UID BODY.PEEK[1]<0.20> RFC822.HEADER)
* 1 FETCH (UID 383 BODY[1]<0> {20}
=20
=20
Liran Poll RFC822.HEADER<0> {20}
Received: from fiat.)
U100 OK UID FETCH Completed
Make a quick perusal of this fragment of an IMAP session and the server's 
misbehavior should be very obvious.  The protocol just does not allow for 
the partial fetch of an RFC822.HEADER.

That, in my view, is not the real problem though.  As a server developer 
myself, I am inclined to be extremely forgiving of server bugs.  The 
problem, as I see it, is that when a colleague of mine reported the problem 
to the vendor, Merak, the response that came back refused to acknowledge 
any error but apparently attempted to justify the behavior citing RFC 1370 
("Applicability Statement for OSPF"), a document that has nothing 
whatsoever to do with IMAP.

Now, this is not Courier we are talking about.  What sort of reputation 
does Merak have?  Does anyone connected with Merak read this mailing list?

Pete Maclean
--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: Understanding Reference Names

[Pete Maclean]

Mark,
Your discussion is very enlightening, thank you.  I have forwarded your 
message to someone who, I hope, will pass it on to the Thunderbird developers.

Pete Maclean

Re: Understanding Reference Names

[Pete Maclean]

>>As an aside, I disapprove of the practices of mailboxes under INBOX and 
Trash mailboxes.
>>IMAP does not prohibit either of these practices, but these practices 
are associated with
>>substantial client and user confusion.

Mark, would you expand on this a little, please?  I am currently looking at 
a situation with Thunderbird.  When one asks it to delete an IMAP mailbox, 
it tries to do so by moving it to the Trash.  (I.e. using RENAME mailbox 
Trash/mailbox).  This is a problem because some servers do not (and cannot) 
support a rename that has the effect of moving the mailbox within the 
hierarchy.

Pete Maclean

Re: UIDVALIDITY response optional?

[Pete Maclean]

At 07:38 PM 5/27/2004, Mark Crispin wrote:
On Thu, 27 May 2004, Pete Maclean wrote:
One element seems wrong but I am not 100% certain.  This server (which I 
cannot identify since it has not been identified to me) claims IMAP4Rev1 
compliance by virtue of its initial response (* OK IMAP4rev1 Service Ready).
IMAP4rev1 compliance is indicated by "IMAP4REV1" appearing in the 
CAPABILITY list.
I stand corrected.  Thank you!
But when sent a SELECT command that succeeds, it does not return a 
UIDVALIDITY response.
Unless that server is IMAP2, it is broken.
There is no CAPABILITY response in the session so I will have to find out 
what the server thinks it is.  (It is clear from other things that it's 
seriously broken no matter what version of IMAP it is pretending to operate.)

Now RFC 3501 states that, "If [the UIDVALIDITY response] is missing, the 
server does not support unique identifiers."
Please read a few paragraphs higher in RFC 3501:
  Note that earlier versions of this protocol only required the
  FLAGS, EXISTS, and RECENT untagged data; consequently, client
  implementations SHOULD implement default behavior for missing data
  as discussed with the individual item.
In other words, if UIDVALIDITY is missing, the server is an IMAP2 server.
May I suggest that, for implementers who have no knowledge of IMAP2 and no 
requirement to provide backward compatibility, this could be made clearer.

Pete Maclean

UIDVALIDITY response optional?

[Pete Maclean]

I have been looking at the transcript of an IMAP testing session that shows 
some very strange behavior on the part of the server concerned.  Some of 
its behavior is unequivocally wrong.  One element seems wrong but I am not 
100% certain.  This server (which I cannot identify since it has not been 
identified to me) claims IMAP4Rev1 compliance by virtue of its initial 
response (* OK IMAP4rev1 Service Ready).  But when sent a SELECT command 
that succeeds, it does not return a UIDVALIDITY response.

Now RFC 3501 states that, "If [the UIDVALIDITY response] is missing, the 
server does not support unique identifiers."  After trying to reconcile 
this with the rest of the document I think it must be inaccurate, perhaps a 
vestige of some historical variance.  Elsewhere the spec leaves no 
ambiguity:  UIDs and UIDVALIDITYs are absolutely required.  And, since 
there is no way to get a UIDVALIDITY other than in a UIDVALIDITY response 
to a SELECT or EXAMINE command, that response must be required.  Is my 
analysis correct?

Pete Maclean
--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: Off-topic request

[Pete Maclean]

Rick,

Thanks for your response.

>>I'm not sure you're going to get sensible answers to your questions
>>without providing a lot more details about what you're talking about.
I realize that I am not making it easy for people.  Unfortunately, I am
constrained by being under heavy non-disclosure regarding the work
I am doing.  It is difficult both because there are things I cannot divulge
and because, in some areas, it is hard to know just how much I can say
and how much I should not.
>>Guessing that you're talking about a client development SDK of some
>>kind
I guess I can say that what I am working on could be described as an
email gateway and the issues I am concerned about involve exactly
that, a client-development SDK.
>>I think the questions that need to be answered are what
>>programming language and what OS(s) are you targeting?
The programming language is C.  No particular OS but, for the most
part, not traditional desktop platforms.  (The gateway does run on one
specific OS but that has limited relevance to clients.) The Unicode
interface needs to be UTF-8; that much is beyond my decision or control.
Pete

Re: Off-topic request

[Pete Maclean]

Thank you all for your responses to my request.  I expected a few pointers 
and instead I received something much better.  I am in awe of how much some 
of you know about these matters.

Now, since you are being so helpful, I want to narrow down the issue of 
charset support in a couple of ways and ask some further questions that are 
more practical and specific to the environment I am working in.

It is very much our interest and intention to promote the use of Unicode in 
applications generally and email clients in particular.  Now, Mark Keasling 
wrote, "If SHIFT_JIS, EUC-JP and ISO-2022-JP aren't supported you'd have a 
difficult time entering the Japanese market."  But to what extent is such 
support going to be expected (or demanded)?  Thanks to our use of c-client, 
we have no problem handling incoming messages with text in these 
charsets.  But would someone developing  a Japanese-market email client be 
content to work internally using only Unicode or would he want to actually 
handle the data in those other charsets?  We have two apparent 
options:  (1) converting every text part to UTF-8 while keeping a record of 
the original charset or (2) giving the developer an option of reading text 
in either UTF-8 or the original charset.  We would prefer (1) but are quite 
ready to provide (2) if necessary.  And is there a (3)?

I assume that for outgoing messages, a parallel situation would apply.  Are 
there in fact any differences there?  We can either (1) require that the 
application supply all outgoing text in UTF-8 and then convert it to any 
requested charset before the message is sent or (2) accept outgoing text 
either in UTF-8 and convert it as requested, or in any other labeled 
charset in which case we treat it as binary data.

Again, many thanks for your generous assistance,

Pete Maclean

Re: Off-topic request

[Pete Maclean]

Mark,

Many, many thanks.  That is a great help.  I had not even been aware of 
ISO-8859-15 -- and I am European.

As it happens we are using c-client for this project so I give you many 
thanks for that as well.  Without it, we would be very hard pressed to meet 
our deadline.  With it, we are almost on schedule.

Pete

At 12:49 PM 3/25/2004 -0800, Mark Crispin wrote:

On Thu, 25 Mar 2004, Pete Maclean wrote:
   - what charsets are commonly used for email?
The ones that I see most commonly are: US-ASCII, UTF-8, ISO-8859-1, 
ISO-8859-2, ISO-8859-15, KOI8-R, ISO-2022-JP, GB2312, BIG5, EUC-KR, 
WINDOWS-1251.  However, others do appear.

   - can most clients handle text received in UTF-8?
Not yet.  However, the number is growing; and the fact that Outlook 
supports it means that it is likely that a majority of email users (as 
opposed to clients) can deal with UTF-8.

The forthcoming release of Pine (Pine 4.60) can not send UTF-8 mail; 
however, it will render UTF-8 mail into the user's local character set.

   - will client developers be happy if they can send text only in 
US_ASCII, ISO-8859-1 or UTF-8?
ISO-8859-1 is dead, or at least the death warrant has been signed.  If you 
want to cater to the 8-bit western European market, support ISO-8859-15, 
not ISO-8859-1.  The Europeans really need the Euro character...

I *think* that a client which does only US-ASCII (not US_ASCII) and UTF-8 
will be viable in the future, but not yet.  On the other hand, if I was 
specifying the development of a new client today, I would go UTF-8 only on 
the grounds that by release time UTF-8 will have hit critical mass.

   - when providing resources for MUA developers, what charset-related
 facilities should be supplied?
The UW c-client library's utf8 section has functions to transfer from lots 
of local character sets into UTF-8.  For some time to come, a client will 
need to recognize messages in other character sets and render them *into* 
UTF-8, if not necessarily generate UTF-8.

Some more modern UNIX systems have the iconv() function which is useful if 
you're not using c-client.

I would not use Pine's approach in a new client.  Pine was designed to use 
local character sets, and as a result is approaching the UTF-8 transition 
backwards (the hard way).  I know because I've written much of that code. 
We have to do it that way because of our legacy base (and a *lot* of 
code), but that doesn't mean that you should (do what I say, not what I do).

It's much easier to start off using UTF-8 internally for everything, 
convert all other character sets into UTF-8, and possibility have a 
mechanism to generate other character sets from UTF-8.

And generally, what does one need to know when providing client-oriented 
facilities for developers targeting foreign-language/character-set users?
Yeegs.  That's a lot to ask.  Your best option IMHO is to ask people 
questions as they come in -- and be sure to include questions about what 
they think they did *wrong* or could have done *better*.  This is 
something in which a lot of learning can be from the mistakes of others.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Off-topic request

[Pete Maclean]

Please excuse this rather off-topic message.  As this list has given me my 
most valuable contacts with other email developers over the last few years, 
I thought it would be a good forum in which to put out this request.

I have many years of experience working with email protocols but, in some 
ways, I have little sense of how people, especially non-English-speaking 
people, actually use email.  I am currently working on a project 
(IMAP-related) that is very international in scope and I am finding that I 
need to know answers to such questions as:

- what charsets are commonly used for email?
- can most clients handle text received in UTF-8?
- will client developers be happy if they can send text only in 
US_ASCII, ISO-8859-1 or UTF-8?
- when providing resources for MUA developers, what charset-related
  facilities should be supplied?

And generally, what does one need to know when providing client-oriented 
facilities for developers targeting foreign-language/character-set users?

The one resource I have found so far that has proved very helpful is the 
IMC's report "Using International Characters in Internet Mail" 
<http://www.imc.org/mail-i18n.html>.

If anyone can suggest any other good resources (mailing lists, newsgroups, 
Web sites, books, w.h.y.), I would appreciate it very much.

Thanks,

Pete Maclean

--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: Assumption of hierarchy?

[Pete Maclean]

At 06:51 PM 1/6/2004 -0700, Lyndon Nerenberg wrote:

--On 2004-1-6 7:32 PM -0500 Pete Maclean <[EMAIL PROTECTED]> wrote:

While we are on the subject of clarification, I am moved to suggest
one other related area where I think some clarification would be
beneficial.  That is to specify the minimum requirements for a
mailstore to be suitable for use with an IMAP server.
I disagree. That would be the same as saying we need a set of minimum 
requirements for a filesystem that supports SQL.

The IMAP protocol stands alone, and is independent of the underlying 
infrastructure. Anything that violates that layering principal is doomed 
to seeing the rapid end of the clue bat.
Lyndon,  Thank you for responding.  I don't understand much of what you say 
-- I know next to nothing about SQL -- but, given that no one else has said 
anything, I will defer to your position.

Pete

Re: Assumption of hierarchy?

[Pete Maclean]

I think that this discussion indicates that at the very least some
documented clarification is required...
While we are on the subject of clarification, I am moved to suggest one 
other related area where I think some clarification would be 
beneficial.  That is to specify the minimum requirements for a mailstore to 
be suitable for use with an IMAP server.  I have not come across any 
discussion of this matter and, it seems, one must infer the necessaries 
from the details of the protocol.  And even then some issues, notably what 
may and may not be present in the hierarchy, are far from evident.

Some relevant pieces of information are present in RFC 2683 (IMAP4 
Implementation Recommendations) and these could be referenced.

Any thoughts?

Pete Maclean

Re: Multiple command clarification.

[Pete Maclean]

At 05:14 AM 1/5/2004 +0200, Timo Sirainen wrote:
Something I just thought about doing optionally: Mark the message
expunged using some non-standard flag and expunge it later (maybe in
some nightly run). Don't show expunge-marked messages to clients, unless
they haven't been notified that it's been expunged. Don't do the real
expunging unless the flag was set less than hour ago. That should make
it highly unlikely that anyone hasn't been notified of the EXPUNGE yet.

Maybe you could do something similiar.
This is an interesting idea, thank you.  It is not useful to me 
though.  That's because, within the domain of my IMAP server, I could do 
better and the real problem is that of messages being expunged by other 
(non-IMAP) mechanisms (as was mentioned by the Exchange folks).

Pete

Re: Trailing hierarchy delimiter in name

[Pete Maclean]

There was a good discussion related to this on this list a few months ago, 
presumably during the time you were off.  If you search the archives for 
September 2003 I think you will find a lot of relevant information.  There 
is one thread with a subject of LIST and there may be more.

Cheers,

Pete

At 03:23 PM 1/5/2004 +1300, David Harris wrote:

I'm sure that this must be covered somewhere in RFC3501, but I can't
find it, so I'd appreciate some guidance... Consider the following LIST
response from an IMAP server, where the hierarchy delimiter is '/':
   * LIST (\\Noselect) "/" "Public Folders/"

How should the trailing hierarchy delimiter on the mailbox name be
interpreted? Does it have a special meaning of some kind?
I can find the section covering the semantics of this for CREATE
commands, but I can't see anything indicating what this would mean in
a LIST response.
Any assistance would be appreciated.

Cheers!

-- David --

-- David Harris -+- Pegasus Mail --
  Box 5451, Dunedin, New Zealand | e-mail: [EMAIL PROTECTED]
   Phone: +64 3 453-6880 | Fax: +64 3 453-6612
On the menu in a Chinese Restaurant:
   "We hope you will enjoy our hostility."


--
-
 For information about this mailing list, and its archives, see:
 http://www.washington.edu/imap/imap-list.html
-

Re: Multiple command clarification.

[Pete Maclean]

I have been paying avid attention to this thread because it deals with an 
issue that affects my server as well.  Currently my server works much like 
the Cyrus one does with expunged messages -- and I take some comfort in 
finding that I am not alone.

The only alternatives I see, given the particular constraints upon my 
server, are to behave like the UW server and disallow multiple concurrent 
selections or make a copy of every mailbox as it is selected.  However, 
implementing the latter would entail an amount of work and impose an 
efficiency toll of such magnitude that I cannot see undertaking it.  Adding 
the former as an option is something I am considering.

Pete Maclean

Re: What Server answer of SELECT Command?

[Pete Maclean]

Have you considered outsourcing an IMAP server for your software? Several
people, including the Cyrus project and Maclean Sofware, offer suitable
software. 
See
http://www.maclean.com/imap/home.html,
for example. Thanks for that site I'll show it to my boss
but we develope in Delphi and the IMAP Server Engine is written in C.

I think it is not real difficult to write an IMAP Server so I think we
get on going with our solution. The only Problem is
that having only the RFCs and no real other document means a lot of
trial-by-error work. So I'm glad there is such an 
mailing list like this. I have never seen people answering so fast and I
use a lot of discussion board, mailing list etc.
You may consider me prejudiced since I would certainly like you to
license my server engine -- and that is fair enough.  However I will
say that writing an IMAP server is neither an easy nor a quick
undertaking.  I reckon that I have put at least two solid years of
work into mine and I found it a challenging task.  And that is
despite the fact that I have loads of experience writing
servers.
Integrating my server code with Delphi should be no problem.  The
most obvious way would be to build the engine as a DLL.
Pete Maclean

Re: ENVELOPE - syntactic assistance

[Pete Maclean]

Interesting!  I did not see the character in question as a no-break 
space.  My email client must have converted it to a regular space.

Pete

At 04:08 PM 11/17/2003 -0800, Mark Crispin wrote:

Hi David -

You didn't look closely enough at the offending message.  I probably
wouldn't have caught it either if it weren't for the fact that I use a
Japanese environment (and thus I was able to see what it really was on my
display).
The offending character isn't a space (0x20); it is an ISO 8859 no-break
space (0xa0).  As it has the 0x80 bit set, it is not permitted in a quoted
string; only 7-bit characters are permitted in atoms or quoted strings.
That is your IMAP syntax error.
In addition, neither a space nor any 8-bit character can be part of a DNS
name.  If that character was a real space, e.g.
From: Fred Flintstone <[EMAIL PROTECTED] >
then you either have a no-parse condition or parse it according to the old
RFC 822 rules which permitted such spaces; that is,
From: Fred Flintstone <[EMAIL PROTECTED]>
and
From: Fred Flintstone < fred @ bedrock.com >
are equivalent.
The reason why RFC 2822 now prohibits such spaces was that RFC 822 also
permitted such things as:
From: Fred Flintstone < fred @ bedrock (not slate) . (not org) com >
which were widely acknowledged as bogus.
A reasonable modern implementation is to disregard any spaces around the
<> pair or the @ in the name of compatibility with the past.  Nobody
rationally argued that it worked to imbed spaces in DNS names.
Of course, since the problem in that message is a bogus 8-bit character,
you have a no-parse condition.  How you handle it is up to you, but you
can't put it into a quoted string in an envelope address and expect it to
work.  You could get around IMAP's syntax rule by using a literal, but
then you'd violate the rule about no 8-bit data without a charset tag
which essentially precludes 8-bit in ENVELOPE and BODYSTRUCTURE.
-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: ENVELOPE - syntactic assistance

[Pete Maclean]

Hi David,

I bet that the problem is the trailing space in "runbox.com ".  Of course, 
it reflects the same in the envelope <[EMAIL PROTECTED] >.  I am not 
certain if the latter is illegal but it would be wise to trim it.

Pete Maclean

At 12:35 PM 11/18/2003 +1300, David Harris wrote:

I have a user reporting that neither Outlook nor Mulberry can retrieve a
particular message from my IMAP server - when they try to retrieve the
envelope of the message, they both behave as though there is a
syntactic (rather than semantic) error in the response and abort the
transaction.
Try as I might, I can't see what the syntactic issues is, and I'd be
grateful for some assistance.
First, here are the RFC822 headers for the offending message:

-- Cut here 
X-Envelope-To: <[EMAIL PROTECTED]>
Received: from HOME (4.40.96.168) by x.com (Mercury/32 v3.32) ID MG000889;
   18 Jul 03 23:37:30 -0700
Received: from hu.7jt0f.com [238.78.176.173] by HOME with ESMTP id 
58693545; Sat, 19 Jul 2003 05:33:18 -0200
Message-ID: <[EMAIL PROTECTED]>
From: "Chris Barron" <[EMAIL PROTECTED] >
Reply-To: "Chris Barron" <[EMAIL PROTECTED] >
To: [EMAIL PROTECTED]
Subject: How much tax have you paid in your life? Want to stop?
Date: Sat, 19 Jul 03 05:33:18 GMT
X-Mailer: Microsoft Outlook Express 5.00.2615.200
MIME-Version: 1.0
Content-Type: multipart/related;
type="multipart/alternative";
boundary="7__F727_5AB_2B_5"
X-Priority: 3
X-MSMail-Priority: Normal
-- Cut here 

And now, here is the envelope response generated by my server for this
message (I have wrapped it for convenience of viewing, but have
checked rigorously that all the space characters are correct in the
original):
-- Cut here 
* 9 FETCH (ENVELOPE ("Sat, 19 Jul 03 05:33:18 GMT" "How much tax
have you paid in your life? Want to stop?" (("Chris Barron" NIL
"toughguyer" "runbox.com ")) (("Chris Barron" NIL "toughguyer"
"runbox.com ")) (("Chris Barron" NIL "toughguyer" "runbox.com ")) ((NIL
NIL "xx" "x.com")) NIL NIL NIL "<4e1287$vi6$$327k-9-
[EMAIL PROTECTED]>"))
a9 OK FETCH complete.
-- Cut here 
Now, I can see some *semantic* problems with this - the trailing space
on the domain portion of the address, for instance; but semantic errors
should not result in the transaction aborting. I'm at a loss for what
*syntactic* error there might be here: could some kind soul please point
it out to me so I can correct it?
Cheers!

-- David --

-- David Harris -+- Pegasus Mail --
  Box 5451, Dunedin, New Zealand | e-mail: [EMAIL PROTECTED]
   Phone: +64 3 453-6880 | Fax: +64 3 453-6612
Real newspaper headlines from US Papers:
   "Milk drinkers are turning to powder".

Re: LIST

[Pete Maclean]

I have been feeling a bit thick as well, reading this thread and struggling 
to understand the issues involved.  With the last few messages, I think I 
may have finally seen the light.

Consider a store that can contain email messages, appointments, addresses, 
notes, etcetera, with items of each class restricted to separate containers 
but with all such containers in a common hierarchy.  Am I correct in 
thinking that this would be an example of a situation where the foo/ issue 
arises?  If so, then I well appreciate the desirability of a client's being 
able to determine that it is unable to create a mailbox called, say, 
Reminders because there happens to already be a notes-container with that 
same name.  I am not sure that I like the solution though.

But there is still something I seem to be missing.  Is the idea that a 
server should list such non-mail containers in, say, a LIST "" * or only in 
a specific case such as LIST "" Reminders/%?

Pete Maclean 

Re: Issues with the BINARY extension

[Pete Maclean]

FETCH 1:* BINARY[1]

I expect it would be rare for a client to issue a FETCH for a specified
body part for multiple messages but it is certainly possible and I can
imagine odd situations where it would be quite plausible.
If the client already knows that all the messages in the range are encoded
using only base64 or qp (the mandatory-to-implement encodings) then it knows
that the request "cannot" fail due to UNKNOWN-CTE, thus it's perfectly
reasonable for the client to issue that command.
Fair enough.  Of course there's no law that says a client has to know what 
it's doing but let us suppose that it does.  Then a lot depends on exactly 
what one means by "know". I was thinking of the kind of situation where a 
client knows implicitly that all messages filtered into a certain mailbox 
are created by a certain automaton and always formatted in an identical 
way.  (And being the kind of client that uses BINARY, the last thing it 
wants to do is waste transmission cost by verifying the format by fetching 
BODYSTRUCTUREs.)  Then one day someone decides to "improve" the automaton 
by having it use a different encoding method but neglects to inform the client.


Suppose that
the server works through the messages, decoding each appropriate MIME
part and sending it.  Then suppose it hits one message that has the
part encoded using a method that the server does not know about and
cannot decode.  The prescribed action is to send a NO response
containing UNKNOWN-CTE.  The problem lies in dealing with the
requirement that the FETCH (like any IMAP command) totally succeed or
totally fail.  What should a server do?
I would have the server return all the decodeable parts, and return a NO
[UNKNOWN-CTE]. From that the client should be able to figure out what's going
on and act appropriately.
Great!  Thanks.  That's the answer I wanted to hear 


The second issue is what should a server do if it comes across a MIME
part that it is asked to send as BINARY but which it cannot decode
because the part is improperly encoded?  Maybe there should be some
response similar to UNKNOWN-CTE for this case?
I would just return UNKNOWN-CTE, since that's exactly what you're faced with.
Will do.

Pete

Re: Issues with the BINARY extension

[Pete Maclean]

Steve, Lyndon,

Thank you for your responses.  I was puzzled by what Steve wrote and, given 
Lyndon's last message, suspect now that I was not as specific in my 
question as I might have been.

True I asked for guidance on how much data the server should return in the 
circumstances described, but that was not the crux of my question.  What I 
was actually trying to get at is this:  should the server set the \Seen 
flags for messages for which it has returned data or not?

Lyndon, I took your response as indicating that it should.  However, I 
realize now that that was an inference that I was perhaps too hasty in 
making.  Another thing:  on further thought I am surprised that you said 
that you "*would* have the server return all the decodeable parts".  Given 
my understanding that you have implemented this, I am wondering what you 
*did* have the server return.  Of course I realize that for confidentiality 
concerns, or other possible reasons, you may choose not to say.

Pete Maclean

At 11:16 PM 8/12/2003 -0600, you wrote:

Steve nails this on the head. IMAP is a state machine. If yopu're trying to
build context around command/response-type exchanges, you just don't get IMAP.
IMAP is stateless at the protocol level. If you don't get that, you've 
missed the boat and fallen off the dock.

--lyndon

Re: Issues with the BINARY extension

[Pete Maclean]

 What I was actually trying to get at is this:  should the server set 
the \Seen flags for messages for which it has returned data or not?
Yes, it should.
Better question is should it set \Seen flag for messages for which it 
didn't return the BINARY data (but might have returned other things). I 
would guess not, but RFC doesn't mention this. IMHO the whole implicit 
\Seen setting is just evil and it'd been better for BINARY not to support 
it at all.
Argh!  Very good point.

RE: IMAP not good enough?

[Pete Maclean]

IMHO, Steve's exactly right.

Remember that Steve's comparing the proprietary Exchange MAPI RPC
protocol with the IMAP protocol.
The Exchange protocol is orders of magnitude richer than IMAP, but it's
not standard (which is why it's totally proprietary :)).
But is it fair to say that the Exchange protocol is a protocol in the same 
sense as IMAP, or even a true protocol at all?  I am not meaning to suggest 
these things, I am really asking.  The reason is because, it seems to me, 
RPC is really the protocol and what you can do over RPC is so unlimited it 
seems hardly appropriate to call that a protocol.  Am I off base here?

Pete Maclean

Issues with the BINARY extension

[Pete Maclean]

I have been looking at implementing the BINARY extension (as per RFC 3516) 
in my server.  I was tempted because it seemed that it might be a good 
marketing item and because, at first glance, it looked very easy to 
do.  After some deeper thought however I have come to see two issues with 
handling FETCH BINARY commands that concern me.

The first concerns a situation that might never happen but which, it seems, 
a server must be prepared for.  Suppose a client sends a command such as:

	FETCH 1:* BINARY[1]

I expect it would be rare for a client to issue a FETCH for a specified 
body part for multiple messages but it is certainly possible and I can 
imagine odd situations where it would be quite plausible.  Suppose that the 
server works through the messages, decoding each appropriate MIME part and 
sending it.  Then suppose it hits one message that has the part encoded 
using a method that the server does not know about and cannot decode.  The 
prescribed action is to send a NO response containing UNKNOWN-CTE.  The 
problem lies in dealing with the requirement that the FETCH (like any IMAP 
command) totally succeed or totally fail.  What should a server do?  Abort 
the command at this point and send a tagged NO response?  Carry on to the 
end and send a tagged NO response?  Send an untagged NO response?  Avoid 
the contingency by going through all the messages twice, once to verify 
that it can decode each and then once more to send each one?

A related issue is that I feel the need for some clarification on the whole 
business of commands completely succeeding or completely failing.  I have 
come to regard this as an important dictum for IMAP implementation as it 
has been mentioned a few times on this list.  (For example, in a message 
from April 2000, Mark stated "IMAP guarantees that commands succeed 
completely or fail completely.")  However I can find no statement to this 
effect in RFC 3501 and am not sure just what it means to fail or succeed 
completely?  Given that Mark has also made it clear that there is no 
problem with a command failing but still resulting in useful data being 
sent to the client, I have tentatively concluded that the rule pertains 
only to commands that change multiple and non-atomic things on the 
server.  Obvious contenders are APPEND, COPY, DELETE, EXPUNGE and STORE, 
plus instances of FETCH  that have the side effect of setting \Seen 
flags.  This last case makes a FETCH BINARY all the more awkward in that it 
is uniquely allowed to fail in the way discussed above.  I am left tempted 
to think that the only way out of the quandary I described in my first 
paragraph is the last option, that of going through all the messages 
twice.  Maybe that is just what has to be done but it is unappealing for 
two reasons.  First, it could result in a substantial delay before the 
first response data gets sent.  Second, it means implementing the double 
traversal to deal with a situation that might never happen.  This 
difficulty, however, seems in contradiction to a statement made by Lyndon 
Nerenberg in a message from this May: "I implemented Binary in the Cyrus 
server, and it was a very straight-forward modification."  Am I taking the 
completely-succeed-or-completely-fail requirement too seriously?

The second issue is what should a server do if it comes across a MIME part 
that it is asked to send as BINARY but which it cannot decode because the 
part is improperly encoded?  Maybe there should be some response similar to 
UNKNOWN-CTE for this case?

My thanks for any input,

Pete Maclean 

--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: Untagged responses

[Pete Maclean]

Alexey,

>> * 5 EXISTS
>> * 3 EXPUNGE
>> * 4 EXISTS
> The last EXISTS response should not be sent in this case, as it doesn't 
change
> the number of messages.

I question this.  There is no argument that the second EXISTS is redundant, 
no argument that without the EXPUNGE it would be invalid.  However, as I 
understand things, it is not actually wrong as shown.

I note that RFC 3501 says, "The EXPUNGE response also decrements the number 
of messages in the mailbox; it is not necessary to send an EXISTS response 
with the new value."  It is not necessary but it is not disallowed.

I ask this question not to be picky but because my IMAP server will, in 
certain circumstances, send redundant EXISTS responses.  And I have never 
seen any problem because of this.  Now I am wondering if this is something 
one should be careful to avoid.

Pete Maclean

Re: Regarding Deletion of mails...

[Pete Maclean]

I am using Outlook express as my IMAP client. After Deletion of mails from 
inbox, they were marked with delete flag. One copy is saved in Trash 
folder. I tried to delete those mails stored in Trash also, they were also 
marked with deleted flag. So if want to delete completely from server, 
What should i do in Outlook express?
Select "Purge Deleted Messages" from Outlook Express' Edit menu.

RE: Why is a message immutable?

[Pete Maclean]

I want to throw a few ideas (and some strong opinions!) into this discussion:

(1).  There are IMAP servers that, in certain cases, do not assign new UIDs 
to messages that are changed.  This is because the servers have no way of 
telling that a message has been changed.  This makes the servers 
non-compliant and creates a regrettable situation.  But it is a fact of life.

(2).  I happen to be in the habit of editing messages that I receive.  I 
don't do it often but I certainly do it.  The most common case is with 
messages that have generic or non-descriptive subjects.  I may add text to 
the subject field in order to help me identify a message more easily in 
case I need to go back later and look for it.  Another case is when I read 
an encrypted message; I may choose to re-store it in plaintext form.

(3).  Saying that people do not or should not edit received messages 
strikes me as rather irrelevant because whatever we are discussing here 
would presumably apply to all messages and there is one category of message 
that people inevitably do edit, drafts.

(4).  Adding any capability to IMAP to allow messages to be changed in any 
piecemeal fashion would, in my opinion, make a nightmare of the 
protocol.  (Not saying that anyone is suggesting that.)  Providing a way to 
change a message by completely replacing it could be reasonable.

(5).  IMAP could, in principle, be modified to allow a client to be 
notified that a message has been changed by using a per-message 
UIDVALIDITY.  However I see no way to implement such a thing in a way that 
would be compatible for clients that don't support it.  In other words, 
such a thing could not be an extension and therefore would be unacceptable.

(6).  If we allow messages to be changed, how much do we allow them to be 
changed?  Suppose a message is changed beyond all recognition?  Suppose one 
message is essentially replaced by another?  At what point do we insist 
that it be given a new UID -- assuming we must because at some point it 
essentially becomes a new message?  To me, this seems such a morass that it 
is not worth discussing.  My conclusion:  messages SHOULD be immutable and 
that's that.

(7).  The fact that messages are immutable should not be a big deal.  You 
can change a message in any way at all as long as you retire its UID and 
give it a new one.  If you need a link between the original and the changed 
version then that can be done outside of IMAP by using a thread id or 
something similar in the headers.

IMAP cannot be perfect, at least because it has to be used in conditions 
that are imperfect.  But IMAP is awfully good and thinking about changing 
it often serves just to reinforce how good it is.

Pete 

Re: extensions implemented by servers and clients

[Pete Maclean]

>Even better if such list would also contain list of known bugs that the
>client has with it's IMAP implementation.
I have been compiling such a list for a while but have been apprehensive 
about making it public.

Pete

Re: speaking of storing flags

[Pete Maclean]

Would you consider these clients broken?


I deem Outlook Express to be considerably broken.  I have come across many 
problems with it including one of a similar nature:  it insistently tries 
to delete messages, that is do STORE n +FLAGS (\deleted) commands, when the 
server does not advertise \deleted as a handled flag for the active mailbox.

Pete Maclean

RE: Has IMAP been a slow starter?

[Pete Maclean]

Larry,

I appreciate your response, thank you.  You have clarified what I suspected 
and your agreement about mobility is a great boost to me in what I am 
trying to do with IMAP.

So, given the success of the HotMail model, is the next extension we need 
to IMAP an ADVERTISEMENT command?  

Pete

At 09:23 AM 12/10/2002 -0800, Larry Osterman wrote:

IMHO, the biggest reason IMAP hasn't caught on more is simply that ISPs
have no economic incentive to offer IMAP services.  In the corporate
space, there are many incentives to offer IMAP - server-centric email
storage is a GOOD thing - you have centralized backup, the opportunity
for centralized virus scanning, etc.

But in the ISP space (which is 99% of all email users), server-centric
email storage is a BAD thing.

Email on the ISP's server==more hard disk space being used by the
users==more costs for the ISP.

The ISP could go to a "IMAP costs more" model but I don't think that
they've got a good story to justify the extra costs for the customer.
The only currently existing business model for server-hosted email is
the hotmail model - the customers email is kept on the server, primarily
to allow the server the opportunity to plaster ads in the customers
face.

So I suspect that IMAP will continue to grow in the corporate
environment, but given the current technology, in the end-user space,
it's probably a non-starter.

On the other hand, I do agree with you w.r.t. mobility.  If there's
anything that's going to break IMAP loose in the ISP space, it's mobile
devices - when you're dealing with small devices, then server-centric
email storage is a requirement, and I suspect that small devices may be
the thing that makes IMAP reach critical mass.



Larry Osterman



-Original Message-
From: Pete Maclean [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, December 10, 2002 8:46 AM
To: [EMAIL PROTECTED]
Subject: Has IMAP been a slow starter?


A customer recently posed the following question to me:

 >> I am curious about something based upon observation.Has IMAP
been a
slow
 >> starter because most people do not understand its capabilities
intuitively??
 >> From what I have seen, people understand at least some of the
sharing and
 >> folder features of MS Exchange and most of the capabilities of POP.
But
 >> these same people do not seem to understand that there is a protocol
 >> difference between the two and that sharing and folder access is
enabled
 >> through this protocol. Has this been your experience as well?? Also,
do
you
 >> think that imap.org does a good job about getting the word out about
IMAP??

I am unable to do a good job of answering this and wonder if anyone out
there on the list would like to chime in with some helpful comments.  I
would like to create a good answer partly because the customer concerned

has been extraordinarily helpful to me in marketing my IMAP products and

partly because I would like to gain a better grasp of these matters
myself.  I consider my technical knowledge of IMAP to be excellent but
my
cultural knowledge of it is hazy.

I will offer a few ideas that I would include in a response and I invite

people to critique them.

Has IMAP been a slow starter?  I think it has although it is also true
that
it is well used in certain quarters.

Is it hard to understand its capabilities intuitively?  I think this is
true.  For example, there are excellent reasons why IMAP includes no
facilities for sending messages but those reasons are rather technical
and
may seem strange to an average email user.  I think it may also be
difficult to get a grasp of what IMAP can do because of the way that
clients implement it.  Not that clients should expose IMAP capabilities
as
such but many clients make poor use of them.

Why has IMAP not caught on more?  Many email service providers have no
incentive to provide IMAP support.  Or is that the wrong way to look at
it?  Email providers could perhaps do better by offering IMAP as an
extra,
at a price.  But maybe there would be too few takers?  I feel strongly
that
another reason IMAP has not caught on is because of the sorry state of
clients.  While there are a couple of glowing exceptions, the state of
IMAP
support in general among popular clients is a disgrace.

Should IMAP catch on more?  I think it should and I think it will.  The
biggest reason that I see is mobility.  (My view that this is the
biggest
reason may be created by the market that I operate in.)  When you want
to
read your mail from a handheld device (be it a PDA, mobile phone, or
something else), IMAP is ideal.  You can arrange for your mail to be
stored
in one place and access any piece of it from your device.  You can
selectively access the various components of a message.  You can search
folders -- with awesome search power.  The trouble is that there is no
email client I know of for handheld devices that has anythi

Has IMAP been a slow starter?

[Pete Maclean]

A customer recently posed the following question to me:

>> I am curious about something based upon observation.Has IMAP been a 
slow
>> starter because most people do not understand its capabilities 
intuitively??
>> From what I have seen, people understand at least some of the sharing and
>> folder features of MS Exchange and most of the capabilities of POP. But
>> these same people do not seem to understand that there is a protocol
>> difference between the two and that sharing and folder access is enabled
>> through this protocol. Has this been your experience as well?? Also, do 
you
>> think that imap.org does a good job about getting the word out about IMAP??

I am unable to do a good job of answering this and wonder if anyone out 
there on the list would like to chime in with some helpful comments.  I 
would like to create a good answer partly because the customer concerned 
has been extraordinarily helpful to me in marketing my IMAP products and 
partly because I would like to gain a better grasp of these matters 
myself.  I consider my technical knowledge of IMAP to be excellent but my 
cultural knowledge of it is hazy.

I will offer a few ideas that I would include in a response and I invite 
people to critique them.

Has IMAP been a slow starter?  I think it has although it is also true that 
it is well used in certain quarters.

Is it hard to understand its capabilities intuitively?  I think this is 
true.  For example, there are excellent reasons why IMAP includes no 
facilities for sending messages but those reasons are rather technical and 
may seem strange to an average email user.  I think it may also be 
difficult to get a grasp of what IMAP can do because of the way that 
clients implement it.  Not that clients should expose IMAP capabilities as 
such but many clients make poor use of them.

Why has IMAP not caught on more?  Many email service providers have no 
incentive to provide IMAP support.  Or is that the wrong way to look at 
it?  Email providers could perhaps do better by offering IMAP as an extra, 
at a price.  But maybe there would be too few takers?  I feel strongly that 
another reason IMAP has not caught on is because of the sorry state of 
clients.  While there are a couple of glowing exceptions, the state of IMAP 
support in general among popular clients is a disgrace.

Should IMAP catch on more?  I think it should and I think it will.  The 
biggest reason that I see is mobility.  (My view that this is the biggest 
reason may be created by the market that I operate in.)  When you want to 
read your mail from a handheld device (be it a PDA, mobile phone, or 
something else), IMAP is ideal.  You can arrange for your mail to be stored 
in one place and access any piece of it from your device.  You can 
selectively access the various components of a message.  You can search 
folders -- with awesome search power.  The trouble is that there is no 
email client I know of for handheld devices that has anything like a 
complete implementation of IMAP, never mind one that works very 
well.  (Although I have a wee notion that this situation could change for 
one particular PDA, the Sharp Zaurus.)

Do I think that imap.org does a good job about getting the word out about 
IMAP?  I have little idea.  I am not even sure that that is a purpose 
imap.org is intended for.

My thanks in advance for any comments,

Pete Maclean

--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

re: Tags

[Pete Maclean]

The client concerned is Microsoft's Inbox for Pocket PC.

At 08:34 AM 12/6/2002 -0800, Mark Crispin wrote:


On Fri, 06 Dec 2002 11:15:07 -0500, Pete Maclean wrote:
> If my analysis is correct, the non-obvious reason is that the client gets
> unhappy if it fails to obtain certain tagged responses in single read/recv
> chunks.  Which effectively means that it expects said responses to be
> delivered in single packets.  Now it clearly seems a good idea for
> efficiency that a server try to arrange things so that a tagged response is
> sent as one packet.  But surely it cannot be right for a client to expect
> that?

It is certainly not right.  A client should read a line from the server, where
"line" is defined as "arbitrarily long sequence of octets ending with CR LF",
unless it is reading a literal, in which case it is reading a precise number
of octets without regard to CR LF.

A line can certainly be much longer than a packet.

What client has the problem?

Tags

[Pete Maclean]

I have just completed analysis of an incompatibility between a particular 
IMAP client and my server.  The client would occasionally fail with a 
generic error message for no obvious reason.

If my analysis is correct, the non-obvious reason is that the client gets 
unhappy if it fails to obtain certain tagged responses in single read/recv 
chunks.  Which effectively means that it expects said responses to be 
delivered in single packets.  Now it clearly seems a good idea for 
efficiency that a server try to arrange things so that a tagged response is 
sent as one packet.  But surely it cannot be right for a client to expect that?

The root of the incompatibility was that the server would issue a 
write/send for the tag and then another for the rest of the 
response.  Combining them seems to have totally solved the problem.  The 
reason I issued the response in two pieces stems from the fact that there 
is no length limit on a tag.  It seems unlikely that a client would ever 
want to use a tag longer than a few characters (I don't recall seeing more 
than 6) but I am careful to allow for the possibility.  Is there any 
positive reason for not placing a maximum length limit on tags?  Have other 
server developers dealt with this issue?

I appreciate any comments,

Pete Maclean

--
-
For information about this mailing list, and its archives, see: 
http://www.washington.edu/imap/imap-list.html
-

Re: UIDNEXT / UIDVALIDITY for unseen mailbox

[Pete Maclean]

Andy,

This is an important question. It has led me to realize that the same issue 
exists in one edition of my server.  I just confirmed this by creating a 
new mailbox outside of IMAP and doing a STATUS on it which went as follows:

2 status new (messages uidnext uidvalidity)
* STATUS "new" (messages 3 uidnext 0 uidvalidity 0)
2 OK STATUS completed

Pete Maclean

re: to logout or not...

[Pete Maclean]

Mark, thank you for setting us straight on this.  For me, another question 
arises:  when a server detects a connection break, should it process any 
IMAP commands that it has pending for the session, or should it discard 
them?  Seems to me that a good strategy would be to do so 
selectively.  Obviously you don't want to process a SEARCH but it would be 
helpful to perform a CLOSE or LOGOUT.


At 07:44 AM 10/9/2002 -0700, Mark Crispin wrote:

>If you do not send a LOGOUT and just close the connection:
>
>Depending upon how the operating system fields the event to the server, the
>server will see it as a "Hangup" (SIGINT), "Terminated" (SIGTERM), or as "End
>of file on stdin".  In my experience, the last is the most common.
>
>If the server logs these events distinctly, the system manager of the server
>system is going to see that this happened instead of a logout, and think that
>something is wrong.
>
>It is a VERY frequent question "My server's syslog shows many `Command stream
>end of file' messages.  How do I fix this?"
>
>There are worse consequences.
>
>In some mailbox formats (most notably traditional UNIX format), flag settings
>do not take immediate effect in the file, because it's an expensive operation
>to update the file.  Therefore, the new status is just noted in the server,
>and the updates are cached for a later "checkpoint" or "expunge" operation.
>LOGOUT does a checkpoint, as does SELECT/EXAMINE to a new mailbox.
>
>Now, of course the server will try to save the flag updates if the client
>rudely hangs up on it.  But it can't always do so.  The server or the mailbox
>may be in a state that precludes it.  There are also cases in which an
>[x]inetd server gets a SIGKILL instead of a SIGHUP.
>
>The result is that the flag changes are lost.
>
>The only excuse for a client not to send a proper LOGOUT is if the client
>crashes.  The LOGOUT command is in the protocol for a reason, and that reason
>was not cosmetic.
>
>You should send a LOGOUT command, and wait for the response.  If LOGOUT takes
>a while, then it is doing something, such as saving user updates.  You really
>don't want to cause the server to get a SIGKILL while it is doing such.

Re: to logout or not...

[Pete Maclean]

> > I find this a tough question to answer.  My inclination is just to close
> > the connection but I cannot offer much justification for that.  One
> > consideration is that, if you send a LOGOUT and then immediately close the
> > connection, it seems likely that the server will never see the LOGOUT.
>
>it will see it because it is TCP (reliable) layer.

Forgive me, I should have been more specific.  The server will see the 
LOGOUT but it may not process it.  (Is it guaranteed that the server will 
see it?  I am hazy about how TCP drains connections.)

> > Seems to me that a more critical question in the circumstance is 
> whether or
> > not to send a CLOSE command if the session is in the selected state (so
> > that expunging is performed).  But since you don't raise that question
> > maybe you already dealt with it.
>
>expunging is done on EXPUNGE.

And on CLOSE.

>CLOSE has to be sent before any other new SELECT.
>
>but generally, IMAP servers accept SELECT without CLOSE and do the
>required operations implicitely.

I don't see what this has to do with the matter at hand.

Pete

Re: to logout or not...

[Pete Maclean]

 >>if I (as an IMAP client) have to exit quickly and have an open IMAP
 >>connection, I can't wait around for the IMAP server's responses.

 >>In this case, should I send LOGOUT and immediately close the TCP
 >>connection, or should I just close the TCP connection?  I can't quite make
 >>up my mind.

I find this a tough question to answer.  My inclination is just to close 
the connection but I cannot offer much justification for that.  One 
consideration is that, if you send a LOGOUT and then immediately close the 
connection, it seems likely that the server will never see the LOGOUT.

Seems to me that a more critical question in the circumstance is whether or 
not to send a CLOSE command if the session is in the selected state (so 
that expunging is performed).  But since you don't raise that question 
maybe you already dealt with it.

Pete

Re: possible draft 19 changes for sequence

[Pete Maclean]

>Thus, a UID range of 559:* always
>indicates at least one message, unless the mailbox is
>empty.

It might be stronger and clearer to state that "any UID range involving * 
indicates at least one message, unless the mailbox is empty."

>; * is the largest number in use.  For message...

A very minor quibble but I would prefer "* represents the largest number in 
use".  I realize that, no matter how you state it, this suggests an anomaly 
in that, if the mailbox is empty, * has no meaning since there is no 
largest number in use.  But this is a triviality -- I do not suggest that 
you need to address it.

Pete Maclean

Re: INTERNALDATE semantics

[Pete Maclean]

>I'm implementing an IMAP interface to a message store where every message 
>has properties create_time and last_modified_time. Which one should be 
>exposed by the IMAP server as message's INTERNALDATE?

In a similar situation I use the creation time.

>Is it possible to synchronize IMAP folders of this type? How should I 
>announce to the client that some message has changed?

Assuming my understanding is correct then, if any component of a message 
exposed by IMAP other than its flags has changed, it must be made to appear 
as a new message in IMAP terms.  Which means that it must be assigned a new 
UID.  How this would be announced to the client depends on the 
context.  For example, if the client has the folder selected at the time 
the change occurs, it would be signaled by expressing that the old UID has 
been expunged and a new message added.  With some mailstores, implementing 
this can be a challenge.

Pete Maclean

Re: C++ IMAP implementations

[Pete Maclean]

We
are working on the Unified messaging system and need to provide IMAP
interface for it.
Are there ready to use IMAP
toolkits that can be used for the such purpose 
?
I have a toolkit (we actually call it an IMAP Server Engine) that, while
not meeting your requirements perfectly, comes so close that it may well
be suitable.
 Requirements
are as follows:
C++
Our engine is in pure, plain C.
full
source code
Full source code is supplied and selected portions may be viewed prior to
licensing.
portable
Parts of the package, notably our RFC822/MIME parsing library which is
incorporated, are fully portable across Windows, DOS, UNIX, OS/2 and
VMS.  We developed the IMAP code proper initially for Windows and
have been waiting for demand before porting it to other platforms. 
We did cut a few corners and include some Windows-specific code but it
will require little effort to render it fully portable as well.
simple
easy to
use
The engine supports a flexible API for connecting it to mailstores. 
So far it has been adapted to work with 5 different stores, with each
adaption taking anything from a few days to a month.  We can supply
you with documentation on this API.  We also supply test programs
for debugging the mailstore interface.
support
most of the IMAP features
It supports IMAP4rev1 in full plus CRAM-MD5 authentication, SSL/TLS
(using the OpenSSL library) and the UIDPLUS IMAP extension.
For further information please see our Web page on the product
(http://www.maclean.com/imap/)
or contact me directly.
Pete Maclean
+1 434-361-2386

Re: INBOX by any other name

[Pete Maclean]

At 05:45 PM 7/9/2002 +0200, Arnt Gulbrandsen wrote:
>All IMAP servers must present the user with an INBOX. INBOX may also be
>present under another name.

This is a matter that I have had to deal with.

>Is it common that IMAP servers also make INBOX available under a different
>name?

I have no idea if it is common but I chose not to do so.

>Is there a way to find out whether two different names refer to the
>same mailbox?

The only way I can think of would be to generate a unique message, APPEND 
it to one and then see if it appears in both.

Pete

Re: Netscape IMAP Client Authenticate

[Pete Maclean]

At 07:53 AM 6/30/2002 -0700, Mike Oliver wrote:
>How do I tell the Netscape IMAP Clients to use basic LOGIN to my IMAP server.

I test my IMAP server using Netscape Messenger and it's my experience that 
nothing special needs to be done to make it use a plain LOGIN.  My 
CAPABILITY happens to include AUTH=CRAM-MD5 but Messenger does not use 
that; just sends a simple LOGIN command.

What type of AUTHENTICATE is it trying to use in your case?

Pete Maclean

Re: SEARCH command limit in uw-imapd?

[Pete Maclean]

>Each OR effectively takes you down a level of nesting.  That is:
>
> OR (FROM "a") OR (FROM "b") OR (FROM "c") (FROM "d")
>is effectively
> OR (FROM "a") (OR (FROM "b") (OR (FROM "c") (FROM "d")))

I never realized this!  And my server does not handle such a thing 
appropriately.  It makes sense but it never occurred to me.  Is this 
detailed in the spec?  I cannot find any statement that clarifies it.

Pete

Progress Responses

[Pete Maclean]

be 
useful in preventing client time-outs:

 * PROGRESS 124

Servers would be encouraged to issue an initial progress response as soon 
as possible after receiving a command iff that response can convey a count 
of elements to be processed and/or a time-to-go, and the operation is 
expected to take more than a few seconds.

It is important that a client must not take any progress response, even one 
indicating 100% completion, as meaning that the command will actually 
complete successfully.

I don't feel competent to select a syntax that fits best into the IMAP 
mold.  Perhaps something like the following would be preferable to my 
suggestion above:

 * OK [PROGRESS  TTC ] 

Where TTC means time to completion.

One other thing I am unsure about is the number of seconds between progress 
responses.  Should this be decided by the client?  By the server?  By 
negotiation between the two?  Or should it simply be a fixed number?  My 
inclination is that it be chosen by the server within some allowable range, 
say 5 to 60.

Pete Maclean


-- 
-
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/imap-list.html
-

Re: Eudora and SEARCH

[Pete Maclean]

At 11:34 AM 6/11/2002 -0700, Mark Crispin wrote:
>On Tue, 11 Jun 2002, Simon Josefsson wrote:
> > I need to find all valid UIDs on the server in order to synch the
> > local header cache against the server.
>
>OK, this is the correct reason.
>
>Now, take it a step further.  If, instead of getting a list of valid UIDs
>via UID SEARCH ALL, you get the UID/sequence map via FETCH 1:* UID, do you
>see that you no longer need the UIDs except to in store your local cache?
>
>There are a lot of things that you can do using both UIDs and sequence
>numbers if you are imaginative.  Note, in particular, the very useful
>property that UIDs are strictly ascending.

I find it fascinating that I think I know this protocol so well yet I keep 
gaining new insights.

I always thought of FETCH 1:* UID as the way to get a list of UIDs -- 
probably since that is what most clients I deal with use.  But UID SEARCH 
ALL effectively provides the same information, right?  And does so with 
considerably less information being transferred.  Of course the question of 
which is truly the more efficient could depend on the particular server and 
its implementation of the two commands.

Eudora and SEARCH

[Pete Maclean]

In the two and a half years that I have been working with IMAP, I have 
become increasingly despondent about the sad state of IMAP implementation 
in email clients and how that is holding back the use of a very valuable 
protocol.  Of course there are clients, notably Mulberry, that use IMAP 
with brilliance, but the popular clients for Windows shape up as a very 
sorry bunch indeed.

Now, this mailing list has not shown itself to be a forum for client 
bashing, and I do not wish to make it one.  I am, however, so amazed by my 
latest discovery of bad IMAP usage that I feel compelled to share it 
here.  I find it surprising that it has not been mentioned before, but I 
searched the archives and can find no mention of it.

I noticed a while ago that Eudora (specifically version 5.1 for Windows) 
can be very slow when searching IMAP folders.  I wondered if it could be 
pulling down messages and searching them itself rather than using the 
SEARCH command.  Yesterday I decided to investigate and discovered what is 
really going on.  Eudora performs searches over messages in batches of 60 
messages at a time, as exemplified by this command:

00015 UID SEARCH UID 
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60
 
BODY target

This organization is intelligent and user-friendly in that it allows Eudora 
to display a progress bar and a "Stop" button.  (There seems to be a bug 
though in that the progress bar sometimes does not appear.)  It's a small 
matter, but one must wonder however why Eudora does not express this 
command as:

00015 UID SEARCH UID 1:60 BODY target

The trouble is that Eudora sends SEARCH commands for every possible UID 
from the lowest to the highest irrespective of whether or not the UID is 
actually represented in the selected mailbox.  Which means, in the worst 
pathological case of a mailbox containing just two messages, one with a UID 
of 1 and one with a UID of 4294967295 (the highest possible), a search 
would take 2 weeks or more.  (This is calculated based on timings made with 
my own server with the client running on the same computer.)

Having a few spare moments on my hands, I decided to test this.  I set up a 
server mailbox with two messages identified as described above.  I 
connected with Eudora and launched a search.  It did not take two 
weeks!  It took no time at all -- Eudora crashed at the very start.  (This 
is not so reprehensible perhaps.  I'll give Eudora the benefit of any doubt 
and assume this was a bug due to a boundary problem.  Most of us have them.)

I am glad we don't use 64-bit UIDs.  If we did, a worst case search of a 
mailbox holding two messages would take a little over 150 million years.

Pete


-- 
-
 For information about this mailing list, and its archives, see: 
 http://www.washington.edu/imap/imap-list.html
-

Re: Parse Error, Outlook Express, Implementing IMAP Server

[Pete Maclean]

Andy,

I am not sure that this is the only thing causing a problem, but the thing 
that jumps out for me is the message-id in the ENVELOPE response:

 <[EMAIL PROTECTED]>

It is required that this be sent either as a quoted string of a literal.

Pete Maclean

>Hi all,
>
>Interesting problem.. I'm putting together an IMAP Server and have run
>into a few problems.
>
>- Extract from Outlook Express IMAP Log -
>
>Microsoft Internet Messaging API 6.00.2600.
>IMAP Log started at 06/09/2002 14:51:28
>IMAP: 14:51:28 [db] Connecting to 'localhost' on port 143.
>IMAP: 14:51:28 [db] OnNotify: asOld = 0, asNew = 4, ae = 0
>IMAP: 14:51:28 [db] OnNotify: asOld = 4, asNew = 5, ae = 2
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 4
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * OK IMAP4 Pony Express Ready
>IMAP: 14:51:28 [tx] 0003 CAPABILITY
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * CAPABILITY IMAP4rev1
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] 0003 OK CAPABILITY completed
>IMAP: 14:51:28 [tx] LOGIN command sent
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] 0004 OK LOGIN Completed
>IMAP: 14:51:28 [tx] 0005 SELECT "INBOX"
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * 1 EXISTS
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * 1 RECENT
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] 0005 OK [READ-WRITE] SELECT completed
>IMAP: 14:51:28 [tx] 0006 UID FETCH 1:* (BODY.PEEK[HEADER.FIELDS
>(References X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)]
>ENVELOPE RFC822.SIZE UID FLAGS INTERNALDATE)
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] * 1 FETCH (FLAGS () UID 2 INTERNALDATE " 6-June-2002
>16:32:39 +0100" RFC822.SIZE 948 ENVELOPE ("Thu, 6 Jun 2002 16:32:39
>+0100" NIL (("Andy" NIL "andy" "neillans.co.uk")) (("Andy" NIL "andy"
>"neillans.co.uk")) (("Andy" NIL "andy" "neillans.co.uk")) ((NIL NIL
>"pexpress" "neillans.co.uk")) NIL NIL NIL
><[EMAIL PROTECTED]>) BODY[HEADER.FIELDS
>(References X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)]
>{44}
>IMAP: 14:51:28 [rx] Buffer (literal) of length 44
>IMAP: 14:51:28 [rx]
>IMAP: 14:51:28 [db] PARSE ERROR: hr=2148322516
>IMAP: 14:51:28 [rx] )
>IMAP: 14:51:28 [db] PARSE ERROR: hr=2148322522
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 5, ae = 3
>IMAP: 14:51:28 [rx] 0006 OK FETCH Completed
>IMAP: 14:51:28 [tx] Dropping connection, LOGOUT sent
>IMAP: 14:51:28 [db] Connection to 'localhost' closed.
>IMAP: 14:51:28 [db] OnNotify: asOld = 5, asNew = 0, ae = 5
>
>- End Extract -
>
>- Extract from server side log -
>
><: 0004 OK LOGIN Completed
>andy :> SELECT Command, 0005 "INBOX"
><: 0005* 1 EXISTS
><: 0005* 1 RECENT
><: 0005* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
><: 0005 OK [READ-WRITE] SELECT completed
>andy :> FETCH Command, 0006 1:* (BODY.PEEK[HEADER.FIELDS (References
>X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)] ENVELOPE
>RFC822.SIZE UID FLAGS INTERNALDATE)
><: * 1 FETCH (FLAGS () UID 2 INTERNALDATE " 6-June-2002 16:32:39 +0100"
>RFC822.SIZE 948 ENVELOPE ("Thu, 6 Jun 2002 16:32:39 +0100" NIL (("Andy"
>NIL "andy" "neillans.co.uk")) (("Andy" NIL "andy" "neillans.co.uk"))
>(("Andy" NIL "andy" "neillans.co.uk")) ((NIL NIL "pexpress"
>"neillans.co.uk")) NIL NIL NIL
><[EMAIL PROTECTED]>) BODY[HEADER.FIELDS
>(References X-Ref X-Priority X-MSMail-Priority X-MSOESRec Newsgroups)]
>{44}
>X-Priority: 3
>X-MSMail-Priority: Normal
>
>
>)
><: 0006 OK FETCH Completed
>
>- End Extract -
>
>Anyone have any ideas what the problem is? This has been driving me
>nutty for the last 4 - 5 days :p !
>
>If it's any help can open the server up so you guys can access it
>remotely.
>
>Thanks in advance!
>
>Andy Neillans
>
>--
>-
>  For information about this mailing list, and its archives, see:
>  http://www.washington.edu/imap/imap-list.html
>-

Re: Trash folders...

[Pete Maclean]

Well, I said that I wanted a clean way to do a move while completely 
forgetting about UID EXPUNGE -- in spite of the fact that I have that 
implemented in my own server.  So, I apologize for making a fuss about 
nothing.  I like UID EXPUNGE because it serves both for moves between 
server mailboxes and for moves from a server mailbox to an external one.

Of course (he said in a probably futile attempt to deflect some of the 
guilt born of his forgetfulness), the real problem ("real" as in something 
I cannot be held the least bit accountable for) is that clients do not use 
UID EXPUNGE.  I just ran a test with most of the clients in my small 
collection and not one of them took advantage of it to do a clean move.

I will also take this opportunity to throw in my two cents' worth of 
opinion about a MOVE command.  I feel it may be a pity that a MOVE was not 
incorporated into IMAP4 originally.  It is really hard to say though 
because, with a MOVE, the protocol would have been a rather different 
animal from the one we know.  However I think that adding a MOVE at this 
point would be a really bad, uh, move.  For one thing, I suspect that 
making it a requirement would cause a storm while making it optional would 
lead to very few people implementing it.

Pete

RE: Trash folders...

[Pete Maclean]

 >Folks, creeping featurism is not a good thing. It is especially not a good
 >thing when the end user function is already possible with the existing
 >functionality set, and the proposed new feature is a minor tweak that may be
 >more efficient on some servers.

I totally, earnestly agree about creeping featurism.  However I still yearn 
for a clean way for IMAP to implement a move operation.  (I mean "move" now 
from the point of view of the user, not implying anything about a MOVE per 
se in IMAP.)  And by "clean", I mean a move that does not leave 
flagged-deleted messages behind.  This seems so basic to me that I think it 
undeserving to brand it a symptom of creeping featurism.  Email clients 
lead users to expect to be able to move messages around, and there's no 
turning back on that.  At this point, maybe a MOVE in IMAP is a bad idea, 
but there should be other ways to do it, an atomic EXPUNGE as Mark 
mentioned for example.  Is that a really bad idea?  Can anyone see another way?

Now I am going to stick my neck out, risk the wrath of the keepers of IMAP, 
and suggest one other thing I want.  This, I admit, smells a little more of 
feature creep but it would make life very much more comfortable for many 
users.  I would like a feedback mechanism for the COPY command, so the 
server sends an untagged response as each message in a set is copied.  And 
let me hasten to add that this is not with any intention of making COPY 
other than an all-or-nothing affair, absolutely not.  It is for two 
reasons.  The first is so that the email client can display a progress 
indicator to the user.  (Outlook Express displays a progress bar anyway 
even though no actual progress is ever shown!)  The second is to prevent a 
client from timing out when a COPY takes a long time.

Pete

RE: Trash folders...

[Pete Maclean]

 > MOVE is unimplementable unless messages are stored in
 > a method that permits messages to be moved from one mailbox to another.
 > IMAP explicitly does not place such requirements upon the message store.

Indeed, but surely the same holds true for COPY!  One of the mailstores my 
server works with supports the moving of messages among folders but not 
copying.  I had to do a load of kludgey work to fake a copy operation.  It 
happens to work perfectly well but it is horribly slow.

Pete

RE: Trash folders...

[Pete Maclean]

 >> I understand that there was a well thought-out reason for not including a
 >> MOVE command in IMAP but I don't know what it was. Could somebody please
 >> fill us in? Thanks.

 >The entire world does not consist of mailbox formats in which each 
message is
 >a discrete file.

Please forgive me, but I totally do not understand that as a reason.  And I 
say this as someone whose server works with four distinct mailstores, only 
one of which stores messages as discrete files.

By the way, while thinking about this, a plausible reason did occur to 
me.  That is that it could be very difficult to implement a multi-message 
MOVE in such a way that it always either completely succeeded or completely 
failed.  Of course that obstacle could be alleviated either by limiting a 
MOVE command to moving a single message or by relaxing the all-or-nothing 
requirement and providing feedback on each message moved.

Pete

RE: Trash folders...

[Pete Maclean]

I am dealing with a somewhat similar problem.  Some users of my IMAP-server 
products find it odd that, when they move a message from one server folder 
to another, a flagged-as-deleted copy remains in the original folder.  This 
is of course an artifact of the necessity to implement the move as a COPY 
followed by a STORE FLAGS \DELETED.  One way to deal with this would be to 
have the server unilaterally expunge a message when its \DELETED flag is 
set when the server knows that a copy has successfully been made.  I am 
sure many users would like to have this as an option.  The trouble is that, 
when the user moves a message from a server folder to an external folder, 
the server cannot be sure that a copy has been made; it would therefore be 
dangerous and undesirable to expunge the message.

I understand that there was a well thought-out reason for not including a 
MOVE command in IMAP but I don't know what it was.  Could somebody please 
fill us in?  Thanks.

Pete