Re: Outlook 2010 woes

[Bryan Holloway]

So after several days of more troubleshooting, I have some things to 
report to the list.


First and foremost, I have discovered that the issue has nothing to do 
with SSL/TLS, which was my earlier suspicion because after doing some 
PCAPs I discovered that the transactions were negotiating TLS 1.2 on the 
new server, as opposed to 1.0 on the old.


Also thank you for the rawlog suggestion: that helped a lot in 
determining what was happening on the IMAP level.


That all said, this is what I've discovered:

There is a very curious and reproducible four-second delay during the 
negotiation between server and client which is not present in Dovecot 
2.1. This is what our customer is complaining about using Outlook 2010.


During a plaintext TCP stream, I'm seeing this:

1. Client connects (SYN) to server.

2. Server ACKs and throws back CAPABILITIES.

3. User attempts to auth with DIGEST-MD5.

4. Server says, "no thanks." (Not sure why, but I don't believe this is 
relevant.)


5. User attempts to auth with plaintext.

6. Server says, "Yup. You are you. You're logged in."

7. Client sends the following: ID ("name" "Microsoft Outlook" "version" 
"14.0")


8. Server sends an ACK

... and then there's this very curious four-second delay.

9. Server then sends out new CAPABILITIES, and everything proceeds 
thereafter as normal and zippy and fast.


Does this shed any light on the subject?


On 10/13/16 11:21 AM, Bryan Holloway wrote:

On 10/13/16 11:01 AM, Aki Tuomi wrote:



On October 13, 2016 at 6:52 PM Konstantin Khomoutov
<flatw...@users.sourceforge.net> wrote:


On Thu, 13 Oct 2016 10:35:14 -0500
Bryan Holloway <br...@shout.net> wrote:


[...]

Is there a way to see the IMAP commands coming from the client?
I've tried looking at PCAPs, but of course they're encrypted so I
can't see the actual dialog going on between the server and
client. I didn't see an obvious way to do this in the docs.


If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the
IMAP traffic using the `ssldump` utility (which builds on
`tcpdump`).


I do, but the client is using a DH key exchange so I only have the
server-side private key.

Tried that using Wireshark's decoder features and ran into this
problem. I'm assuming I'd run into the same using ssldump, but I'll
give it a shot!


I think DH is not the culprit: just to be able to actually decode SSL
traffic, you must have the server private key when you're decoding the
SSL handshake phase -- to be able to recover the session keys, which
you then use to decode the actual tunneled data.


You can also enable only non DH algorithms in ssl settings if rawlog
isn't working for you.

Aki



Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report
my findings to the list.

Re: Outlook 2010 woes

[Bryan Holloway]

In case anyone is interested, we finally found the problem:

The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along 
with "plain". This is what was causing the four-second delay, but only 
with Outlook clients.


Everything is working great now across the board.

Thanks again to everyone's suggestions.

- bryan


On 10/27/16 7:09 PM, Bryan Holloway wrote:

So after several days of more troubleshooting, I have some things to
report to the list.

First and foremost, I have discovered that the issue has nothing to do
with SSL/TLS, which was my earlier suspicion because after doing some
PCAPs I discovered that the transactions were negotiating TLS 1.2 on the
new server, as opposed to 1.0 on the old.

Also thank you for the rawlog suggestion: that helped a lot in
determining what was happening on the IMAP level.

That all said, this is what I've discovered:

There is a very curious and reproducible four-second delay during the
negotiation between server and client which is not present in Dovecot
2.1. This is what our customer is complaining about using Outlook 2010.

During a plaintext TCP stream, I'm seeing this:

1. Client connects (SYN) to server.

2. Server ACKs and throws back CAPABILITIES.

3. User attempts to auth with DIGEST-MD5.

4. Server says, "no thanks." (Not sure why, but I don't believe this is
relevant.)

5. User attempts to auth with plaintext.

6. Server says, "Yup. You are you. You're logged in."

7. Client sends the following: ID ("name" "Microsoft Outlook" "version"
"14.0")

8. Server sends an ACK

... and then there's this very curious four-second delay.

9. Server then sends out new CAPABILITIES, and everything proceeds
thereafter as normal and zippy and fast.

Does this shed any light on the subject?


On 10/13/16 11:21 AM, Bryan Holloway wrote:

On 10/13/16 11:01 AM, Aki Tuomi wrote:



On October 13, 2016 at 6:52 PM Konstantin Khomoutov
<flatw...@users.sourceforge.net> wrote:


On Thu, 13 Oct 2016 10:35:14 -0500
Bryan Holloway <br...@shout.net> wrote:


[...]

Is there a way to see the IMAP commands coming from the client?
I've tried looking at PCAPs, but of course they're encrypted so I
can't see the actual dialog going on between the server and
client. I didn't see an obvious way to do this in the docs.


If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the
IMAP traffic using the `ssldump` utility (which builds on
`tcpdump`).


I do, but the client is using a DH key exchange so I only have the
server-side private key.

Tried that using Wireshark's decoder features and ran into this
problem. I'm assuming I'd run into the same using ssldump, but I'll
give it a shot!


I think DH is not the culprit: just to be able to actually decode SSL
traffic, you must have the server private key when you're decoding the
SSL handshake phase -- to be able to recover the session keys, which
you then use to decode the actual tunneled data.


You can also enable only non DH algorithms in ssl settings if rawlog
isn't working for you.

Aki



Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report
my findings to the list.

Re: Outlook 2010 woes

[Bryan Holloway]

On 11/1/16 6:20 PM, Bryan Holloway wrote:

In case anyone is interested, we finally found the problem:

The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along
with "plain". This is what was causing the four-second delay, but only
with Outlook clients.

Everything is working great now across the board.

Thanks again to everyone's suggestions.

- bryan



Sorry -- that wasn't very clear, was it:

Removing "digest-md5" fixed the issue.



On 10/27/16 7:09 PM, Bryan Holloway wrote:

So after several days of more troubleshooting, I have some things to
report to the list.

First and foremost, I have discovered that the issue has nothing to do
with SSL/TLS, which was my earlier suspicion because after doing some
PCAPs I discovered that the transactions were negotiating TLS 1.2 on the
new server, as opposed to 1.0 on the old.

Also thank you for the rawlog suggestion: that helped a lot in
determining what was happening on the IMAP level.

That all said, this is what I've discovered:

There is a very curious and reproducible four-second delay during the
negotiation between server and client which is not present in Dovecot
2.1. This is what our customer is complaining about using Outlook 2010.

During a plaintext TCP stream, I'm seeing this:

1. Client connects (SYN) to server.

2. Server ACKs and throws back CAPABILITIES.

3. User attempts to auth with DIGEST-MD5.

4. Server says, "no thanks." (Not sure why, but I don't believe this is
relevant.)

5. User attempts to auth with plaintext.

6. Server says, "Yup. You are you. You're logged in."

7. Client sends the following: ID ("name" "Microsoft Outlook" "version"
"14.0")

8. Server sends an ACK

... and then there's this very curious four-second delay.

9. Server then sends out new CAPABILITIES, and everything proceeds
thereafter as normal and zippy and fast.

Does this shed any light on the subject?


On 10/13/16 11:21 AM, Bryan Holloway wrote:

On 10/13/16 11:01 AM, Aki Tuomi wrote:



On October 13, 2016 at 6:52 PM Konstantin Khomoutov
<flatw...@users.sourceforge.net> wrote:


On Thu, 13 Oct 2016 10:35:14 -0500
Bryan Holloway <br...@shout.net> wrote:


[...]

Is there a way to see the IMAP commands coming from the client?
I've tried looking at PCAPs, but of course they're encrypted so I
can't see the actual dialog going on between the server and
client. I didn't see an obvious way to do this in the docs.


If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the
IMAP traffic using the `ssldump` utility (which builds on
`tcpdump`).


I do, but the client is using a DH key exchange so I only have the
server-side private key.

Tried that using Wireshark's decoder features and ran into this
problem. I'm assuming I'd run into the same using ssldump, but I'll
give it a shot!


I think DH is not the culprit: just to be able to actually decode SSL
traffic, you must have the server private key when you're decoding the
SSL handshake phase -- to be able to recover the session keys, which
you then use to decode the actual tunneled data.


You can also enable only non DH algorithms in ssl settings if rawlog
isn't working for you.

Aki



Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report
my findings to the list.

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 9:07 AM, Aki Tuomi wrote:



On October 13, 2016 at 4:55 PM Jerry <je...@seibercom.net> wrote:


On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:


I also extended the "Server Timeout" setting in OT2010 to 10 minutes,
which doesn't seem to help either. (!)


Outlook 2010 is a very old version. Why not update to the 2016 version.
I am running it without any problems. If you do update, remember to
remove the old version completely first.

--
Jerry


I do wonder if the real culprit is some firewall that timeouts the idle 
connection.

Aki



I considered that, but again everything worked fine until we moved them 
from 2.1 to 2.2. Their same firewall is in use.


Is there a way to see the IMAP commands coming from the client? I've 
tried looking at PCAPs, but of course they're encrypted so I can't see 
the actual dialog going on between the server and client. I didn't see 
an obvious way to do this in the docs.

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 10:23 AM, Konstantin Khomoutov wrote:

On Thu, 13 Oct 2016 09:53:19 -0500
Bryan Holloway <br...@shout.net> wrote:

[...]

Is there a way to see the IMAP commands coming from the client? I've
tried looking at PCAPs, but of course they're encrypted so I can't
see the actual dialog going on between the server and client. I
didn't see an obvious way to do this in the docs.


If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the IMAP
traffic using the `ssldump` utility (which builds on `tcpdump`).



I do, but the client is using a DH key exchange so I only have the 
server-side private key.


Tried that using Wireshark's decoder features and ran into this problem. 
I'm assuming I'd run into the same using ssldump, but I'll give it a shot!


Stupid privacy. :)

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 10:42 AM, Urban Loesch wrote:



Am 13.10.2016 um 16:53 schrieb Bryan Holloway:

On 10/13/16 9:07 AM, Aki Tuomi wrote:



On October 13, 2016 at 4:55 PM Jerry <je...@seibercom.net> wrote:


On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:


I also extended the "Server Timeout" setting in OT2010 to 10 minutes,
which doesn't seem to help either. (!)


Outlook 2010 is a very old version. Why not update to the 2016 version.
I am running it without any problems. If you do update, remember to
remove the old version completely first.

--
Jerry


I do wonder if the real culprit is some firewall that timeouts the
idle connection.

Aki



I considered that, but again everything worked fine until we moved
them from 2.1 to 2.2. Their same firewall is in use.

Is there a way to see the IMAP commands coming from the client? I've
tried looking at PCAPs, but of course they're encrypted so I can't see
the actual
dialog going on between the server and client. I didn't see an obvious
way to do this in the docs.



There is a "rawlog" feature, which writes down the hole decrypted imap
session in files.

...
service imap {
...
executable = imap postlogin
...
}

...

service postlogin {
  executable = script-login -d rawlog
  unix_listener postlogin {
  }
}
...

This should write *.in an *.out files to
"$mail_location/dovecot.rawlog/" directory for each imap session.
The directory should be writeable by the dovecot user. I tested this
some years ago, so I'm not shure if the configuration
is still valid.

Regards
Urban


Great! I will try this.

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 11:01 AM, Aki Tuomi wrote:



On October 13, 2016 at 6:52 PM Konstantin Khomoutov 
<flatw...@users.sourceforge.net> wrote:


On Thu, 13 Oct 2016 10:35:14 -0500
Bryan Holloway <br...@shout.net> wrote:


[...]

Is there a way to see the IMAP commands coming from the client?
I've tried looking at PCAPs, but of course they're encrypted so I
can't see the actual dialog going on between the server and
client. I didn't see an obvious way to do this in the docs.


If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the
IMAP traffic using the `ssldump` utility (which builds on
`tcpdump`).


I do, but the client is using a DH key exchange so I only have the
server-side private key.

Tried that using Wireshark's decoder features and ran into this
problem. I'm assuming I'd run into the same using ssldump, but I'll
give it a shot!


I think DH is not the culprit: just to be able to actually decode SSL
traffic, you must have the server private key when you're decoding the
SSL handshake phase -- to be able to recover the session keys, which
you then use to decode the actual tunneled data.


You can also enable only non DH algorithms in ssl settings if rawlog isn't 
working for you.

Aki



Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report 
my findings to the list.

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/12/16 4:11 PM, Joseph Tam wrote:



Old server:
* Ubuntu 10.04.4 LTS
* Dovecot 2.1.13
* Maildir++
* Local auth via passwd/shadow files

New server:
* Debian GNU/Linux 8.6
* Dovecot 2.2.13
* Maildir++
* Quotas enabled
* LDAP

Basically what's happening is that users are seeing large delays when
navigating between different IMAP folders. So, for example, user "X" is
sitting idle in their INBOX.


Rebuilding caches?  Do you get the same delay when going back to the folder
after the initial delay.

Joseph Tam 


No, but once sitting idle again for 10-15 seconds, the delay occurs 
again regardless of which folder you choose.


Am I understanding your question correctly? It really seems to me like 
Outlook is prematurely ending IMAP sessions.


I also extended the "Server Timeout" setting in OT2010 to 10 minutes, 
which doesn't seem to help either. (!)


I was considering enabling the auth_cache feature to see if that helps.

I'll let the list know what happens -- planning on doing that today.

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 8:55 AM, Jerry wrote:

On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:


I also extended the "Server Timeout" setting in OT2010 to 10 minutes,
which doesn't seem to help either. (!)


Outlook 2010 is a very old version. Why not update to the 2016 version.
I am running it without any problems. If you do update, remember to
remove the old version completely first.



Yeah -- totally not disagreeing with that statement ... the problem is 
that the customer is putting their foot down since everything worked 
fine with Dovecot 2.1.


But yes, I have mentioned that to them ...

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/13/16 9:06 AM, Bryan Holloway wrote:

On 10/13/16 8:55 AM, Jerry wrote:

On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:


I also extended the "Server Timeout" setting in OT2010 to 10 minutes,
which doesn't seem to help either. (!)


Outlook 2010 is a very old version. Why not update to the 2016 version.
I am running it without any problems. If you do update, remember to
remove the old version completely first.



Yeah -- totally not disagreeing with that statement ... the problem is
that the customer is putting their foot down since everything worked
fine with Dovecot 2.1.

But yes, I have mentioned that to them ...


I guess I should add that it would be one thing if there were a specific 
IMAP feature that a newer Dovecot version (2.2) supported and the client 
didn't, but I haven't been able to pinpoint it.


Obviously the behavior is different than what it was, but it would be a 
lot easier to convince the customer to upgrade if I could point a finger 
right at the "feature" in question.


In the meantime, I have to try and figure out what's changed ...

Outlook 2010 woes

[Bryan Holloway]

Hello, everyone.

We have recently begun migrating folks from an older server to a newer 
one, and things have been going quite well except for -- you guessed it 
-- Outlook 2010 users. Specifically it appears to be one customer in 
particular, and this particular customer has many nested mailboxes and 
lots of e-mail in general. Not sure if this is a factor.


Old server:
* Ubuntu 10.04.4 LTS
* Dovecot 2.1.13
* Maildir++
* Local auth via passwd/shadow files

New server:
* Debian GNU/Linux 8.6
* Dovecot 2.2.13
* Maildir++
* Quotas enabled
* LDAP

Basically what's happening is that users are seeing large delays when 
navigating between different IMAP folders. So, for example, user "X" is 
sitting idle in their INBOX. If they then click on another folder 
there's a good 6-7 second delay before you can view its contents. If you 
immediately then navigate to other folders, you get a rapid response. 
But if the client then goes idle again for 10+ seconds, you will get 
this delay again. Some are reporting the OS saying "Outlook is not 
responding." (Everyone is running Windows 7.)


Disclaimer: Yes, I know Outlook 2010 is a giant steaming pile of .

However, everything worked dandy with Dovecot 2.1 and of course my 
customer is harping on this fact.


Things I have tried/read about in the Dovecot list:
* Check headers only -- doesn't seem to help.
* Had a user completely remove their IMAP profile and re-add: no change.
* Had a user set up an entirely new account on a new computer: same 
symptoms.
* Someone mentioned to enable the "delay-newmail" workaround in 
20-imap.conf, but this wasn't enabled in our 2.1 install, so that seems 
like it would not help our case.


I was looking through the release notes of later Dovecots, and I noticed 
that in 2.2.14 there's mention of some issues with Outlook that were 
fixed, but it wasn't particularly specific about which versions. Should 
I be looking into that? And if so, is there a separate repository for 
newer dovecot? Currently the standard Debian 8.6 only has 2.2.13.


Any help, suggestions, or pointers would be greatly appreciated. Thank you!

- bryan

Re: Outlook 2010 woes

[Bryan Holloway]

On 10/12/16 11:51 AM, Konstantin Khomoutov wrote:

On Wed, 12 Oct 2016 10:59:33 -0500
Bryan Holloway <br...@shout.net> wrote:

[...]

New server:
* Debian GNU/Linux 8.6
* Dovecot 2.2.13
* Maildir++
* Quotas enabled
* LDAP

Basically what's happening is that users are seeing large delays when
navigating between different IMAP folders. So, for example, user "X"
is sitting idle in their INBOX. If they then click on another folder
there's a good 6-7 second delay before you can view its contents. If
you immediately then navigate to other folders, you get a rapid
response. But if the client then goes idle again for 10+ seconds, you
will get this delay again. Some are reporting the OS saying "Outlook
is not responding." (Everyone is running Windows 7.)

[...]

Any help, suggestions, or pointers would be greatly appreciated.


Do you see

  imap(username) Disconnected for inactivity in=X out=Y

in the logs?

From your description, it appears as if outlook gets disconnected at
some point, and that's why fast changing of the folders works OK (the
connection is live) and doing this after a pause forces a reconnect
with the following relogin.

Just a shot in the dark but still...


I do see quite a few of those, but I tend to see those for many clients 
across the board.


That does make a lot of sense though. I'm looking at the logs right now, 
and I see what amounts to a login, followed by a disconnect 10-15 
seconds later.


I thought that a standard IMAP connection stays open for at least 30 
minutes based on the RFC. (?)