NTLM authentication popups, etc

[Adrian Chadd]

I'm working on a couple of paid squid + active directory deployments
and they're both seeing the occasional NTLM auth popup happening.

The workaround is pretty simple - just enable the IP auth cache. This
however doesn't solve the fundamental problem(s), whatever they are.

The symptom is logs like this:

[2009/06/15 16:20:17, 1] libsmb/ntlmssp.c:ntlmssp_update(334)
  got NTLMSSP command 1, expected 3

And vice versa (expected 3, got 1.) These correspond to states in
samba/source/include/ntlmssp.h - 1 is NTLMSSP_NEGOTIATE; 3 is
NTLMSSP_AUTH.

The conclusion here is that there's a disconnect between the
authentication state of the client -and- the authentication state of
ntlm_auth.

I'm trying to eliminate the possibilities here.

The stateful helper stuff seems correct enough, so requests aren't
being queued to already busy stateful helpers.

The other two possibilities I can immediately think of:

* 1 - authentication is aborted somewhere for whatever reason; an
authentication helper is stuck at the wrong point in the state engine;
the next request coming along starts at NTLMSSP_NEGOTIATE but the
ntlm_auth helper it is handed to is at NTLMSSP_AUTH (from the partial
authentication attempt earlier); error
* 2 - the web browser is stuffing different phases of the negotiation
down different connections to the proxy.

Now, debugging (1) shouldn't be difficult at all. I'm going to try and
determine the code paths that lead to and from an aborted auth
request, add in some debugging and see if the helper is closed.

Debugging (2) without full logs (impractical in this environment) and
full traffic dump (again, impractical in production) is going to be a
bit more difficult. I'm thinking about adding some hacky code to the
Squid ntlm auth class which keeps a log of the auth blobs
sent/received from/to the client and ntlm_auth. I can then dump the
entire conversation out to cache.log whenever authentication
fails/errors. This should at least give me a hint as to what is going
on.

(1) can explain the client state == NTLMSSP_NEGOTATE but ntlm_auth
state is NTLMSSP_AUTH problem but not vice versa. (2) explains both.
It is quite possible it is the combination of both however.

Now, the reason this is getting somewhat annoying and why I'd like to
try and understand/fix it is that -another- problem seen by one of
these clients is negotiate/ntlm authentication from IE (at least IE8)
through Squid. I've got packet dumps showing the browser sending
different phases of the negotiation down separate proxy connections
and then reusing the original one incorrectly. My medium term plan is
to take whatever evidence I have of this behaviour and throw it at the
IE group(s) at Microsoft but in the short term I'd like to make
certain the proxy authentication side of things is completely
blameless before I hand off stuff to third parties.

Ideas? Comments?



adrian

Re: NTLM authentication popups, etc

[Robert Collins]

On Tue, 2009-06-16 at 15:14 +0800, Adrian Chadd wrote:
 
 
 The stateful helper stuff seems correct enough, so requests aren't
 being queued to already busy stateful helpers.

IIRC you can multiplex requests on single stateful helpers now (see
deferred_requests - the count of how many concurrent transactions are
going on on that helper). I may be wrong. Regardless, you *must* queue
to the same helper though.

A trace of 84,9 29,9 may help.

 The other two possibilities I can immediately think of:
 
 * 1 - authentication is aborted somewhere for whatever reason; an
 authentication helper is stuck at the wrong point in the state engine;
 the next request coming along starts at NTLMSSP_NEGOTIATE but the
 ntlm_auth helper it is handed to is at NTLMSSP_AUTH (from the partial
 authentication attempt earlier); error

You can detect this by ascertaining the ntlm user request state (which
the trace above should give us) at the time of the failure. We preprend
the data with things like 'KK' to indicate where we think the state is
at.

 * 2 - the web browser is stuffing different phases of the negotiation
 down different connections to the proxy.

This would be fundamentally broken. OTOH I've seen firefox do this with
digest authentication so its entirely possible.

 Now, debugging (1) shouldn't be difficult at all. I'm going to try and
 determine the code paths that lead to and from an aborted auth
 request, add in some debugging and see if the helper is closed.

authenticateNTLMReleaseServer should be called to release a in progress
transaction on the helper.

 Debugging (2) without full logs (impractical in this environment) and
 full traffic dump (again, impractical in production) is going to be a
 bit more difficult. I'm thinking about adding some hacky code to the
 Squid ntlm auth class which keeps a log of the auth blobs
 sent/received from/to the client and ntlm_auth. I can then dump the
 entire conversation out to cache.log whenever authentication
 fails/errors. This should at least give me a hint as to what is going
 on.

Just the two settings I'm asking for should be enough. Just get a couple
of failed occurrences then reconfigure back to normal.

 (1) can explain the client state == NTLMSSP_NEGOTATE but ntlm_auth
 state is NTLMSSP_AUTH problem but not vice versa. (2) explains both.
 It is quite possible it is the combination of both however.
 
 Now, the reason this is getting somewhat annoying and why I'd like to
 try and understand/fix it is that -another- problem seen by one of
 these clients is negotiate/ntlm authentication from IE (at least IE8)
 through Squid. I've got packet dumps showing the browser sending
 different phases of the negotiation down separate proxy connections
 and then reusing the original one incorrectly. My medium term plan is
 to take whatever evidence I have of this behaviour and throw it at the
 IE group(s) at Microsoft but in the short term I'd like to make
 certain the proxy authentication side of things is completely
 blameless before I hand off stuff to third parties.
 
 Ideas? Comments? 

Blame MS :).

-Rob


signature.asc
Description: This is a digitally signed message part