Christopher,
Am 2019-07-08 um 19:55 schrieb Christopher Schultz:
Michael,
On 7/8/19 03:58, Osipov, Michael wrote:
Christopher,
Am 2019-07-05 um 19:07 schrieb Christopher Schultz:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Michael,
On 7/5/19 11:00, Osipov, Michael wrote:
Hi Christopher,
Am 2019-07-02 um 17:49 schrieb [ext] Osipov, Michael:
[...]
During your ~1min stall, Tomcat is still waiting for data,
right? When the connection fails, Tomcat drops its error
message at the same time, right? Can you post a stack trace of
what the Tomcat thread is doing at that time? I assume it's
blocked on a read of some kind.
I need to check this with jstack. I'll get back to you as soon
as possible.
So I checked this and was able to get the dump right in the moment
the request stalled. To my disappointment the offending thread did
not lock or did not wait for read() on the native socket.
I have noticed this:
"http-apr-127.0.1.2-8081-exec-3" #33 daemon prio=5 os_prio=15
tid=0x0000000a68036800 nid=0x188be runnable [0x00007fffdd1cc000]
java.lang.Thread.State: RUNNABLE at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
va:350)
- - locked <0x0000000965edc140> (a java.net.SocksSocketImpl)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
Impl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589) at
java.net.Socket.connect(Socket.java:538) at
java.net.Socket.<init>(Socket.java:434) at
java.net.Socket.<init>(Socket.java:211) at
com.sun.jndi.ldap.Connection.createSocket(Connection.java:375) at
com.sun.jndi.ldap.Connection.<init>(Connection.java:215) at
com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) at
com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1609) at
com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2749) at
com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:199)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:217
)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:195)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:217
)
at
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.ja
va:156)
at
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.jav
a:86)
at
javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:6
84)
at
javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313
)
at javax.naming.InitialContext.init(InitialContext.java:244)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at
javax.naming.directory.InitialDirContext.<init>(InitialDirContext.jav
a:101)
at
net.sf.michaelo.dirctxsrc.DirContextSource$GSSInitialDirContext.<init
(DirContextSource.java:115)
at
net.sf.michaelo.dirctxsrc.DirContextSource$1.run(DirContextSource.jav
a:606)
at
net.sf.michaelo.dirctxsrc.DirContextSource$1.run(DirContextSource.jav
a:583)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422) at
net.sf.michaelo.dirctxsrc.DirContextSource.getGssApiDirContext(DirCon
textSource.java:583)
at
net.sf.michaelo.dirctxsrc.DirContextSource.getDirContext(DirContextSo
urce.java:692)
at
net.sf.michaelo.tomcat.realm.ActiveDirectoryRealm.open(ActiveDirector
yRealm.java:321)
at
net.sf.michaelo.tomcat.realm.ActiveDirectoryRealm.getPrincipal(Active
DirectoryRealm.java:268)
at
net.sf.michaelo.tomcat.realm.ActiveDirectoryRealm.authenticate(Active
DirectoryRealm.java:255)
at
net.sf.michaelo.tomcat.authenticator.SpnegoAuthenticator.doAuthentica
te(SpnegoAuthenticator.java:166)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authentica
torBase.java:575)
at
org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.j
ava:556)
We query the Active Directory via LDAP with the user's Kerberos
principal. As you can see the thread is waiting for a socket to
connect. No DCs are hardcoded, they are all retreived via DNS SRV
lookups for our AD site. The point here is that we have major
trouble with two of four DCs at our site not properly respoding to
services like DNS, Kerberos, and LDAP. (Completely out of my
department's control) I have made a quick standalone reproducer to
try those faulty DCs on port 389/3268 and I had my confirmation.
They do block the thread for more than a minute (OS connect
timeout).
Our counter measures were to reduce the default connect timeout
for InitialDirContext down to 1000 ms and query another local AD
site which is not serving our subnet.
So, thank you very much giving me the right pointer to start!
Strange that everything seems to work well when you connect directly
to Tomcat. Can you confirm that you *never* have any issues connecting
directly to Tomcat? Or did you just get lucky a few times?
The issue did not show up via Tomcat directly because Tomcat does not
drop the request (timeout), the client simply waits for it. Our previous
services never used HTTPd as reserve proxy. I started to use it to gain
some experience and prepare for potential balancing requirements.
One question arises though: How do I properly size the
ProxyTimeout parameter? The longest possible request?
I think that's really up to you. If it's too low, you'll end up with
probably many hung LDAP queries with no client waiting on them, right?
If it's too high, you'll make users wait and they might just stop and
try again, which comes to the same conclusion.
What if you add a timeout to your LDAP queries instead?
The queries do not hang. The connect does. AS soon as the connection is
established, it is pretty fast. I have not set
"com.sun.jndi.ldap.connect.timeout=1000" and verified it to work. It
will quickly fail over to the next available DC from SRV RRs.
That's good that (a) it can be configured and (b) it actually works. Is
that LDAP connect timeout set as a system property? Can it be set on a
per-connection basis using e.g. a connection URL parameter? I'm mostly
asking for my own edification, and possibly to help anyone with a
similar problem in the future.
That's actually straight forward:
<GlobalNamingResources>
<Resource name="gc/ad001.siemens.net"
type="net.sf.michaelo.dirctxsrc.DirContextSource"
factory="net.sf.michaelo.dirctxsrc.DirContextSourceFactory"
urls="ldap://ad001.siemens.net:3268";
auth="gssapi" loginEntryName="tomcat-initiate" referral="ignore"
additionalProperties="com.siemens.dymowerk.activedirectory.site=S-DEBLN-01;com.sun.jndi.ldap.connect.timeout=1000"
/>
</GlobalNamingResources>
and the wire it on the context.xml:
<Valve className="net.sf.michaelo.tomcat.authenticator.SpnegoAuthenticator"
loginEntryName="tomcat-accept" />
<Realm className="net.sf.michaelo.tomcat.realm.ActiveDirectoryRealm"
dirContextSourceName="gc/ad001.siemens.net" />
Regards,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
