Great idea! Here was the stack trace.
"qtp1094674892-17-acceptor-0@92ae196-ServerConnector@68b6f0d6{HTTP/1.1,
(http/1.1, h2c)}{10.248.13.143:32000}" #17 prio=5 os_prio=0 cpu=46454.44ms
elapsed=3686267.13s tid=0x00007fa2bd392800 nid=0x2700f runnable
[0x00007fa29f6bb000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.getHostByAddr([email protected]/Native
Method)
at
java.net.InetAddress$PlatformNameService.getHostByAddr([email protected]
/InetAddress.java:935)
at java.net.InetAddress.getHostFromNameService([email protected]
/InetAddress.java:659)
at java.net.InetAddress.getHostName([email protected]
/InetAddress.java:602)
at java.net.SocketPermission.getCanonName([email protected]
/SocketPermission.java:665)
at java.net.SocketPermission.impliesIgnoreMask([email protected]
/SocketPermission.java:987)
at java.net.SocketPermissionCollection.implies([email protected]
/SocketPermission.java:1437)
at java.security.Permissions.implies([email protected]
/Permissions.java:179)
at sun.security.provider.PolicyFile.implies([email protected]
/PolicyFile.java:994)
at java.security.ProtectionDomain.implies([email protected]
/ProtectionDomain.java:321)
at
java.security.ProtectionDomain.impliesWithAltFilePerm([email protected]
/ProtectionDomain.java:353)
at
java.security.AccessControlContext.checkPermission([email protected]
/AccessControlContext.java:450)
at java.security.AccessController.checkPermission([email protected]
/AccessController.java:897)
at java.lang.SecurityManager.checkPermission([email protected]
/SecurityManager.java:322)
at java.lang.SecurityManager.checkAccept([email protected]
/SecurityManager.java:940)
at sun.nio.ch.ServerSocketChannelImpl.accept([email protected]
/ServerSocketChannelImpl.java:305)
at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run([email protected]/Thread.java:829)
So it looks like the security manager is the culprit. I then
- added SOLR_SECURITY_MANAGER_ENABLED=false to my startup
- added a bad nameserver to my /etc/resolv.conf
- retested with the curl script
And got instant results. So I think this solves the issue. Thanks!
Since the jvm security manager is deprecated - Is there any compelling
reason to turn it back on? (Especially in a standalone environment
available only by app servers?)
-Tim
On Wed, Mar 22, 2023 at 2:50 PM Mikhail Khludnev <[email protected]> wrote:
> Hello,
> You can check jstack or Thread Dump in Solr Admin to localize the clog.
>
> On Wed, Mar 22, 2023 at 5:01 PM Tim Funk <[email protected]> wrote:
>
> > I ran into an interesting situation today with respect to latency due to
> > failed DNS lookups.
> >
> > THE SETUP
> > Using RHEL8, openjdk 11, solr 9.1.1. The solr instance was standalone in
> a
> > mostly out of the box config. (No SSL, only http)
> >
> > I had 3 entries in /etc/resolv.conf and the first one in the list was
> > "broken". It was timing out on requests.
> >
> > The situation - When I would issue a query - I'd have upwards of a 5
> second
> > pause prior to completion of the request. Which feels like a timeout
> being
> > hit. For example, while on the server "10.0.0.200"
> > curl -v '
> > http://10.0.0.200:8983/solr/mycore/select?q=*:*&fq=locale_s:en-us&rows=1
> '
> >
> > I then edited jetty.xml to
> > remove "org.eclipse.jetty.server.handler.InetAccessHandler" (since I
> wasn't
> > using an ACL anyways). From there - results seemed to get a little
> better.
> > I tended to get instant responses, but occasionally, I'd still see the
> > multi-second delay.
> >
> > I'm not sure how to track down this second piece of latency when DNS
> fails.
> > Is this an opportunity to also remove these lookups to improve
> performance?
> >
>
