RE: ConcurrentModificationException due to high volume of requests
Thank you Shalin. I've created SOLR-9414 KATHERINE MORA Senior Engineer -Original Message- From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Sent: Saturday, August 13, 2016 7:22 PM To: solr-user@lucene.apache.org Subject: Re: ConcurrentModificationException due to high volume of requests This is a bug. Please open a Jira issue. We need to dig where else this param map is being used. It is possible that there are more such bugs. On Fri, Aug 12, 2016 at 10:57 PM, Katherine Mora < km...@searchtechnologies.com> wrote: > Hi All, > > We are using Solr 5.2.1 in a production environment where we have a > high volume of requests. We are not having any timeouts or performance issues. > However, the logs are filled with this exception. > We've been researching and there isn't a lot of information about this > problem or why it happens. We have tried to reproduce it by sending > queries using JMeter and it only happens when multiple queries are > sent at the same time. > > Checking the code we found that this exception is within the core Solr > code and part of the exception caught our attention as it is logging > debug information (we are using INFO logging level) > (org.apache.solr.search.stats. > LocalStatsCache.get(LocalStatsCache.java:40)). > > Has anyone seen this exception before? Would it be OK to generate a patch? > We were thinking about commenting the debug line or adding try/catch > statements. > > Thank you! > > > null:java.util.ConcurrentModificationException > at java.util.LinkedHashMap$LinkedHashIterator.nextEntry( > LinkedHashMap.java:394) > at java.util.LinkedHashMap$EntryIterator.next( > LinkedHashMap.java:413) > at java.util.LinkedHashMap$EntryIterator.next( > LinkedHashMap.java:412) > at > org.apache.solr.common.params.ModifiableSolrParams.toString( > ModifiableSolrParams.java:201) > at java.lang.String.valueOf(String.java:2849) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at org.apache.solr.request.SolrQueryRequestBase.toString( > SolrQueryRequestBase.java:165) > at org.apache.solr.search.stats.LocalStatsCache.get( > LocalStatsCache.java:40) > at org.apache.solr.handler.component.QueryComponent. > process(QueryComponent.java:366) > at org.apache.solr.handler.component.SearchHandler. > handleRequestBody(SearchHandler.java:255) > at org.apache.solr.handler.RequestHandlerBase.handleRequest( > RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute( > HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call( > HttpSolrCall.java:450) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:227) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:196) > at org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1652) > at org.eclipse.jetty.servlet.ServletHandler.doHandle( > ServletHandler.java:585) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:143) > at org.eclipse.jetty.security.SecurityHandler.handle( > SecurityHandler.java:577) > at org.eclipse.jetty.server.session.SessionHandler. > doHandle(SessionHandler.java:223) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:143) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:97) > at org.eclipse.jetty.server.handler.RequestLogHandler. > handle(RequestLogHandler.java:95) > at org.eclipse.jetty.server.handler.ContextHandler. > doHandle(ContextHandler.java:1129) > at org.eclipse.jetty.servlet.ServletHandler.doScope( > ServletHandler.java:515) > at org.eclipse.jetty.server.session.SessionHandler. > doScope(SessionHandler.java:185) > at org.eclipse.jetty.server.handler.ContextHandler. > doScope(ContextHandler.java:1061) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.ContextHandlerCollection. > handle(ContextHandlerCollection.java:215) > at org.eclipse.jetty.server.handler.HandlerCollection. > handle(HandlerCollection.java:110) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:310) >
ConcurrentModificationException due to high volume of requests
Hi All, We are using Solr 5.2.1 in a production environment where we have a high volume of requests. We are not having any timeouts or performance issues. However, the logs are filled with this exception. We've been researching and there isn't a lot of information about this problem or why it happens. We have tried to reproduce it by sending queries using JMeter and it only happens when multiple queries are sent at the same time. Checking the code we found that this exception is within the core Solr code and part of the exception caught our attention as it is logging debug information (we are using INFO logging level) (org.apache.solr.search.stats.LocalStatsCache.get(LocalStatsCache.java:40)). Has anyone seen this exception before? Would it be OK to generate a patch? We were thinking about commenting the debug line or adding try/catch statements. Thank you! null:java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) at org.apache.solr.common.params.ModifiableSolrParams.toString(ModifiableSolrParams.java:201) at java.lang.String.valueOf(String.java:2849) at java.lang.StringBuilder.append(StringBuilder.java:128) at org.apache.solr.request.SolrQueryRequestBase.toString(SolrQueryRequestBase.java:165) at org.apache.solr.search.stats.LocalStatsCache.get(LocalStatsCache.java:40) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:366) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) KATHERINE MORA Senior Engineer
RE: Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)
Hi Zara, I think that is done when you generate the self-signed certificate and the key. If you check the documentation: https://cwiki.apache.org/confluence/display/solr/Enabling+SSL#EnablingSSL-Generateaself-signedcertificateandakey it says: The "-ext SAN=..." keytool option allows you to specify all the DNS names and/or IP addresses that will be allowed during hostname verification -Original Message- From: Zara Parst [mailto:edotserv...@gmail.com] Sent: Thursday, March 03, 2016 2:42 PM To: solr-user@lucene.apache.org Subject: Re: Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1) Hello Katherine, I am sorry to ask this question. But really i need some light on bellow matter. I want to run solr in cloud mode . So obliviously I am going to use zookeeper. My quorum are distributed on 3 server with static ip , lets say server.1=xx.xx.x1:2888:3888 server.2=xx.xx.x2:2889:3889 server.3=xx.xx.x3:2890:3890 With solr pointing to this ensemble. Now my concern is how should I protect it to other unauthorized zkClient to connect above quorum. One way could be don't open the port for the client but then how will solr connect ? other problem is how to safeguard quorum interconnection. I observed a weird behavior that I can point a fourth zookeeper from my local to the above quorum (i have to know only ip and port which is not tough to find) and it will be absorbed as a part of quorum and then I can use my local zkClient to connect my local zoookeeper and have access to quorum which we don't want. I want to define quorum in a way that foreign zookeeper server is not able to become part already configured quorum. Again one more strange behavior about znode of zookeeper, User A can set ACL of a znode and user B which can connect to zookeeper but can't see the content as it will throw ACL error that is fine but strange thing is user B can still delete the znode of A which he cant see. :( I think a hell lot of things is not clear about zoookeeper security. Please can you help me ? And don't forget my thanks in advance. On Thu, Mar 3, 2016 at 7:32 PM, Katherine Mora wrote: > Hi Jeff, > > Are you still having the same issue or did you manage to fix it? > > I have the default files that come with the Solr 5.2.1 installation > (I’m copying those below). > I’m pretty sure my password is correct, unless the tool is generating > one that does not match my version? I’m using the jetty jar that comes > with the Solr installation to generate it, so I’m assuming this is not the > case. > > > Jetty-ssl.xml > > http://www.eclipse.org/jetty/configure_9_0.dtd";> > > > > > > > class="org.eclipse.jetty.util.ssl.SslContextFactory"> >default="./etc/solr-ssl.keystore.jks"/> >name="solr.jetty.keystore.password" default="secret"/> >default="./etc/solr-ssl.keystore.jks"/> >name="solr.jetty.truststore.password" default="secret"/> >name="solr.jetty.ssl.needClientAuth" default="false"/> >name="solr.jetty.ssl.wantClientAuth" default="false"/> > > > SSLv3 > > > > > SSL_RSA_WITH_DES_CBC_SHA > SSL_DHE_RSA_WITH_DES_CBC_SHA > SSL_DHE_DSS_WITH_DES_CBC_SHA > SSL_RSA_EXPORT_WITH_RC4_40_MD5 > SSL_RSA_EXPORT_WITH_DES40_CBC_SHA > SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA > SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA > > > > > > > > > >class="org.eclipse.jetty.server.HttpConfiguration"> > > >class="org.eclipse.jetty.server.SecureRequestCustomizer"/> > > > > > > jetty-https.xml > > http://www.eclipse.org/jetty/configure_9_0.dtd";> > > > > > > > > > > > > > > > > > > > > > > > > name="solr.jetty.ssl.acceptors" default="-1"/> > name="solr.jetty.ssl.selectors" default="-1"/> > > > > > http/1.1 > refid="sslContextFactory"/> > > > > > > > > > > > default="8983" /> > default="5"/> > name="solr.jetty.https.soLingerTime" default="-1"/> > name=&q
RE: Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)
Hi Jeff,
Are you still having the same issue or did you manage to fix it?
I have the default files that come with the Solr 5.2.1 installation (I’m
copying those below).
I’m pretty sure my password is correct, unless the tool is generating one that
does not match my version? I’m using the jetty jar that comes with the Solr
installation to generate it, so I’m assuming this is not the case.
Jetty-ssl.xml
http://www.eclipse.org/jetty/configure_9_0.dtd";>
SSLv3
SSL_RSA_WITH_DES_CBC_SHA
SSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHA
SSL_RSA_EXPORT_WITH_RC4_40_MD5
SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
jetty-https.xml
http://www.eclipse.org/jetty/configure_9_0.dtd";>
http/1.1
-Original Message-
From: Jeff Wu [mailto:wuhai...@gmail.com]
Sent: Monday, February 29, 2016 12:58 PM
To: solr-user@lucene.apache.org
Subject: Re: Prevent the SSL Keystore and Truststore password from showing up
in the Solr Admin and Linux processes (Solr 5.2.1)
Hi Katherine, we had exact the same issue, we need to protect our password.
Anyone who can access to solr server can do "ps -elf|grep java" to grep the
solr commandline, and it has all the password in plain text.
The /bin/solr shell will set 10 related system property:
SOLR_SSL_OPTS=" -Dsolr.jetty.keystore=$SOLR_SSL_KEY_STORE \
-Dsolr.jetty.keystore.password=$SOLR_SSL_KEY_STORE_PASSWORD \
-Dsolr.jetty.truststore=$SOLR_SSL_TRUST_STORE \
-Dsolr.jetty.truststore.password=$SOLR_SSL_TRUST_STORE_PASSWORD \
-Dsolr.jetty.ssl.needClientAuth=$SOLR_SSL_NEED_CLIENT_AUTH \
-Dsolr.jetty.ssl.wantClientAuth=$SOLR_SSL_WANT_CLIENT_AUTH"
SOLR_SSL_OPTS+=" -Djavax.net.ssl.keyStore=$SOLR_SSL_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$SOLR_SSL_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$SOLR_SSL_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$SOLR_SSL_TRUST_STORE_PASSWORD"
and also
SOLR_JETTY_CONFIG+=("--module=https")
The questions we have:
1. We doubt "OBF:XYZ" does not work when you set to solr.in.sh. the
javax.net.ssl can't work with jetty OBF. What we saw is Incorrect password
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:852)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:732)
at sun.reflect.GeneratedConstructorAccessor280.newInstance(Unknown Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)
2. Is there any good sample we can referent to configure in jetty-https.xml and
jetty-ssl.xml to leverage Jetty OBF?
Katherine, can you share your jetty-ssl.xml and jetty-https.xml?
From this link:
http://www.eclipse.org/jetty/documentation/9.2.6.v20141205/configuring-ssl.html,
did you put keystore files under jetty home?
and followed this sample?
/etc/keystore
OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4
OBF:1u2u1wml1z7s1z7a1wnl1u2g
/etc/keystore
OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4
2016-02-15 13:23 GMT-05:00 Katherine Mora
mailto:km...@searchtechnologies.com>>:
> Hello All,
>
> I've configured Solr 5.2.1 to enable SSL by following the instructions
> listed in the Wiki in Enabling SSL<
> https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This
> is working fine. However, if I go to the Solr Admin (Dashboard -> JVM
> -> Args) or if I list the processes running in the computer, I can see
> the password that I set in the solr.in.sh script for
> SOLR_SSL_KEY_STORE_PASSWORD and
> SOLR_SSL_TRUST_STORE_PASSWORD:
>
> -Dsolr.jetty.truststore.password=XYZ
> -Dsolr.jetty.keystore.password=XYZ
> -Djavax.net.ssl.trustStorePassword=XYZ
> -Djavax.net.ssl.
Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)
der.JavaKeyStore.engineLoad(JavaKeyStore.java:770) ... 68 more Katherine Mora
