Error on searches containing specific character pattern

2020-09-01 Thread Andy @ BlueFusion 

Hi All,

I have an 8.6.0 instance that is working well with one exception.

It returns an error when the search term follows a pattern of numbers & 
alpha characters such as:


 * 1a1 aa
 * 1a1 1aa
 * 1a1 11

Similar patterns that don't error

 * 1a1 a
 * 1a1 1
 * 1a11 aa
 * 11a1 aa
 * 1a1aa
 * 11a11 aa

The error is:

|"trace":"java.lang.ArrayIndexOutOfBoundsException: 0\n\t at 
org.apache.lucene.util.QueryBuilder.newSynonymQuery(QueryBuilder.java:653)\n\t 
at 
org.apache.solr.parser.SolrQueryParserBase.newSynonymQuery(SolrQueryParserBase.java:617)\n\t 
at 
org.apache.lucene.util.QueryBuilder.analyzeGraphBoolean(QueryBuilder.java:533)\n\t 
at 
org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:320)\n\t 
at 
org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:240)\n\t 
at 
org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:524)\n\t 
at 
org.apache.solr.parser.QueryParser.newFieldQuery(QueryParser.java:62)\n\t 
at 
org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1122)\n\t 
at 
org.apache.solr.parser.QueryParser.MultiTerm(QueryParser.java:593)\n\t 
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:142)\n\t at 
org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at 
org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at 
org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at 
org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at 
org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at 
org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at 
org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)\n\t 
at 
org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:260)\n\t 
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)\n\t 
at org.apache.solr.search.QParser.getQuery(QParser.java:174)\n\t at 
org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)\n\t 
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)\n\t 
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\t 
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)\n\t at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)\n\t 
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)\n\t 
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)\n\t 
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)\n\t 
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\t 
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\t 
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\t 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\t 
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\t 
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\t 
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\t 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\t 
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\t 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\t 
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\t 
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\t 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\t 
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\t 
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\t 
at org.eclipse.jetty.server.Server.handle(Server.java:505)\n\t at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\t at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\t 
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\t 
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\t 
at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\t 
at

Re: Addreplica throwing error when authentication is enabled

2020-09-01 Thread yaswanth kumar 
Hi Ben

Thanks for looking.. but I am not understanding about the file encrypted stuff 
that you mentioned?? Which file are you saying encrypted ? Security.json??

Sent from my iPhone

> On Sep 1, 2020, at 10:56 PM, Ben  wrote:
> 
> It appears the issue is with the encrypted file. Are these files encrypted?
> If yes, you need to decrypt it first.
> 
> moreCaused by: javax.crypto.BadPaddingException: RSA private key operation
> failed
> 
> Best,
> Ben
> 
>> On Tue, Sep 1, 2020, 10:51 PM yaswanth kumar  wrote:
>> 
>> Can some one please help me on the below error??
>> 
>> Solr 8.2; zookeeper 3.4
>> 
>> Enabled authentication and authentication and make sure that the role gets
>> all access
>> 
>> Now just add a collection with single replica and once done .. now try to
>> add another replica with addreplica solr api and that’s throwing error ..
>> note: this is happening only when security.json was enabled with
>> authentication
>> 
>> Below is the error
>> Collection: test operation: restore
>> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replicaCollection: test operation: restore
>> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
>> replica at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
>> at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd$$Lambda$798/.run(Unknown
>> Source) at
>> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
>> at
>> org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
>> at
>> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
>> at
>> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/.run(Unknown
>> Source) at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
>> org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
>> private key operation failed at
>> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325) at
>> org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
>> at
>> org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
>> at
>> org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:468)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.makeRequest(Http2SolrClient.java:455)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:364)
>> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
>> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) at
>> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
>> at
>> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
>> at
>> org.apache.solr.handler.component.HttpShardHandler$$Lambda$512/.call(Unknown
>> Source) at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ... 5 moreCaused by: javax.crypto.BadPaddingException: RSA private key
>> operation failed at
>> java.base/sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
>> at java.base/sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) at
>> java.base/sun.security.rsa.RSACore.rsa(RSACore.java:149) at
>> java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355) at
>> java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392)
>> at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2260) at
>>

Re: Addreplica throwing error when authentication is enabled

2020-09-01 Thread Ben 
It appears the issue is with the encrypted file. Are these files encrypted?
If yes, you need to decrypt it first.

moreCaused by: javax.crypto.BadPaddingException: RSA private key operation
failed

Best,
Ben

On Tue, Sep 1, 2020, 10:51 PM yaswanth kumar  wrote:

> Can some one please help me on the below error??
>
> Solr 8.2; zookeeper 3.4
>
> Enabled authentication and authentication and make sure that the role gets
> all access
>
> Now just add a collection with single replica and once done .. now try to
> add another replica with addreplica solr api and that’s throwing error ..
> note: this is happening only when security.json was enabled with
> authentication
>
> Below is the error
> Collection: test operation: restore
> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
> replicaCollection: test operation: restore
> failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create
> replica at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd$$Lambda$798/.run(Unknown
> Source) at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
> at
> org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
> at
> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/.run(Unknown
> Source) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
> org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
> private key operation failed at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325) at
> org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
> at
> org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
> at
> org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:468)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.makeRequest(Http2SolrClient.java:455)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:364)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) at
> org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
> at
> org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
> at
> org.apache.solr.handler.component.HttpShardHandler$$Lambda$512/.call(Unknown
> Source) at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ... 5 moreCaused by: javax.crypto.BadPaddingException: RSA private key
> operation failed at
> java.base/sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
> at java.base/sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) at
> java.base/sun.security.rsa.RSACore.rsa(RSACore.java:149) at
> java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355) at
> java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392)
> at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2260) at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:323) ...
> 20 more
>
> That's the error stack trace I am seeing, as soon as I call the restore
> API I am seeing the collection test with a single core on the cloud but its
> in down state.
>
> No of nodes that I configured with solr cloud is : 2
> Testing on a single collection with 2 replicas
> Here

Addreplica throwing error when authentication is enabled

2020-09-01 Thread yaswanth kumar 
Can some one please help me on the below error??

Solr 8.2; zookeeper 3.4

Enabled authentication and authentication and make sure that the role gets all 
access 

Now just add a collection with single replica and once done .. now try to add 
another replica with addreplica solr api and that’s throwing error .. note: 
this is happening only when security.json was enabled with authentication 

Below is the error
Collection: test operation: restore 
failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create 
replicaCollection: test operation: restore 
failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create 
replica at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030)
 at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013)
 at 
org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177)
 at 
org.apache.solr.cloud.api.collections.AddReplicaCmd$$Lambda$798/.run(Unknown
 Source) at 
org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199)
 at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708)
 at org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286) 
at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
 at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
 at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
 at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/.run(Unknown
 Source) at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:834)Caused by: 
org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA 
private key operation failed at 
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325) at 
org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
 at 
org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61)
 at 
org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239)
 at 
org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:468)
 at 
org.apache.solr.client.solrj.impl.Http2SolrClient.makeRequest(Http2SolrClient.java:455)
 at 
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:364)
 at 
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746)
 at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) at 
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238)
 at 
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199)
 at 
org.apache.solr.handler.component.HttpShardHandler$$Lambda$512/.call(Unknown
 Source) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
 ... 5 moreCaused by: javax.crypto.BadPaddingException: RSA private key 
operation failed at 
java.base/sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149)
 at java.base/sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) at 
java.base/sun.security.rsa.RSACore.rsa(RSACore.java:149) at 
java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355) at 
java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) 
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2260) at 
org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:323) ... 20 
more
 
That's the error stack trace I am seeing, as soon as I call the restore API I 
am seeing the collection test with a single core on the cloud but its in down 
state.
 
No of nodes that I configured with solr cloud is : 2 
Testing on a single collection with 2 replicas
Here is my security.json looks like
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":
{ "admin":"", "dev":""}
,
"":{"v":11},
"blockUnknown":true,
"forwardCredentials":true},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":
{ "solradmin":[ "admin", "dev"], "dev":["read"]}
,
"":{"v":9},
"permissions":[
{

Re: Understanding Solr heap %

2020-09-01 Thread yaswanth kumar 
I got some understanding now about my actual question.. thanks all for your 
valuable theories

Sent from my iPhone

> On Sep 1, 2020, at 2:01 PM, Joe Doupnik  wrote:
> 
>  As I have not received the follow-on message to mine I will cut 
> it below.
> My comments on that are the numbers are the numbers. More importantly, I 
> have run large imports ~0.5M docs and I have watched as that progresses. My 
> crawler paces material into Solr. Memory usage (Linux "top") shows cyclic 
> small rises and falls, peaking at about 2GB as the crawler introduces 1 and 3 
> second pauses every hundred and thousand submissions.. The test shown in my 
> original message is sufficient to show the nature of Solr versions and the 
> choice of garbage collector, and other folks can do similar experiments on 
> their gear. The quoted tests are indeed representative of large and small 
> amounts of various kinds of documents, and I say that based on much 
> experience observing the details.
> Quibble about GC names if you wish, but please do see those experimental 
> results. Also note the difference in our SOLR_HEAP values: 2GB in my work, 
> 8GB in yours. I have found 2GB to work well for importing small and very 
> large collections (of many file varieties).
> Thanks,
> Joe D.
>> This is misleading and not particularly good advice.
>> 
>> Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
>> it with Java 8 and Solr 6.6.2 for a few years.
>> 
>> A test with eighty documents doesn’t test anything. Try a million documents 
>> to
>> get Solr memory usage warmed up.
>> 
>> GC_TUNE has been in the solr.in.sh file for a long time. Here are the 
>> settings
>> we use with Java 8. We have about 120 hosts running Solr in six prod 
>> clusters.
>> 
>> SOLR_HEAP=8g
>> # Use G1 GC  -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
> 
>> On 01/09/2020 16:39, Joe Doupnik wrote:
>> Erick states this correctly. To give some numbers from my experiences, 
>> here are two slides from my presentation about installing Solr 
>> (https://netlab1.net/, locate item "Solr/Lucene Search Service"):
>>> 
>> 
>>> 
>> 
>> Thus we see a) experiments are the key, just as Erick says, and b) the 
>> choice of garbage collection algorithm plays a major role.
>> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, 
>> plus stock GC_TUNE values. Your "memorage" may vary.
>> Thanks,
>> Joe D.
>> 
>>> On 01/09/2020 15:33, Erick Erickson wrote:
>>> You want to run with the smallest heap you can due to Lucene’s use of 
>>> MMapDirectory, 
>>> see the excellent:
>>> 
>>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>> 
>>> There’s also little reason to have different Xms and Xmx values, that just 
>>> means you’ll
>>> eventually move a bunch of memory around as the heap expands, I usually set 
>>> them both
>>> to the same value.
>>> 
>>> How to determine what “the smallest heap you can” is? Unfortunately there’s 
>>> no good way
>>> outside of stress-testing your application with less and less memory until 
>>> you have problems,
>>> then add some extra…
>>> 
>>> Best,
>>> Erick
>>> 
> On Sep 1, 2020, at 10:27 AM, Dominique Bejean  
> wrote:
> 
> Hi,
> 
> As all Java applications the Heap memory is regularly cleaned by the
> garbage collector (some young items moved to the old generation heap zone
> and unused old items removed from the old generation heap zone). This
> causes heap usage to continuously grow and reduce.
> 
> Regards
> 
> Dominique
> 
> 
> 
> 
> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar  a
> écrit :
> 
> Can someone make me understand on how the value % on the column Heap is
> calculated.
> 
> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
> not yet live neither interms of indexing or searching, but I do see some
> spikes in the HEAP column against nodes when I refresh the page multiple
> times. Its like almost going to 95% (sometimes) and then coming down to 
> 50%
> 
> Solr version: 8.2
> Zookeeper: 3.4
> 
> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
> RAM size on the node is 16GB)
> 
> Basically need to understand if I need to worry about this heap % which
> was quite altering before making it live? or is that quite normal, because
> this is new UI change on solr cloud is kind of new to us as we used to 
> have
> solr 5 version before and this UI component doesn't exists then.
> 
> --
> Thanks & Regards,

Using Solr's zkcli.sh

2020-09-01 Thread Victor Kretzer 
Thank you in advance. This is my first time using a mailing list like this so 
hopefully I am doing so correctly.

I am attempting to setup SolrCloud (Solr 6.6.6) and an external zookeeper 
ensemble on Azure. I have three dedicated to the zookeeper ensemble and two for 
solr all running Ubuntu 18.04 LTS. I've been relying on the following documents:


  *   Taking Solr to 
Production
  *   Enbabling 
SSL

I was able to complete the stand-alone portion of Enabling SSL on each of the 
solr machines and have successfully navigated to the Admin page using 
https://private.address/solr.


I am now trying to complete the section, SSL with SolrCloud, but I cannot get 
past the Configure Zookeeper section. Whenever I try to run
server/scripts/cloud-scripts/zkcli.sh it says:
it says
-bash: server/scripts/cloud-scripts/zkcli.sh: Permission denied

I've tried using sudo server/...  but then it says:
sudo: server/scripts/cloud-scripts/zkcli.sh: command not found

What am I doing wrong? Any help getting this set up would be greatly 
appreciated.

Thanks,

Victor

Re: external field file size

2020-09-01 Thread matthew sporleder 
Okay thanks for the tip.  I am pretty wary of streaming logs into my
main set of documents + tons of $stat_updated_at fields + resetting
stats on ~every document every day + whatever else we feel like
trending.  It just feels like a lot of churn.

I will lean towards the !join on stats-$DATE probably.

On Tue, Sep 1, 2020 at 11:32 AM Erick Erickson  wrote:
>
> I wouldn’t use ExternalFileField if your use-case is served by in-place 
> updates. See
>
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates
>
> EFFs were put in in order to have _some_ capability to change individual 
> fields in a doc
> long before in-place updates were around and long before SolrCloud. Using EFF 
> in any
> kind of sharded system will cause you significant heartburn in terms of 
> keeping the
> file up to date on all replicas.
>
> Best,
> Erick
>
> > On Sep 1, 2020, at 11:21 AM, matthew sporleder  wrote:
> >
> > We are researching the canonical use case for external fields --
> > traffic-based rankings
> >
> > What are the practical limits on the size of the external field file?
> > A k=v text file seems like it might fall over if it grows into the GB
> > range?
> >
> > Our other thought is to use rolling cores where we stream in web logs
> > and use !join queries.
> >
> > Does anyone have practical experience with this that they might want to 
> > share?
> >
> > Thanks,
> > Matt
>

Re: Understanding Solr heap %

2020-09-01 Thread Walter Underwood 
This is misleading and not particularly good advice.

Solr 8 does NOT contain G1. G1GC is a feature of the JVM. We’ve been using
it with Java 8 and Solr 6.6.2 for a few years.

A test with eighty documents doesn’t test anything. Try a million documents to
get Solr memory usage warmed up.

GC_TUNE has been in the solr.in.sh file for a long time. Here are the settings
we use with Java 8. We have about 120 hosts running Solr in six prod clusters.

SOLR_HEAP=8g
# Use G1 GC  -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Sep 1, 2020, at 8:39 AM, Joe Doupnik  wrote:
> 
> Erick states this correctly. To give some numbers from my experiences, 
> here are two slides from my presentation about installing Solr 
> (https://netlab1.net/ , locate item "Solr/Lucene Search 
> Service"):
>> 
> 
>> 
> 
> Thus we see a) experiments are the key, just as Erick says, and b) the 
> choice of garbage collection algorithm plays a major role.
> In my setup I assigned SOLR_HEAP to be 2048m, SOLR_OPTS has -Xss1024k, 
> plus stock GC_TUNE values. Your "memorage" may vary.
> Thanks,
> Joe D.
> 
> On 01/09/2020 15:33, Erick Erickson wrote:
>> You want to run with the smallest heap you can due to Lucene’s use of 
>> MMapDirectory, 
>> see the excellent:
>> 
>> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html 
>> 
>> 
>> There’s also little reason to have different Xms and Xmx values, that just 
>> means you’ll
>> eventually move a bunch of memory around as the heap expands, I usually set 
>> them both
>> to the same value.
>> 
>> How to determine what “the smallest heap you can” is? Unfortunately there’s 
>> no good way
>> outside of stress-testing your application with less and less memory until 
>> you have problems,
>> then add some extra…
>> 
>> Best,
>> Erick
>> 
>>> On Sep 1, 2020, at 10:27 AM, Dominique Bejean  
>>>  wrote:
>>> 
>>> Hi,
>>> 
>>> As all Java applications the Heap memory is regularly cleaned by the
>>> garbage collector (some young items moved to the old generation heap zone
>>> and unused old items removed from the old generation heap zone). This
>>> causes heap usage to continuously grow and reduce.
>>> 
>>> Regards
>>> 
>>> Dominique
>>> 
>>> 
>>> 
>>> 
>>> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar  
>>>  a
>>> écrit :
>>> 
 Can someone make me understand on how the value % on the column Heap is
 calculated.
 
 I did created a new solr cloud with 3 solr nodes and one zookeeper, its
 not yet live neither interms of indexing or searching, but I do see some
 spikes in the HEAP column against nodes when I refresh the page multiple
 times. Its like almost going to 95% (sometimes) and then coming down to 50%
 
 Solr version: 8.2
 Zookeeper: 3.4
 
 JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
 RAM size on the node is 16GB)
 
 Basically need to understand if I need to worry about this heap % which
 was quite altering before making it live? or is that quite normal, because
 this is new UI change on solr cloud is kind of new to us as we used to have
 solr 5 version before and this UI component doesn't exists then.
 
 --
 Thanks & Regards,
 Yaswanth Kumar Konathala.
 yaswanth...@gmail.com 
 
 Sent from my iPhone
>

Re: external field file size

2020-09-01 Thread Erick Erickson 
I wouldn’t use ExternalFileField if your use-case is served by in-place 
updates. See 

https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html#in-place-updates

EFFs were put in in order to have _some_ capability to change individual fields 
in a doc
long before in-place updates were around and long before SolrCloud. Using EFF 
in any
kind of sharded system will cause you significant heartburn in terms of keeping 
the
file up to date on all replicas.

Best,
Erick

> On Sep 1, 2020, at 11:21 AM, matthew sporleder  wrote:
> 
> We are researching the canonical use case for external fields --
> traffic-based rankings
> 
> What are the practical limits on the size of the external field file?
> A k=v text file seems like it might fall over if it grows into the GB
> range?
> 
> Our other thought is to use rolling cores where we stream in web logs
> and use !join queries.
> 
> Does anyone have practical experience with this that they might want to share?
> 
> Thanks,
> Matt

external field file size

2020-09-01 Thread matthew sporleder 
We are researching the canonical use case for external fields --
traffic-based rankings

What are the practical limits on the size of the external field file?
A k=v text file seems like it might fall over if it grows into the GB
range?

Our other thought is to use rolling cores where we stream in web logs
and use !join queries.

Does anyone have practical experience with this that they might want to share?

Thanks,
Matt

Re: Understanding Solr heap %

2020-09-01 Thread Erick Erickson 
You want to run with the smallest heap you can due to Lucene’s use of 
MMapDirectory, 
see the excellent:

https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

There’s also little reason to have different Xms and Xmx values, that just 
means you’ll
eventually move a bunch of memory around as the heap expands, I usually set 
them both
to the same value.

How to determine what “the smallest heap you can” is? Unfortunately there’s no 
good way
outside of stress-testing your application with less and less memory until you 
have problems,
then add some extra…

Best,
Erick

> On Sep 1, 2020, at 10:27 AM, Dominique Bejean  
> wrote:
> 
> Hi,
> 
> As all Java applications the Heap memory is regularly cleaned by the
> garbage collector (some young items moved to the old generation heap zone
> and unused old items removed from the old generation heap zone). This
> causes heap usage to continuously grow and reduce.
> 
> Regards
> 
> Dominique
> 
> 
> 
> 
> Le mar. 1 sept. 2020 à 13:50, yaswanth kumar  a
> écrit :
> 
>> Can someone make me understand on how the value % on the column Heap is
>> calculated.
>> 
>> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
>> not yet live neither interms of indexing or searching, but I do see some
>> spikes in the HEAP column against nodes when I refresh the page multiple
>> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>> 
>> Solr version: 8.2
>> Zookeeper: 3.4
>> 
>> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
>> RAM size on the node is 16GB)
>> 
>> Basically need to understand if I need to worry about this heap % which
>> was quite altering before making it live? or is that quite normal, because
>> this is new UI change on solr cloud is kind of new to us as we used to have
>> solr 5 version before and this UI component doesn't exists then.
>> 
>> --
>> Thanks & Regards,
>> Yaswanth Kumar Konathala.
>> yaswanth...@gmail.com
>> 
>> Sent from my iPhone

Re: Understanding Solr heap %

2020-09-01 Thread Dominique Bejean 
Hi,

As all Java applications the Heap memory is regularly cleaned by the
garbage collector (some young items moved to the old generation heap zone
and unused old items removed from the old generation heap zone). This
causes heap usage to continuously grow and reduce.

Regards

Dominique




Le mar. 1 sept. 2020 à 13:50, yaswanth kumar  a
écrit :

> Can someone make me understand on how the value % on the column Heap is
> calculated.
>
> I did created a new solr cloud with 3 solr nodes and one zookeeper, its
> not yet live neither interms of indexing or searching, but I do see some
> spikes in the HEAP column against nodes when I refresh the page multiple
> times. Its like almost going to 95% (sometimes) and then coming down to 50%
>
> Solr version: 8.2
> Zookeeper: 3.4
>
> JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually
> RAM size on the node is 16GB)
>
> Basically need to understand if I need to worry about this heap % which
> was quite altering before making it live? or is that quite normal, because
> this is new UI change on solr cloud is kind of new to us as we used to have
> solr 5 version before and this UI component doesn't exists then.
>
> --
> Thanks & Regards,
> Yaswanth Kumar Konathala.
> yaswanth...@gmail.com
>
> Sent from my iPhone

Shard splitting and router.field

2020-09-01 Thread Niko Himanen 
Hello,

I recently ran into a problem that documents disappear from our collections
when I split a shard. To be specific, they are not copied to new shards
made by the split command.

After some debugging I figured out that it is related to router.field we
have defined for our collections and that field happens to be not indexed.
In this case ShardSplitter (
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.5.0/solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java#L553)
here gets null terms, empty docSet is returned and documents are not copied.

This behavior is not mentioned in Collection API documentation for
router.field (
https://lucene.apache.org/solr/guide/8_6/collection-management.html#create)
or in split shard documentation (
https://lucene.apache.org/solr/guide/8_6/shard-management.html#splitshard).

This applies at least for solr 7.5.

Is this a known problem? I would think that split operation should fail in
this case and at least by minimum it should be mentioned in documentation.
What do you think?

Br,

Niko Himanen

Understanding Solr heap %

2020-09-01 Thread yaswanth kumar 
Can someone make me understand on how the value % on the column Heap is 
calculated.

I did created a new solr cloud with 3 solr nodes and one zookeeper, its not yet 
live neither interms of indexing or searching, but I do see some spikes in the 
HEAP column against nodes when I refresh the page multiple times. Its like 
almost going to 95% (sometimes) and then coming down to 50%

Solr version: 8.2 
Zookeeper: 3.4

JVM size configured in solr.in.sh is min of 1GB to max of 10GB (actually RAM 
size on the node is 16GB)

Basically need to understand if I need to worry about this heap % which was 
quite altering before making it live? or is that quite normal, because this is 
new UI change on solr cloud is kind of new to us as we used to have solr 5 
version before and this UI component doesn't exists then.

-- 
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanth...@gmail.com

Sent from my iPhone

[ANNOUNCE] Apache Solr 8.6.2 released

2020-09-01 Thread Ignacio Vera 
01 September 2020, Apache Solr™ 8.6.2 available

The Lucene PMC is pleased to announce the release of Apache Solr 8.6.2

Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search and analytics, rich document
parsing, geospatial search, extensive REST APIs as well as parallel SQL.
Solr is enterprise grade, secure and highly scalable, providing fault
tolerant distributed search and indexing, and powers the search and
navigation features of many of the world's largest internet sites.

This release contains two bug fixes. The release is available for immediate
download at:

The release is available for immediate download at:

https://lucene.apache.org/solr/downloads.html

Solr 8.6.2 Bug Fixes:

SOLR-14751: Zookeeper Admin screen not working for old ZK versions.

Solr 8.6.2 also includes 1 bugfix in the corresponding Apache Lucene
release:



Please report any feedback to the mailing lists (
https://lucene.apache.org/solr/community.html#mailing-lists-irc)

Note: The Apache Software Foundation uses an extensive mirroring network
for distributing releases. It is possible that the mirror you are using may
not have replicated the release yet. If that is the case, please try
another mirror. This also goes for Maven access.