HEY, are you using the Analytics contrib?

[David Smiley]

I wonder who is using the Analytics contrib?  Why do you use it instead of
other Solr features like the JSON Faceting module that seem to have
competing functionality.  My motivation is to ascertain if it ought to be
maintained as a 3rd party plugin/package or remain as a 1st party contrib
where Solr maintainers continue to maintain it.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

Re: What is the Best way to block certain types of queries/ query patterns in Solr?

[David Smiley]

The general assumption in deploying a search platform is that you are going
to front it with a service you write that has the search features you care
about, and only those.  Only this service or other administrative functions
should reach Solr.  Be wary of making your service so flexible to support
arbitrary parameters you pass to Solr as-is that you don't know about in
advance (i.e. use an allow-list).

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Aug 31, 2020 at 10:57 AM Mark Robinson 
wrote:

> Hi,
> I had come across a mail (Oct, 2019 one) which suggested the best way is to
> handle it before it reaches Solr. I was curious whether:-
>1. Jetty query filter can be used (came across something like
> that,, need to check)
> 2. Any new features in Solr itself (like in a request handler...or
> solrconfig, schema etc..)
>
> Thanks!
> Mark
>

Re: Error on searches containing specific character pattern

[David Smiley]

Hi,

I looked at the code at those line numbers and it seems simply impossible
that an ArrayIndexOutOfBoundsException could be thrown there because it's
guarded by a condition ensuring the array is of length 1.
https://github.com/apache/lucene-solr/blob/2752d50dd1dcf758a32dc573d02967612a2cf1ff/lucene/core/src/java/org/apache/lucene/util/QueryBuilder.java#L653

If you can reproduce this with the "techproducts" schema, please share the
complete query.  If there's a problem here, I suspect the synonyms you have
may be pertinent.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Sep 1, 2020 at 11:50 PM Andy @ BlueFusion 
wrote:

> Hi All,
>
> I have an 8.6.0 instance that is working well with one exception.
>
> It returns an error when the search term follows a pattern of numbers &
> alpha characters such as:
>
>   * 1a1 aa
>   * 1a1 1aa
>   * 1a1 11
>
> Similar patterns that don't error
>
>   * 1a1 a
>   * 1a1 1
>   * 1a11 aa
>   * 11a1 aa
>   * 1a1aa
>   * 11a11 aa
>
> The error is:
>
> |"trace":"java.lang.ArrayIndexOutOfBoundsException: 0\n\t at
> org.apache.lucene.util.QueryBuilder.newSynonymQuery(QueryBuilder.java:653)\n\t
>
> at
> org.apache.solr.parser.SolrQueryParserBase.newSynonymQuery(SolrQueryParserBase.java:617)\n\t
>
> at
> org.apache.lucene.util.QueryBuilder.analyzeGraphBoolean(QueryBuilder.java:533)\n\t
>
> at
> org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:320)\n\t
>
> at
> org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:240)\n\t
>
> at
> org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:524)\n\t
>
> at
> org.apache.solr.parser.QueryParser.newFieldQuery(QueryParser.java:62)\n\t
> at
> org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1122)\n\t
>
> at
> org.apache.solr.parser.QueryParser.MultiTerm(QueryParser.java:593)\n\t
> at org.apache.solr.parser.QueryParser.Query(QueryParser.java:142)\n\t at
> org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at
> org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at
> org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at
> org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at
> org.apache.solr.parser.QueryParser.Clause(QueryParser.java:282)\n\t at
> org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)\n\t at
> org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)\n\t
> at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:260)\n\t
>
> at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)\n\t
> at org.apache.solr.search.QParser.getQuery(QParser.java:174)\n\t at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)\n\t
>
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)\n\t
>
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\t
>
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)\n\t at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)\n\t
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)\n\t
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)\n\t
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)\n\t
>
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\t
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\t
>
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\t
>
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\t
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\t
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\t
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\t
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\t
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\t
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\t
>
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\t
>
> at
> 

SolrCloud (6.6.6) SSL Setup - Unable to create collection

[Victor Kretzer]

BACKGROUND: I'm attempting to setup SolrCloud (Solr 6.6.6) with an external 
zookeeper ensemble on Azure. I have three dedicated vms for the zookeeper 
ensemble and two for solr all running Ubuntu 18.04 LTS. I'm new to Solr (and 
Linux) and have been heavily relying on the Solr Ref Guide 6.6, most recently 
the following section on enabling ssl:



https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html



So far I have:

Installed and setup zookeeper

Installed Solr (using install_solr_service.sh script) on both vms.

Followed the steps under Basic SSL Setup, generating certificates on each of 
the nodes.

Set the cluster-wide property to https per the Configure Zookeeper section of 
SolrCloud in the document

Started both nodes and have been able to navigate to them in my browser with 
https



If I do bin/solr status I get:



Solr process 13106 running on port 8983

{

  "solr_home":"/opt/solr-6.6.6/cloud/test2",

  "version":"6.6.6 68fa249034ba8b273955f20097700dc2fbb7a800 - ishan - 
2019-03-29 09:13:13",

  "startTime":"2020-09-03T18:15:34.092Z",

  "uptime":"0 days, 0 hours, 43 minutes, 29 seconds",

  "memory":"52.7 MB (%10.7) of 490.7 MB",

  "cloud":{

"ZooKeeper":"zk1:2181,zk2:2181,zk3:2181/solr",

"liveNodes":"2",

"collections":"0"}}







THE ISSUE

When I try to create a collection using the steps outlined in the above 
document, I get the following error:



azureuser@solr-node-01-test:/opt/solr$ sudo bin/solr create -c mycollection 
-shards 2 -force



Connecting to ZooKeeper at zk1:2181,zk2:2181,zk3:2181/solr ...

INFO  - 2020-09-03 18:21:26.784; 
org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at 
zk1:2181,zk2:2181,zk3:2181/solr ready

Re-using existing configuration directory mycollection



Creating new collection 'mycollection' using command:

https://Solr1:8983/solr/admin/collections?action=CREATE=mycollection=2=1=1=mycollection



ERROR: Failed to create collection 'mycollection' due to: 
{Solr2:8983_solr=org.apache.solr.client.solrj.SolrServerException:IOException 
occured when talking to server at: https://Solr2:8983/solr}

*I've attached logs at the bottom of this email.



QUESTIONS:

What am I doing wrong and how can I fix it?

Was I right to create separate certificates on each of the nodes (one cert on 
vm1, another cert on vm 2)?

Do I need to copy the certs for each node into the other (if so how)?



CONCLUSION

Thank you so much in advance and if there's any other information you need 
please let me know.

Victor

2020-09-03 18:15:35.240 INFO  
(zkCallback-5-thread-1-processing-n:Solr1:8983_solr) [   ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={indexInfo=false=json&_=1599156956818} status=0 QTime=23
2020-09-03 18:15:40.134 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} 
status=0 QTime=29
2020-09-03 18:15:40.171 INFO  (qtp401424608-13) [   ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params 
action=LIST=json&_=1599156956818 and sendToOCPQueue=true
2020-09-03 18:15:40.172 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/collections 
params={action=LIST=json&_=1599156956818} status=0 QTime=1
2020-09-03 18:15:40.174 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} 
status=0 QTime=8
2020-09-03 18:15:58.225 INFO  (qtp401424608-14) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={indexInfo=false=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.231 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} 
status=0 QTime=7
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params 
action=LIST=json&_=1599156974989 and sendToOCPQueue=true
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/collections 
params={action=LIST=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.263 INFO  (qtp401424608-21) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} 
status=0 QTime=7
2020-09-03 18:19:38.661 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=6
2020-09-03 18:19:38.687 INFO  (qtp401424608-45) [   ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with 
params action=CLUSTERSTATUS=json and sendToOCPQueue=true
2020-09-03 18:19:38.691 INFO  

Re: NullPointerException when select query on multiple IDs after splitting a shard

[Erick Erickson]

that syntax isn’t isn’t a syntax that Solr recognize intentionally at all.
At very best, the select handler is using the default field (if it’s defined).

Although not having a “q” parameter means all bets are off. On a local copy of 
7.3
I have lying around I get a valid response, but using a MatchNoDocsQuery
(added =query to something similar) so it finds no documents. And no
NPE even with a sharded collection.

What probably happened after you split the shard was that the sub-requests were
being sent slightly differently than the input. When a collection has more than 
one
shard, a subrequest is sent out to one replica of each shard. Apparently the 
syntax
you’re using doesn’t quite survive that process unscathed.

So your single-shard happened to be working with incorrect syntax by chance.

It’d be interesting to see how the query worked, if you have a copy of your
single-shard setup working try adding =query to the url, the return
should have the parsed query which would be informative.

Best,
Erick

> On Sep 3, 2020, at 2:04 PM, Louis  wrote:
> 
> Yes, we are sure that this is not typo.
> 
> Actually we did more experiments and found that
> 
> 1) https://hostname:8983/solr/my_collection/select?ids=169455599|1
> 2) https://hostname:8983/solr/my_collection/select?q=id:169455599|1 
> 3) https://hostname:8983/solr/my_collection/get?ids=169455599|1 
> 
> 1) throws NPE, but 2) and 3) is working.
> 
> However 1) is also working with one shard in the same collection which I
> backed up before splitting a shard
> 
> It looks very weird and we are wondering about missing any config. 
> 
> Or, 1) is the type of the syntax Solr discourages to use?
> 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: NullPointerException when select query on multiple IDs after splitting a shard

[Louis]

Yes, we are sure that this is not typo.

Actually we did more experiments and found that

1) https://hostname:8983/solr/my_collection/select?ids=169455599|1
2) https://hostname:8983/solr/my_collection/select?q=id:169455599|1 
3) https://hostname:8983/solr/my_collection/get?ids=169455599|1 

1) throws NPE, but 2) and 3) is working.

However 1) is also working with one shard in the same collection which I
backed up before splitting a shard

It looks very weird and we are wondering about missing any config. 

Or, 1) is the type of the syntax Solr discourages to use?




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Cloud Query when one of the Solr instances goes down

[Jeffrey McCallum]

Hi Tyrone,
We use an external load balancer across the nodes.

If you use the java client you can query the zookeepers

https://lucene.apache.org/solr/guide/7_1/solrcloud-query-routing-and-read-tolerance.html
SolrCloud Query Routing And Read Tolerance | Apache Solr Reference Guide 
7.1
A Solr node will return the results of a search request as long as it can 
communicate with at least one replica of every shard that it knows about, even 
if it can not communicate with ZooKeeper at the time it receives the request. 
This is normally the preferred behavior from a fault tolerance standpoint, but 
may result in stale or incorrect results if there have been major changes to 
the ...
lucene.apache.org

Read Side Fault 
Tolerance

In a SolrCloud cluster each individual node load balances read requests across 
all the replicas in collection. You still need a load balancer on the 'outside' 
that talks to the cluster, or you need a smart client which understands how to 
read and interact with Solr’s metadata in ZooKeeper and only requests the 
ZooKeeper ensemble’s address to start discovering to which nodes it should send 
requests. (Solr provides a smart Java SolrJ client called 
CloudSolrClient.)

Hope it helps.






[cid:ee5928bc-8841-4ba1-9750-6051d036412b]

Jeff McCallum

Software Test / DevOps

Forensic Logic / COPLINK

Office: 520-367-2844

email: jmccal...@forensiclogic.com

www.forensiclogic.com


From: Tyrone Tse 
Sent: Thursday, September 3, 2020 10:14 AM
To: solr-user@lucene.apache.org 
Subject: Solr Cloud Query when one of the Solr instances goes down

I have setup the example Solr Cloud that comes with the built in Zoo Keeper
that runs on localhost:9993.

I created my Solr Cloud instance with 2 nodes.
Node 1 url is http://localhost:8983/solr/#/~cloud
Node 2 url is http://localhost:7574/solr/#/~cloud

Currently all Solr queries go through Node 1
http://localhost:8983/solr/files/select?q=*%3A*

Is there a way to query solr cloud collections though the Zoo Keeper or
common host:port
So that if node 1 or node 2 goes down.
The Solr query will still work, so I don't have to change the Solr
host:port in the query if one solr node goes down?

Currently in our Java application, if Solr 1 is going to brought down for
maintenance, then the our solr configuration file solr url property has to
be changed to from http://localhost:8983  to http://localhost:7574

Thanks
Tyrone

Solr Cloud Query when one of the Solr instances goes down

[Tyrone Tse]

I have setup the example Solr Cloud that comes with the built in Zoo Keeper
that runs on localhost:9993.

I created my Solr Cloud instance with 2 nodes.
Node 1 url is http://localhost:8983/solr/#/~cloud
Node 2 url is http://localhost:7574/solr/#/~cloud

Currently all Solr queries go through Node 1
http://localhost:8983/solr/files/select?q=*%3A*

Is there a way to query solr cloud collections though the Zoo Keeper or
common host:port
So that if node 1 or node 2 goes down.
The Solr query will still work, so I don't have to change the Solr
host:port in the query if one solr node goes down?

Currently in our Java application, if Solr 1 is going to brought down for
maintenance, then the our solr configuration file solr url property has to
be changed to from http://localhost:8983  to http://localhost:7574

Thanks
Tyrone

Re: NullPointerException when select query on multiple IDs after splitting a shard

[Erick Erickson]

Hmm, an NPE is weird in any case, but assuming “ids” is a field, your syntax
is wrong
q=ids:111|222

or 
q=ids: (111 222) would do too.

Are you sure you used this syntax before? Or is it a typo?
Erick

> On Sep 3, 2020, at 1:02 PM, Louis  wrote:
> 
> We are using SolrCloud 7.7.2 and having some trouble with multi shards.
> 
> The initial number of shards in the collection was one(and 3 replica), and
> we have recently split a shard into 4 shards(and 3 replica for each shard)
> by using API call :
> /admin/collections?action=SPLITSHARD=name=shardID
> 
> And now this query,
> "https://hostname:8983/solr/my_collection/select?ids=169455599|1" , is
> returning NPE, which was working before multi shards.
> 
> Could you please give any advice on why select query on multiple IDs doesn't
> work after shart splitting?
> 
> 
> null:java.lang.NullPointerException
>   at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
>   at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
>   at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
>   at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
>   at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
>   at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
>   at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
>   at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
>   at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
>   at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>   at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>   at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>   at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>   at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>   at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>   at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>   at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>   at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
>   at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>   at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>   at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>   at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>   at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at org.eclipse.jetty.server.Server.handle(Server.java:502)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
>   at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
>   at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>   at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
>   at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
>   at
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>   at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
>   at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>   at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>   at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>   at
> 

NullPointerException when select query on multiple IDs after splitting a shard

[Louis]

We are using SolrCloud 7.7.2 and having some trouble with multi shards.

The initial number of shards in the collection was one(and 3 replica), and
we have recently split a shard into 4 shards(and 3 replica for each shard)
by using API call :
/admin/collections?action=SPLITSHARD=name=shardID

And now this query,
"https://hostname:8983/solr/my_collection/select?ids=169455599|1" , is
returning NPE, which was working before multi shards.

Could you please give any advice on why select query on multiple IDs doesn't
work after shart splitting?


null:java.lang.NullPointerException
at
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)
at
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at 

Query performance degrades when TLOG replica

[Ankit Shah]

We have the following setup , solr 7.7.2 with 1 TLOG Leader & 1 TLOG
replica with a single shard. We have about  34.5 million documents with an
approximate index size of 600GB. I have noticed a degraded query
performance whenever the replica is trying to (guessing here) sync or
perform actual replication. To test this, I fire a very basic query using
solrj client & the query comes back right away, but whenever the
replication is trying to see how far behind it is by comparing the
generation ids the same queries take longer. In production we do not make
these simple queries, but rather complex queries with filter queries &
sorting. These queries take too long as compared to our previous
(standalone solr 6.1.0)

Any help here is appreciated

20-09-02 16:35:30 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:35:30 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:00 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:00 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:30 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
2020-09-02 16:36:30 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458909
status=0 QTime=0
*2020-09-02 16:37:01* INFO  [db_shard1_replica_t3]  webapp=/solr
path=/select params={q=*:*=id=id+desc=1=xml=2.2}
hits=34458909 status=0 QTime=*1011*
*2020-09-02 16:37:01* INFO  [db_shard1_replica_t3]  webapp=/solr
path=/select params={q=*:*=id=id+desc=1=xml=2.2}
hits=34458909 status=0 QTime=*758*
*2020-09-02 16:37:32* INFO  [db_shard1_replica_t3]  webapp=/solr
path=/select params={q=*:*=id=id+desc=1=xml=2.2}
hits=34458957 status=0 QTime=*1077*
*2020-09-02 16:37:32* INFO  [db_shard1_replica_t3]  webapp=/solr
path=/select params={q=*:*=id=id+desc=1=xml=2.2}
hits=34458957 status=0 QTime=*1081*
2020-09-02 16:38:02 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458957
status=0 QTime=*668*
2020-09-02 16:38:03 INFO  [db_shard1_replica_t3]  webapp=/solr path=/select
params={q=*:*=id=id+desc=1=xml=2.2} hits=34458957
status=0 QTime=*1001*


*2020-09-02 16:37:01* INFO  Master's generation: 263116
*2020-09-02 16:37:01* INFO  Master's version: 1599064577322
*2020-09-02 16:37:01* INFO  Slave's generation: 263116
*2020-09-02 16:37:01* INFO  Slave's version: 1599064577322
*2020-09-02 16:37:01* INFO  Slave in sync with master.
2020-09-02 16:37:02 INFO  Master's generation: 104189
2020-09-02 16:37:02 INFO  Master's version: 1599064620532
2020-09-02 16:37:02 INFO  Slave's generation: 104188
2020-09-02 16:37:02 INFO  Slave's version: 1599064560341
2020-09-02 16:37:02 INFO  Starting replication process
2020-09-02 16:37:02 INFO  Number of files in latest index in master: 1010
2020-09-02 16:37:02 INFO  Starting download (fullCopy=false) to
NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2020-09-02 16:37:02 INFO  Bytes downloaded: 837587, Bytes skipped
downloading: 0
2020-09-02 16:37:02 INFO  Total time taken for download
(fullCopy=false,bytesDownloaded=837587) : 0 secs (null bytes/sec) to
NRTCachingDirectory(MMapDirectory@/opt/solr-7.7.2/server/solr/test_shard1_replica_t3/data/index.20200902163702345
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77247ee;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
2020-09-02 16:37:03 INFO  New IndexWriter is ready to be used.
2020-09-02 16:37:03 INFO  Master's generation: 124002
2020-09-02 16:37:03 INFO  Master's version: 1599064617242
2020-09-02 16:37:03 INFO  Slave's generation: 124000
2020-09-02 16:37:03 INFO  Slave's version: 1599064492914
2020-09-02 16:37:03 INFO  Starting replication process
2020-09-02 16:37:04 INFO  [db_shard1_replica_t3]  webapp=/solr path=/update
params={update.distrib=FROMLEADER=
http://178.33.234.1:8983/solr/db_shard1_replica_t25/=javabin=2}{add=[11907382419
(1676740784884285440), 11907383701 (1676740784889528320), 11907383253
(1676740784900014080), 11907379290 (1676740785002774528), 11907382623
(1676740785005920256), 11907378461 (1676740785011163136), 11907382429
(1676740785012211712), 11907380739 (1676740785023746048), 11907381184
(1676740785038426112), 11907380614 (1676740785042620416), ... (36 adds)]} 0
2227
2020-09-02 16:37:04 INFO  Number of files in latest index in master: 1227
2020-09-02 16:37:04 INFO  [db_shard1_replica_t3]  webapp=/solr path=/update
params={update.distrib=FROMLEADER=true=true=true=false=http://

Re: BasicAuth help

[Jason Gerlowski]

Hi Ali,

1. Solr doesn't have any support for LDAP authentication ootb (at
least, as far as I'm aware).  The BasicAuth plugin requires users to
be defined in the JSON configuration.

2. What failed when you ran the documented BasicAuth example?  What
error messages did you get etc.?  If there's something wrong with that
example, maybe we can fix the docs.

Jason

On Fri, Aug 28, 2020 at 3:28 PM Vanalli, Ali A - DOT
 wrote:
>
> Hello,
>
> Solr is running on windows machine and wondering if it possible to setup 
> BasicAuth with the LDAP?
>
> Also, tried the example of Basic-Authentication that is published 
> here
>  but this did not work too.
>
> Thanks...Ali
>
>

Re: Replication in soft commit

[Emir Arnautović]

Hi Tushar,
This is not usecase suitable for MS model. You should go with Solr Cloud, or if 
that is an overhead for you, have separate Solr, each doing indexing on its 
own. Solr provides eventual consistency anyway, so you should have some sort of 
stickiness in place even if you use MS model.

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 3 Sep 2020, at 13:54, Tushar Arora  wrote:
> 
> Hi Emir,
> Thanks for the response.
> Actually the use case is real time indexing from DB to solr in every second
> on the master server using queueing mechanism.
> So, I think instead of doing hard commits every second we should go for
> soft commits. And doing hard commits after some intervals.
> And we have to replicate the data to slave immediately.
> 
> Regards,
> Tushar
> On Thu, 3 Sep 2020 at 16:17, Emir Arnautović 
> wrote:
> 
>> Hi Tushar,
>> Replication is file based process and hard commit is when segment is
>> flushed to disk. It is not common that you use soft commits on master. The
>> only usecase that I can think of is when you read your index as part of
>> indexing process, but even that is bad practice and should be avoided.
>> 
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 
>> 
>> 
>>> On 3 Sep 2020, at 08:38, Tushar Arora  wrote:
>>> 
>>> Hi,
>>> I want to ask if the soft commit works in replication.
>>> One of our use cases deals with indexing the data every second on a
>> master
>>> server. And then it has to replicate to slaves. So if we use soft commit,
>>> then does the data replicate immediately to the slave server or after the
>>> hard commit takes place.
>>> Use cases require transfer of data from master to slave immediately.
>>> 
>>> Regards,
>>> Tushar
>> 
>> 

Re: Replication in soft commit

[Tushar Arora]

Hi Emir,
Thanks for the response.
Actually the use case is real time indexing from DB to solr in every second
on the master server using queueing mechanism.
So, I think instead of doing hard commits every second we should go for
soft commits. And doing hard commits after some intervals.
And we have to replicate the data to slave immediately.

Regards,
Tushar
On Thu, 3 Sep 2020 at 16:17, Emir Arnautović 
wrote:

> Hi Tushar,
> Replication is file based process and hard commit is when segment is
> flushed to disk. It is not common that you use soft commits on master. The
> only usecase that I can think of is when you read your index as part of
> indexing process, but even that is bad practice and should be avoided.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 3 Sep 2020, at 08:38, Tushar Arora  wrote:
> >
> > Hi,
> > I want to ask if the soft commit works in replication.
> > One of our use cases deals with indexing the data every second on a
> master
> > server. And then it has to replicate to slaves. So if we use soft commit,
> > then does the data replicate immediately to the slave server or after the
> > hard commit takes place.
> > Use cases require transfer of data from master to slave immediately.
> >
> > Regards,
> > Tushar
>
>

Re: Replication in soft commit

[Emir Arnautović]

Hi Tushar,
Replication is file based process and hard commit is when segment is flushed to 
disk. It is not common that you use soft commits on master. The only usecase 
that I can think of is when you read your index as part of indexing process, 
but even that is bad practice and should be avoided.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 3 Sep 2020, at 08:38, Tushar Arora  wrote:
> 
> Hi,
> I want to ask if the soft commit works in replication.
> One of our use cases deals with indexing the data every second on a master
> server. And then it has to replicate to slaves. So if we use soft commit,
> then does the data replicate immediately to the slave server or after the
> hard commit takes place.
> Use cases require transfer of data from master to slave immediately.
> 
> Regards,
> Tushar

Replication in soft commit

[Tushar Arora]

Hi,
I want to ask if the soft commit works in replication.
One of our use cases deals with indexing the data every second on a master
server. And then it has to replicate to slaves. So if we use soft commit,
then does the data replicate immediately to the slave server or after the
hard commit takes place.
Use cases require transfer of data from master to slave immediately.

Regards,
Tushar