How do I add my own Streaming Expressions?

[Eric Pugh]

What is the process for adding new Streaming Expressions?   

It appears that the org.apache.solr.client.solrj.io.Lang method statically 
loads all the streaming expressions?

Eric

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Upgrade solr from 7.2.1 to 8.2

[Rahul Goswami]

Hello,

We are planning to upgrade our SolrCloud cluster from 7.2.1 (hosted on
Windows server) to 8.2.
I read the documentation

which mentions that I need to be on Solr 7.3 and higher to be able to
upgrade to 8.x. I want to know if this is a hard requirement or a
requirement for rolling upgrades (?).
Let's say I am fine with bringing the whole cluster down and upgrade all
the nodes to Solr 8.2, and then bring up one node at a time. Will it be ok
to upgrade directly from 7.2.1 to 8.2 in that case?

Thanks in advance!

Regards,
Rahul

Re: $deleteDocByQuery and $deleteDocByID

[Dominique Bejean]

Hi Paresh,

Due to deleteDocByQuery impact on commits and searcher reopen, if a lot of
deletions are done it is preferable when possible to use deletebyid .

Regards

Dominique

Le mar. 12 nov. 2019 à 07:03, Paresh  a écrit :

> Hi Erik,
>
> I am also looking for some example of deleteDocByQuery. Here is my
> requirement -
>
> I want to do the database query and get the list of values of which
> matching
> documents should be deleted from Solr.
>
> I want to delete the docs which matches following query
> SolrColumnName:
>
> This  will come from query executed on RDBMS -
> select columnName from Table where state = 'deleted'
>
> This columnName is the value populated in Solr for SolrColulmnName.
>
> Regards,
> Paresh
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Fetch parent and child document in solr 8.2

[Gajjar, Jigar]

Hello,

I am trying to fetch parent and child document together in one Solr query, I 
was able to do that in solr 7.4 but same query does not work in solr 8.2.
Are there any major changes in the way that we are fetching children?

My requirement is to fetch parent and children both in one call.

I am trying

http://localhost:8983/solr/demo/select?fl=*,[child]={!parent 
which="cat_s:sci-fi AND pubyear_i:1992"}

what are the ways to retrieve parent child as nested documents?



Thanks,
Jigar Gajjar
OCLC · Senior Software  Engineer
6565 Kilgour Place, Dublin, OH, USA, 43017
 M +1-408-334-6379
[OCLC]
OCLC.org · 
Blog · 
Facebook · 
Twitter · YouTube

attempting to get an exact match on a textField

[rhys J]

I am trying to use the API to get an exact match on clt_ref_no.

At one point, I was using ""s to enclose the text such as:

clt_ref_no: "OWL-2924-8", and I was getting 5 results. Which is accurate.

Now when I use it, I only get one match.

If I try to build the url in perl, and then post the url, my response is
this:

http://localhost:8983/solr/debt/select?indent=on=1000=id%20asc=(%20clt_ref_no:%22OWL-2924-8%E2%80%9D%20OR%20contract_number:%22OWL-2924-8%22%20)={!join%20from=debtor_id%20to=debt_id%20fromIndex=dbtr}=*=true


Breaking that down, I've got:

q=( clt_ref_no: "OWL-2924-8" OR contract_number: "OWL-2924-8" )
fq= {!join from=debtor_id to=debt_id fromIndex=dbtr}

"error":{
"trace":"java.lang.NullPointerException\n\tat
org.apache.solr.search.JoinQuery.hashCode(JoinQParserPlugin.java:584)\n\tat
java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)\n\tat
org.apache.solr.util.ConcurrentLRUCache.get(ConcurrentLRUCache.java:130)\n\tat
org.apache.solr.search.FastLRUCache.get(FastLRUCache.java:165)\n\tat
org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:815)\n\tat
org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1026)\n\tat
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1541)\n\tat
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1421)\n\tat
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:568)\n\tat
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1484)\n\tat
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:398)\n\tat
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:305)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat

Re: When does Solr write in Zookeeper ?

[Dominique Bejean]

Thank you Erick for this fast answer
Why is it a best practice to set the zookeeper  connection timeout to 3
instead the default 15000 value?

Regards

Dominique

Le ven. 15 nov. 2019 à 18:36, Erick Erickson  a
écrit :

> Dominique:
>
> In a word, “yes”. You’ve got it. A common misunderstanding is that ZK is
> actively involved in queries/updates/whatever. Basically, what ZK is
> responsible for is maintaining collection-wide resources, i.e. the current
> state of all the replicas, config files, etc., your “global configuration"
> and "collection configuration”, which should change very rarely thus rarely
> generate writes.
>
> The “collection state” (including your “nodes state”) information changes
> more frequently and generates more writes as nodes come up and down, go
> into recovery, etc. That said, for a cluster where all the replicas are
> “active” and don’t go away or go into recovery etc, ZK won’t do any writes.
>
> So the consequence is that when you power up a cluster, there will be a
> flurry of write operations managed by the Overseer, but after all the
> replicas are up, write activity should pretty much cease.
>
> As long as the state is steady, i.e. no replicas changing state, each
> individual Solr node has a copy of the relevant collection’s “state.json”
> znode and has all the information it needs to query or index without asking
> Zookeeper without _either_ reading or writing to ZK.
>
> One rather obscure cause for ZK writes is when using “schemaless” mode.
> When a new field is detected, the schema (and thus the collection’s
> configuration) is changed, which generates writes..
>
> Best,
> Erick
>
>
> > On Nov 15, 2019, at 12:06 PM, Dominique Bejean <
> dominique.bej...@eolya.fr> wrote:
> >
> > Hi,
> >
> > I would like to be certain to understand how Solr use Zookeeper and more
> > precisely when Solr write into Zookeeper.
> >
> > Solr stores various informations in ZK
> >
> >   - globale configuration (autoscaling, security.json)
> >   - collection configuration (configs)
> >   - collections state (state.json, leaders, ...)
> >   - nodes state (live_nodes, overseer)
> >
> >
> > Writes in Zk occur when
> >
> >   - a zookeeper member start or stop
> >   - a solr node start or stop
> >   - a configuration is loaded
> >   - a collection is created, deleted or updated (nearly all call to
> >   collection, core or config API)
> >
> >
> > Write do not occur during
> >
> >   - SolrJ client creation
> >   - indexing data (Solrj, HTTP, DIH, ...)
> >   - searching (Solrj, HTTP)
> >
> >
> > In conclusion, if Solr nodes are stable (no failure, no maintenance), no
> > calls to  collection, core or config API are done, so there is nearly no
> > writes to ZK.
> >
> > Is it correct ?
> >
> >
> > Regards
> >
> > Dominique
>
>

Re: using NOT or - to exclude results with a textField type

[Shawn Heisey]

On 11/15/2019 10:15 AM, rhys J wrote:

I'm trying to exclude results based on the documentation about the boolean
NOT symbol, but I keep getting errors.

I've tried:

http://localhost:8983/solr/debt/select?q=clt_ref_no:-”owl-2924-8”

and

http://localhost:8983/solr/debt/select?q=clt_ref_no:NOT”owl-2924-8”

I have tried with and without quotes too.

Am I not able to use the NOT with a textField?


You've got the NOT keyword and the - character misplaced.

q=*:* AND NOT clt_ref_no:owl-2924-8
q=*:* -clt_ref_no:owl-2924-8

In a URL, use a + sign in place of a space.

In a simple case like this, the purely negative query can be detected by 
Solr and fixed internally, but for the most part, purely negative 
queries do not work.  You have to start with something and then 
subtract, you can't subtract from nothing.  That is why my examples have 
the all docs (*:*) starting point.


The errors are a little strange, but I'm guessing the misplaced 
characters are resulting in something the query parser doesn't know how 
to handle.


Thanks,
Shawn

Re: When does Solr write in Zookeeper ?

[Erick Erickson]

Dominique:

In a word, “yes”. You’ve got it. A common misunderstanding is that ZK is 
actively involved in queries/updates/whatever. Basically, what ZK is 
responsible for is maintaining collection-wide resources, i.e. the current 
state of all the replicas, config files, etc., your “global configuration" and 
"collection configuration”, which should change very rarely thus rarely 
generate writes.

The “collection state” (including your “nodes state”) information changes more 
frequently and generates more writes as nodes come up and down, go into 
recovery, etc. That said, for a cluster where all the replicas are “active” and 
don’t go away or go into recovery etc, ZK won’t do any writes.

So the consequence is that when you power up a cluster, there will be a flurry 
of write operations managed by the Overseer, but after all the replicas are up, 
write activity should pretty much cease.

As long as the state is steady, i.e. no replicas changing state, each 
individual Solr node has a copy of the relevant collection’s “state.json” znode 
and has all the information it needs to query or index without asking Zookeeper 
without _either_ reading or writing to ZK.

One rather obscure cause for ZK writes is when using “schemaless” mode. When a 
new field is detected, the schema (and thus the collection’s configuration) is 
changed, which generates writes..

Best,
Erick


> On Nov 15, 2019, at 12:06 PM, Dominique Bejean  
> wrote:
> 
> Hi,
> 
> I would like to be certain to understand how Solr use Zookeeper and more
> precisely when Solr write into Zookeeper.
> 
> Solr stores various informations in ZK
> 
>   - globale configuration (autoscaling, security.json)
>   - collection configuration (configs)
>   - collections state (state.json, leaders, ...)
>   - nodes state (live_nodes, overseer)
> 
> 
> Writes in Zk occur when
> 
>   - a zookeeper member start or stop
>   - a solr node start or stop
>   - a configuration is loaded
>   - a collection is created, deleted or updated (nearly all call to
>   collection, core or config API)
> 
> 
> Write do not occur during
> 
>   - SolrJ client creation
>   - indexing data (Solrj, HTTP, DIH, ...)
>   - searching (Solrj, HTTP)
> 
> 
> In conclusion, if Solr nodes are stable (no failure, no maintenance), no
> calls to  collection, core or config API are done, so there is nearly no
> writes to ZK.
> 
> Is it correct ?
> 
> 
> Regards
> 
> Dominique

Re: using NOT or - to exclude results with a textField type

[Erick Erickson]

On of Solr’s little quirks… Solr does _not_ implement Boolean logic. See: 
https://lucidworks.com/post/why-not-and-or-and-not/

What you want is something like:
../select/q=*:* -clt_ref_no:“owl-2924-8”

Best,
Erick

> On Nov 15, 2019, at 12:15 PM, rhys J  wrote:
> 
> I'm trying to exclude results based on the documentation about the boolean
> NOT symbol, but I keep getting errors.
> 
> I've tried:
> 
> http://localhost:8983/solr/debt/select?q=clt_ref_no:-”owl-2924-8”
> 
> and
> 
> http://localhost:8983/solr/debt/select?q=clt_ref_no:NOT”owl-2924-8”
> 
> I have tried with and without quotes too.
> 
> Am I not able to use the NOT with a textField?
> 
> Here are the errors I get from the browser:
> 
> "msg":"org.apache.solr.search.SyntaxError: Cannot parse
> 'clt_ref_no:-”=owl-2924-8”': Encountered \" \"-\" \"- \"\" at line 1,
> column 11.\nWas expecting one of:\n ...\n\"(\" ...\n
>   \"*\" ...\n ...\n ...\n ...\n
>...\n ...\n\"[\" ...\n\"{\"
> ...\n ...\n\"filter(\" ...\n ...\n",
> 
> Thanks,
> 
> Rhys

using NOT or - to exclude results with a textField type

[rhys J]

I'm trying to exclude results based on the documentation about the boolean
NOT symbol, but I keep getting errors.

I've tried:

http://localhost:8983/solr/debt/select?q=clt_ref_no:-”owl-2924-8”

and

http://localhost:8983/solr/debt/select?q=clt_ref_no:NOT”owl-2924-8”

I have tried with and without quotes too.

Am I not able to use the NOT with a textField?

Here are the errors I get from the browser:

"msg":"org.apache.solr.search.SyntaxError: Cannot parse
'clt_ref_no:-”=owl-2924-8”': Encountered \" \"-\" \"- \"\" at line 1,
column 11.\nWas expecting one of:\n ...\n\"(\" ...\n
   \"*\" ...\n ...\n ...\n ...\n
...\n ...\n\"[\" ...\n\"{\"
...\n ...\n\"filter(\" ...\n ...\n",

Thanks,

Rhys

When does Solr write in Zookeeper ?

[Dominique Bejean]

Hi,

I would like to be certain to understand how Solr use Zookeeper and more
precisely when Solr write into Zookeeper.

Solr stores various informations in ZK

   - globale configuration (autoscaling, security.json)
   - collection configuration (configs)
   - collections state (state.json, leaders, ...)
   - nodes state (live_nodes, overseer)


Writes in Zk occur when

   - a zookeeper member start or stop
   - a solr node start or stop
   - a configuration is loaded
   - a collection is created, deleted or updated (nearly all call to
   collection, core or config API)


Write do not occur during

   - SolrJ client creation
   - indexing data (Solrj, HTTP, DIH, ...)
   - searching (Solrj, HTTP)


In conclusion, if Solr nodes are stable (no failure, no maintenance), no
calls to  collection, core or config API are done, so there is nearly no
writes to ZK.

Is it correct ?


Regards

Dominique

Re: 2 Async exceptions during distributed update

[Fiz N]

Thanks for your response.





attaching the screenshot of physical memory and cpu.

Please let me know your thoughts on the below issue.



On Fri, Nov 15, 2019 at 2:18 AM Jörn Franke  wrote:

> Do you use a updateprocess factory? How does it look like?
>
> What is the physical memory size and CPU?
> What do you mean by “there are 64 cores sending concurrently?” An
> application has 64 threats that send concurrently those updates?
>
> > Am 15.11.2019 um 02:14 schrieb Fiz N :
> >
> > Hi Solr Experts,
> >
> > SOLR Version : 6.6.2
> >
> > OS – Linux 3.1.2
> >
> > JDK – 1.8
> >
> >
> >
> > Shard – 16 – All are active.
> >
> > Xms – 16 gb
> >
> > Xmx – 16 gb
> >
> >
> >
> > Schema fields count – 91
> >
> > Dynamic fields – 83.
> >
> >
> >
> > There are multiple threads sending products to solr. tested with batch
> size
> > per thread 50 & 100, it worked fine without error and if the batch size
> is
> > 1000 the following error occurs.
> >
> > there are 64 cores that are sending batches concurrently.
> >
> >
> >
> > *I am getting the following error when the batch size is 1000. Please
> > advise.*
> >
> >
> >
> > 2019-11-14T17:45:44,646 - ERROR [qtp876213901-1169:SolrException@YYY] -
> > {collection=c:ducts, core=x:ducts_shard15_replica1,
> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node30,
> shard=s:shard15}
> > -
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > 2 Async exceptions during distributed update:
> >
> > 10.YYY.40.62:8983 failed to respond
> >
> > 10.YYY.40.62:8983 failed to respond
> >
> >
> >
> > 2019-11-14T17:45:48,821 - ERROR [qtp876213901-202:SolrException@YYY] -
> > {collection=c:ducts, core=x:ducts_shard7_replica1,
> > node_name=n:10.YYY.40.81:8983_solr, replica=r:core_node29,
> shard=s:shard7}
> > -
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > Async exception during distributed update: 10.YYY.40.81:8983 failed to
> > respond
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:11,599 - ERROR
> > [updateExecutor-2-thread-176-processing-x:ducts_shard3_replica2
> > http:10.YYY.40.68:8983//solr//ducts_shard11_replica2
> >  r:core_node26
> > n:10.YYY.40.68:8983_solr c:ducts
> >
> s:shard3:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> > - {ConcurrentUpdateSolrClient.url=
> > http://10.YYY.40.68:8983/solr/ducts_shard11_replica2
> > ,
> > collection=c:ducts, core=x:ducts_shard3_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node26,
> shard=s:shard3}
> > - error
> >
> > org.apache.http.NoHttpResponseException: 10.YYY.40.68:8983 failed to
> respond
> >
> >at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> > ~[httpclient-4.4.1.jar:4.4.1]
> >
> >at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> > ~[httpclient-4.4.1.jar:4.4.1]
> >
> >at
> > org.apache.http.impl.io
> .AbstractMessageParser.parse(AbstractMessageParser.java:261)
> > ~[httpcore-4.4.1.jar:4.4.1]
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:14,567 - ERROR
> > [updateExecutor-2-thread-189-processing-x:ducts_shard11_replica2
> > http:10.YYY.40.62:8983//solr//ducts_shard2_replica1
> >  r:core_node25
> > n:10.YYY.40.68:8983_solr c:ducts
> >
> s:shard11:StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient@131
> ]
> > - {ConcurrentUpdateSolrClient.url=
> > http://10.YYY.40.62:8983/solr/ducts_shard2_replica1
> > ,
> > collection=c:ducts, core=x:ducts_shard11_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> > - error
> >
> > java.net.SocketException: Broken pipe (Write failed)
> >
> >at java.net.SocketOutputStream.socketWrite0(Native Method)
> > ~[?:1.8.0_232]
> >
> >at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> > ~[?:1.8.0_232]
> >
> >at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > ~[?:1.8.0_232]
> >
> >
> >
> >
> >
> > 2019-11-14T19:36:38,851 - ERROR [qtp876213901-542:SolrException@159] -
> > {collection=c:ducts, core=x:ducts_shard11_replica2,
> > node_name=n:10.YYY.40.68:8983_solr, replica=r:core_node25,
> shard=s:shard11}
> > -
> >
> null:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
> > Async exception during distributed update: Broken pipe (Write failed)
> >
> >   at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:972)
> >
> >at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1911)
> >
> >
> >
> >
> > Thanks
> >
> > Fiz..
>

Possible bug in cluster status - > solr 8.3

[Jacek Kikiewicz]

I found interesting situation, I've created a collection with only one replica.
Then I scaled solr-cloud cluster, and run  'addreplica' call to add 2 more.
So I have a collection with 3 tlog replicas, cluster status page shows
them but shows also this:
  "core_node2":{
"core":"EDITED_NAME_shard1_replica_t1",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false",
"leader":"true"},
  "core_node5":{
"core":"EDITED_NAME_shard1_replica_t3",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false"},
  "core_node6":{
"core":"EDITED_NAME_shard1_replica_t4",
"base_url":"http://EDITED_NODE:8983/solr;,
"node_name":"EDITED_NODE:8983_solr",
"state":"active",
"type":"TLOG",
"force_set_state":"false",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"1",
"znodeVersion":11,


As you can see I have 3 replicas but then I have also: "tlogReplicas":"1"

If I create collection with tlogReplicas=3 then cluster status shows
"tlogReplicas":"3"
IS that a bug or somehow 'works as it should' ?

Regards,
Jacek

Re: Solr process taking a long time before taking commands after restart

[Jörn Franke]

Do you have some suggester or so that is possible automatically rebuilding 
after restore?

> Am 15.11.2019 um 16:36 schrieb Koen De Groote :
> 
> Greetings all,
> 
> I was testing some backup/restore scenarios.
> 
> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> SolrCloud, with zookeeper. Deleting the data folder on the host, restarting
> the container, resetting the data folder and restoring.
> 
> It all works fine, except it took about 10 minutes before Solr responded to
> a simple CLUSTERSTATUS command, a healthcheck or even load the admin UI.
> 
> We're talking about 20GB of data here, so not very much, but not little
> either.
> 
> Once responsive, the restore happened just fine.
> 
> Is it normal that it takes so long before solr responds? If not, what
> should I look at in order to find the cause?
> 
> Kind regards,
> Koen De Groote

Re: Solr process taking a long time before taking commands after restart

[Erick Erickson]

It’d be interesting to see a thread dump during that time, sounds like 
everything’s blocked until the restore is done, although 10 minutes for 20G 
seems excessive.

Best,
Erick

> On Nov 15, 2019, at 10:35 AM, Koen De Groote  
> wrote:
> 
> Greetings all,
> 
> I was testing some backup/restore scenarios.
> 
> 1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
> SolrCloud, with zookeeper. Deleting the data folder on the host, restarting
> the container, resetting the data folder and restoring.
> 
> It all works fine, except it took about 10 minutes before Solr responded to
> a simple CLUSTERSTATUS command, a healthcheck or even load the admin UI.
> 
> We're talking about 20GB of data here, so not very much, but not little
> either.
> 
> Once responsive, the restore happened just fine.
> 
> Is it normal that it takes so long before solr responds? If not, what
> should I look at in order to find the cause?
> 
> Kind regards,
> Koen De Groote

Solr process taking a long time before taking commands after restart

[Koen De Groote]

Greetings all,

I was testing some backup/restore scenarios.

1 of them is Solr7.6 in a docker container(7.6.0-slim), set up as
SolrCloud, with zookeeper. Deleting the data folder on the host, restarting
the container, resetting the data folder and restoring.

It all works fine, except it took about 10 minutes before Solr responded to
a simple CLUSTERSTATUS command, a healthcheck or even load the admin UI.

We're talking about 20GB of data here, so not very much, but not little
either.

Once responsive, the restore happened just fine.

Is it normal that it takes so long before solr responds? If not, what
should I look at in order to find the cause?

Kind regards,
Koen De Groote

Re: daily SolrCloud collection wipes

[Erick Erickson]

Werner:

Who is sending the delete-by-query? What I’m really wondering is if it’s done 
by something internal to Solr (in which case I’d like to track it down) or 
something outside Solr in which case we don’t need to be concerned...

Thanks,
Erick

> On Nov 15, 2019, at 1:38 AM, Werner Detter  wrote:
> 
> Hi, 
> 
>> Currently logging is disabled due to performance on the live setup. But 
>> tonight, bevor 
>> this happens, we'll enable logging an we'll hopefully see something to track 
>> down the 
>> source for the documents deletion in the collection. 
> 
> tcpdump and the logs revealed the source of the problem: the collections were 
> indeed 
> cleared by a delete *:* query. 
> 
> Cheers,
> Werner
> 

Re: NPE during spell checking when result collapsing is activated and local parameters are used

[Tomás Fernández Löbbe]

Would you create a Jira issue anyway tu fix the fact that it NPE instead of
throwing a bad request?

On Fri, Nov 15, 2019 at 2:31 AM Stefan Walter  wrote:

> Indeed, you are right. Interestingly, it generally worked with the two {!
> ..} in the filter query - besides the problem with the collations, of
> course. Therefore I never questioned it...
>
> Thank you!
> Stefan
>
>
> Am 15. November 2019 um 00:01:52, Tomás Fernández Löbbe (
> tomasflo...@gmail.com) schrieb:
>
> I believe your syntax is incorrect. I believe local params must all be
> included in between the same {!...}, and "{!" can only be at the beginning
>
> have you tried:
>
> ={!collapse tag=collapser field=productId sort='merchantOrder asc,
> price asc, id asc'}
>
>
>
> On Thu, Nov 14, 2019 at 4:54 AM Stefan Walter  wrote:
>
> > Hi!
> >
> > I have an issue with Solr 7.3.1 in the spell checking component:
> >
> > java.lang.NullPointerException at
> >
> >
>
> org.apache.solr.search.CollapsingQParserPlugin$OrdFieldValueCollector.finish(CollapsingQParserPlugin.java:1021)
>
> > at
> >
> >
>
> org.apache.solr.search.CollapsingQParserPlugin$OrdFieldValueCollector.finish(CollapsingQParserPlugin.java:1081)
>
> > at
> >
> >
>
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:230)
>
> > at
> >
> >
>
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)
>
> > at
> >
> >
>
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)
>
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)
> > ...
> >
> > I have found an issue that addresses a similiar problem:
> > https://issues.apache.org/jira/browse/SOLR-8807
> >
> > The fix, which was introduced with this issue seems to miss our
> situation,
> > though. The relevant part of the query is this:
> >
> > ={!tag=collapser}{!collapse field=productId sort='merchantOrder asc,
> > price asc, id asc'}
> >
> > When I remove the local parameter {!tag=collapser} the collation works
> > fine. Looking at the diff of the commit of the issue mentioned above, it
> > seems that the "startsWith" could be the problem:
> >
> > + // Collate testing does not support the Collapse QParser (See
> > SOLR-8807)
> > + params.remove("expand");
> > + String[] filters = params.getParams(CommonParams.FQ);
> > + if (filters != null) {
> > + List filtersToApply = new ArrayList<>(filters.length);
> > + for (String fq : filters) {
> > + if (!fq.startsWith("{!collapse")) {
> > + filtersToApply.add(fq);
> > + }
> > + }
> > + params.set("fq", filtersToApply.toArray(new
> > String[filtersToApply.size()]));
> > + }
> >
> > Can someone confirm this? I would open a bug ticket then. (Since the code
> > is unchanged in the latest version.)
> >
> > Thanks,
> > Stefan
> >
>

RE: Anyway to encrypt admin user plain text password in Solr

[Kommu, Vinodh K.]

Thanks Jason for sharing the details.


Regards,
Vinodh

-Original Message-
From: Jason Gerlowski  
Sent: Friday, November 15, 2019 12:54 AM
To: solr-user@lucene.apache.org
Subject: Re: Anyway to encrypt admin user plain text password in Solr

ATTENTION! This email originated outside of DTCC; exercise caution.

Hi Vinodh,

I don't know of any way to encrypt the credentials in "basicAuth.conf", and 
looking at the code that loads that file I don't see any logic to handle that 
sort of thing.  So I'm near positive there's no way to avoid plaintext here.

But, that said, I don't think this should really be that concerning.
To read this file, an adversary would need (1) access to the machine Solr runs 
on and (2) access to the user/group running Solr.  If an adversary has those 
two things, the credentials are besides the point.
They could kill your Solr process directly.  Or read the index data files 
directly.  Or edit them. etc.  There may be edge cases around using network 
drives or HDFS where encrypting this file is useful, I haven't thought that 
side of things through entirely.  But for most use-cases I'm not sure 
encrypting basicAuth.conf provides anything beyond security theater.

Best,

Jason



On Thu, Nov 14, 2019 at 9:49 AM Mark H. Wood  wrote:
>
> On Thu, Nov 14, 2019 at 11:35:47AM +, Kommu, Vinodh K. wrote:
> > We store the plain text password in basicAuth.conf file. This is a normal 
> > file & we are securing it only with 600 file permissions so that others 
> > cannot read it. We also run various solr APIs in our custom script for 
> > various purposes using curl commands which needs admin user credentials to 
> > perform operations. If admin credentials details from basicAuth.conf file 
> > or from curl commands are exposed/compromised, eventually any person within 
> > the organization who knows credentials can login to admin UI and perform 
> > any read/write operations. This is a concern and auditing issue as well.
>
> If the password is encrypted, then the decryption key must be supplied 
> before the password can be used.  This leads to one of two unfortunate
> situations:
>
> o  The user must enter the decryption key every time.  This defeats
>the purpose of storing credentials at the client.
>
>- or -
>
> o  The decryption key is stored at the client, making it a new secret
>that must be protected (by encrypting it? you see where this is
>going)
>
> There is no way around this.  If the client system stores a full set 
> of credentials, then anyone with sufficient access to the client 
> system can get everything he needs to authenticate an identity, no 
> matter what you do.  If the client system does not store a full set of 
> credentials, then the user must supply at least some of them whenever 
> they are needed.  The best one can usually do is to reduce the 
> frequency at which some credential must be entered manually.
>
> Solr supplies several authentication mechanisms besides BasicAuth.
> Would one of those serve?
>
> --
> Mark H. Wood
> Lead Technology Analyst
>
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> https://nam02.safelinks.protection.outlook.com/?url=www.ulib.iupui.edu
> data=02%7C01%7Cvkommu%40dtcc.com%7C9dab071571614ae9dff508d7693853
> f6%7C0465519d7f554d47998b55e2a86f04a8%7C0%7C0%7C637093562958117939
> ;sdata=Q0881FOYJhLgLOPOiUUqSpY6UbbIw82vUmVlaSAx0AU%3Dreserved=0
DTCC DISCLAIMER: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify us 
immediately and delete the email and any attachments from your system. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email.

RE: Anyway to encrypt admin user plain text password in Solr

[Kommu, Vinodh K.]

Thanks Jorn - will check out these options.


Regards,
Vinodh Kumar K
Middleware Cache and Search Engineering
DTCC Chennai

-Original Message-
From: Jörn Franke  
Sent: Friday, November 15, 2019 1:00 AM
To: solr-user@lucene.apache.org
Subject: Re: Anyway to encrypt admin user plain text password in Solr

ATTENTION! This email originated outside of DTCC; exercise caution.

Use Kerberos or JWT token.

> Am 11.11.2019 um 11:41 schrieb Kommu, Vinodh K. :
>
> Hi,
>
> After creating admin user in Solr when security is enabled, we have to store 
> the admin user's credentials in plain text format. Is there any option or a 
> way to encrypt the plain text password?
>
> Thanks,
> Vinodh
> DTCC DISCLAIMER: This email and any files transmitted with it are 
> confidential and intended solely for the use of the individual or entity to 
> whom they are addressed. If you have received this email in error, please 
> notify us immediately and delete the email and any attachments from your 
> system. The recipient should check this email and any attachments for the 
> presence of viruses. The company accepts no liability for any damage caused 
> by any virus transmitted by this email.
DTCC DISCLAIMER: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify us 
immediately and delete the email and any attachments from your system. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email.

Re: NPE during spell checking when result collapsing is activated and local parameters are used

[Stefan Walter]

Indeed, you are right. Interestingly, it generally worked with the two {!
..} in the filter query - besides the problem with the collations, of
course. Therefore I never questioned it...

Thank you!
Stefan


Am 15. November 2019 um 00:01:52, Tomás Fernández Löbbe (
tomasflo...@gmail.com) schrieb:

I believe your syntax is incorrect. I believe local params must all be
included in between the same {!...}, and "{!" can only be at the beginning

have you tried:

={!collapse tag=collapser field=productId sort='merchantOrder asc,
price asc, id asc'}



On Thu, Nov 14, 2019 at 4:54 AM Stefan Walter  wrote:

> Hi!
>
> I have an issue with Solr 7.3.1 in the spell checking component:
>
> java.lang.NullPointerException at
>
>
org.apache.solr.search.CollapsingQParserPlugin$OrdFieldValueCollector.finish(CollapsingQParserPlugin.java:1021)

> at
>
>
org.apache.solr.search.CollapsingQParserPlugin$OrdFieldValueCollector.finish(CollapsingQParserPlugin.java:1081)

> at
>
>
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:230)

> at
>
>
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1602)

> at
>
>
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1419)

> at
>
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:584)
> ...
>
> I have found an issue that addresses a similiar problem:
> https://issues.apache.org/jira/browse/SOLR-8807
>
> The fix, which was introduced with this issue seems to miss our
situation,
> though. The relevant part of the query is this:
>
> ={!tag=collapser}{!collapse field=productId sort='merchantOrder asc,
> price asc, id asc'}
>
> When I remove the local parameter {!tag=collapser} the collation works
> fine. Looking at the diff of the commit of the issue mentioned above, it
> seems that the "startsWith" could be the problem:
>
> + // Collate testing does not support the Collapse QParser (See
> SOLR-8807)
> + params.remove("expand");
> + String[] filters = params.getParams(CommonParams.FQ);
> + if (filters != null) {
> + List filtersToApply = new ArrayList<>(filters.length);
> + for (String fq : filters) {
> + if (!fq.startsWith("{!collapse")) {
> + filtersToApply.add(fq);
> + }
> + }
> + params.set("fq", filtersToApply.toArray(new
> String[filtersToApply.size()]));
> + }
>
> Can someone confirm this? I would open a bug ticket then. (Since the code
> is unchanged in the latest version.)
>
> Thanks,
> Stefan
>