Adding additional zookeeper on top

[yaswanth kumar]

Hi Team

Can someone let me know if we can do an upgrade to  zookeeper ensemble from 
standalone ??

I have 3 solr nodes with one zookeeper running on one of the node .. and it’s a 
solr cloud .. so now can I install zookeeper on another node just to make sure 
it’s not a single point of failure when the solr node that got zookeeper is 
down??

Also want to understand what’s the best formula of choosing no of zookeeper 
that’s needed for solr cloud like for how many solr nodes .. how many zookeeper 
do we need to maintain for best fault tolerance 

Sent from my iPhone

Re: DIH on SolrCloud

[Issei Nishigata]

Thank you for your quick reply.
Can I make sure that the indexing isn't conducted on the node where the DIH
executed but conducted on the Leader node, right?

As far as I have seen a log, there are errors: the failed establishment of
connection occurred from Node2 on the state of Replica on running DIH to
Node9 where on the state of Replica.
Therefore, for my understanding, I thought there would be errors when the
DIH was implemented at the Node2 and trying to forward a tlog to Node9.

Unless Node9 receives the tlog, if Node1 as Leader receives the tlog, I do
believe there are no worries because Node9 is synchronised with Node1.
But if Node1 as Leader cannot receive the tlog, Replica might be
synchronised to the Leader soon and that makes me a problematic issue.
I want to try to find out the cause as I will check all log files of all
servers through, but could you give me your comment for my understanding of
the indexing architecture on SolrCloud, please?


Thanks,
Issei

2020年8月14日(金) 0:33 Jörn Franke :

> DIH is deprecated in current Solr versions. The general recommendation is
> to do processing outside the Solr server and use the update handler (the
> normal one, not Cell) to add documents to the index. So you should avoid
> using it as it is not future proof .
>
> If you need more Time to migrate to a non-DIH solution:
> I recommend to look at all log files of all servers to find the real error
> behind the issue. If you trigger in Solr cloud mode DIH from node 2 that
> does not mean it is executed there !
>
> What could to wrong:
> Other nodes do not have access to files/database or there is a parsing
> error or a script error.
>
> > Am 13.08.2020 um 17:21 schrieb Issei Nishigata :
> >
> > Hi, All
> >
> > I'm using Solr4.10 with SolrCloud mode.
> > I have 10 Nodes. one of Nodes is Leader Node, the others is Replica.(I
> will
> > call this Node1 to Node10 for convenience)
> > -> 1 Shard, 1 Leader(Node1), 9 Replica(Node2-10)
> > Indexing always uses DIH of Node2. Therefore, DIH may be executed when
> > Node2 is Leader or Replica.
> > Node2 is not forcibly set to Leader when DIH is executed.
> >
> > At one point, when Node2 executed DIH in the Replica state, the following
> > error in Node9 occurred.
> >
> >
> [updateExecutor-1-thread-9737][ERROR][org.apache.solr.common.SolrException]
> > - org.apache.solr.client.solrj.SolrServerException: IOException occured
> > when talking to server at:
> http://samplehost:8983/solr/test_shard1_replica9
> >
> > I think this is the error while sending data from Node2 to Node9. And
> Node9
> > couldn't respond for some reason.
> >
> > The error occurs sometimes however it is not reproducible so that the
> > investigation is troublesome.
> > Is there any possible cause for this problem? I am worrying about if it
> is
> > doing Solr anti-pattern.
> > The thing is, when running DIH by Node2 as Replica, the above error
> occurs
> > towards Node1 as Leader,
> > then soon after, all the nodes might be returning to the index of the
> > Node1.
> > Do you think my understanding makes sense?
> >
> > If using DIH on SolrCloud is not recommended, please let me know about
> this.
> >
> > Thanks,
> > Issei
>

Re: DIH on SolrCloud

[Jörn Franke]

DIH is deprecated in current Solr versions. The general recommendation is to do 
processing outside the Solr server and use the update handler (the normal one, 
not Cell) to add documents to the index. So you should avoid using it as it is 
not future proof .

If you need more Time to migrate to a non-DIH solution:
I recommend to look at all log files of all servers to find the real error 
behind the issue. If you trigger in Solr cloud mode DIH from node 2 that does 
not mean it is executed there !

What could to wrong:
Other nodes do not have access to files/database or there is a parsing error or 
a script error.

> Am 13.08.2020 um 17:21 schrieb Issei Nishigata :
> 
> Hi, All
> 
> I'm using Solr4.10 with SolrCloud mode.
> I have 10 Nodes. one of Nodes is Leader Node, the others is Replica.(I will
> call this Node1 to Node10 for convenience)
> -> 1 Shard, 1 Leader(Node1), 9 Replica(Node2-10)
> Indexing always uses DIH of Node2. Therefore, DIH may be executed when
> Node2 is Leader or Replica.
> Node2 is not forcibly set to Leader when DIH is executed.
> 
> At one point, when Node2 executed DIH in the Replica state, the following
> error in Node9 occurred.
> 
> [updateExecutor-1-thread-9737][ERROR][org.apache.solr.common.SolrException]
> - org.apache.solr.client.solrj.SolrServerException: IOException occured
> when talking to server at: http://samplehost:8983/solr/test_shard1_replica9
> 
> I think this is the error while sending data from Node2 to Node9. And Node9
> couldn't respond for some reason.
> 
> The error occurs sometimes however it is not reproducible so that the
> investigation is troublesome.
> Is there any possible cause for this problem? I am worrying about if it is
> doing Solr anti-pattern.
> The thing is, when running DIH by Node2 as Replica, the above error occurs
> towards Node1 as Leader,
> then soon after, all the nodes might be returning to the index of the
> Node1.
> Do you think my understanding makes sense?
> 
> If using DIH on SolrCloud is not recommended, please let me know about this.
> 
> Thanks,
> Issei

DIH on SolrCloud

[Issei Nishigata]

Hi, All

I'm using Solr4.10 with SolrCloud mode.
I have 10 Nodes. one of Nodes is Leader Node, the others is Replica.(I will
call this Node1 to Node10 for convenience)
-> 1 Shard, 1 Leader(Node1), 9 Replica(Node2-10)
Indexing always uses DIH of Node2. Therefore, DIH may be executed when
Node2 is Leader or Replica.
Node2 is not forcibly set to Leader when DIH is executed.

At one point, when Node2 executed DIH in the Replica state, the following
error in Node9 occurred.

[updateExecutor-1-thread-9737][ERROR][org.apache.solr.common.SolrException]
- org.apache.solr.client.solrj.SolrServerException: IOException occured
when talking to server at: http://samplehost:8983/solr/test_shard1_replica9

I think this is the error while sending data from Node2 to Node9. And Node9
couldn't respond for some reason.

The error occurs sometimes however it is not reproducible so that the
investigation is troublesome.
Is there any possible cause for this problem? I am worrying about if it is
doing Solr anti-pattern.
The thing is, when running DIH by Node2 as Replica, the above error occurs
towards Node1 as Leader,
then soon after, all the nodes might be returning to the index of the
Node1.
Do you think my understanding makes sense?

If using DIH on SolrCloud is not recommended, please let me know about this.

Thanks,
Issei

Re: Force open a searcher in solr.

[Erick Erickson]

Please don’t mess with _version_, that’s used internally for optimistic locking.
I don’t have a clue, really, whether changing the definition will be deleterious
or not.

OTOH, that field was presumably defined by people who put the use of
_version_ in in the first place, so changing it is just asking for trouble.

In general, any variable defined with before-and-after underscores should be
left strictly alone.

Otherwise you’re on the right track.

Best,
Erick

> On Aug 13, 2020, at 8:46 AM, Akshay Murarka  wrote:
> 
> So to make things clear, belows what I am expecting
> 
> I have a document with a unique id field lets say "uniqueID".
> This document has both stored/indexed and not stored/ not indexed fields
> Currently I have my pop values in external files but I will instead define
> a new field in schema (popVal) which will not be stored or indexed and have
> docValues=true.
> I am also moving _version_ field to indexed=false and stored=false, since I
> don't have any case where I retrieve it and use it for searching. Just
> hoping doing this doesn't cause any issues with updates in general (I read
> that keeping this as not stored and not indexed is recommended since solr 7)
> 
> Regards,
> Akshay
> 
> On Thu, Aug 13, 2020 at 4:53 PM Erick Erickson 
> wrote:
> 
>> Let us know how it works. I want to be sure I’m not confusing you
>> though. There isn’t a “doc ID field”. The structure of an eff file is
>> docid:value
>> 
>> where docid is your . What updating numerics does is allow
>> you to update a field in a doc that’s identified by . That
>> field is any name you want as long as it’s defined respecting
>> the limitations in that link.
>> 
>> Best,
>> Erick
>> 
>>> On Aug 13, 2020, at 6:30 AM, Akshay Murarka  wrote:
>>> 
>>> Hey Erick,
>>> 
>>> Thanks for the information about the doc ID field.
>>> So our external file values are single float value fields and we do use
>>> them in functional queries in boost parameter, so based on the definition
>>> the above should work.
>>> So currently we use solr 5.4.0 but are in the process of upgrading our
>>> systems so will try out this change.
>>> 
>>> Regards,
>>> Akshay
>>> 
>>> On Mon, Aug 10, 2020 at 10:19 PM Erick Erickson >> 
>>> wrote:
>>> 
 Right, but you can use those with function queries. Assuming your eff
 entry is a doc ID plus single numeric, I was wondering if you can
 accomplish what you need to with function queries...
 
> On Aug 10, 2020, at 11:30 AM, raj.yadav 
>> wrote:
> 
> Erick Erickson wrote
>> Ah, ok. That makes sense. I wonder if your use-case would be better
>> served, though, by “in place updates”, see:
>> 
 
>> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
>> This has been around in since Solr 6.5…
> 
> As per documentation `in place update` is only available for numeric
> docValues (along with few more conditions). And here its external field
> type.
> 
> Regards,
> Raj
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
 
 
>> 
>> 

Re: Solr 8.3.1 - NullPointer during Autoscaling

[Erick Erickson]

Hmm, not that I know of. This seems like a reasonable JIRA, please raise one. 
Also, that might get the attention of someone who might have a work-around to 
offer.

Best,
Erick

> On Aug 13, 2020, at 9:50 AM, Anton Pfennig  wrote:
> 
> Hi Erick,
> 
> I found the issue with this NullPointer Issue.
> 
> If SolrCloud runs on Kubernetes, and if Kubernetes cluster is being scaled, 
> new pods are created with solr nodes on it.
> The DNS names of the new pods are propagated *AFTER* new solr node instance 
> is created.
> 
> so the autoscaling work like this:
> 1. pod is being created
> 2. solr container starts
> 3. new solr node immediately calls Zookeeper and/or overseer and tells them 
> new solr node url.
> 4. overseer immediately tries to fetch metrics from the new node but at this 
> time the pod creation is still not complete. therefore, there is NO DNS entry 
> of the new pod and metric call fails --> autoscaling fails.
> 5. pod creation is finished and the new solr node is reachable.
> 
> Is there any way to delay the autoscaling part / fetching metrics?
> 
> Br Anton
> 
> 
> 
> 
> On 12.08.20, 14:05, "Erick Erickson"  wrote:
> 
>Autoscaling may be overkill, Is this a one-time thing or something you 
> need automated?
>Because for a one-time event, it’s simpler/faster just to specify 
>createNodeSet with the CREATE command that lists the new machines you want 
> the
>collection to be placed on.
> 
>Note that there’s the special value EMPTY, if you use that then you can 
> ADDREPLICA
>to build out your shards with the “node” parameter to place each one 
> exactly
>where you want.
> 
>Best,
>Erick
> 
>> On Aug 12, 2020, at 7:47 AM, Anton Pfennig  wrote:
>> 
>> Hi guys,
>> 
>> in my solr setup as SolrCloud (8.3.1) I’m using 5 nodes for one collection 
>> (say “collection1”, one on each node.
>> Now I would like to add a new collection on the same solr cluster but 
>> additionally the new collection (say “collection2”) should be only 
>> replicated on only nodes with sysprop.channel=mysysprop.
>> 
>> Whole setup runs on GKE (Kubernetes)
>> 
>> So If i scale and add additionally 5 nodes wich correctly sysprop.channel 
>> being set and cluster gets a new node, autoscaling dies with 
>> nullpointerexception trying to fetch metrics from new node. (see logs 
>> attached).
>> 
>> This is not an IO issue because the nodes and zookeeper can talk to each 
>> other.
>> And if I call /metrics on these new nodes I also see the right properties.
>> 
>> Also I see no violations in the suggester.
>> 
>> Please help 
>> 
>> THX
>> Anton
>> 
>> Attached files:
>> 
>> *   
>> autoscaling.json
>> *   Logs
>> 
>> 
>> 
>> {
>> "policies":{"my-channel-policy[{
>>   "replica":"<100",
>>   "shard":"#EACH",
>>   "collection":"collection2",
>>   "nodeset":{"sysprop.channel":"mychannel"}
>>   }]},
>> "cluster-preferences":[
>>   {
>> "minimize":"cores",
>> "precision":1},
>>   {"maximize":"freedisk"}],
>> "triggers":{
>>   ".auto_add_replicas":{
>> "name":".auto_add_replicas",
>> "event":"nodeLost",
>> "waitFor":120,
>> "enabled":true,
>> "actions":[
>>   {
>> "name":"auto_add_replicas_plan",
>> "class":"solr.AutoAddReplicasPlanAction"},
>>   {
>> "name":"execute_plan",
>> "class":"solr.ExecutePlanAction"}]},
>>   ".scheduled_maintenance":{
>> "name":".scheduled_maintenance",
>> "event":"scheduled",
>> "startTime":"NOW",
>> "every":"+1DAY",
>> "enabled":true,
>> "actions":[
>>   {
>> "name":"inactive_shard_plan",
>> "class":"solr.InactiveShardPlanAction"},
>>   {
>> "name":"inactive_markers_plan",
>> "class":"solr.InactiveMarkersPlanAction"},
>>   {
>> "name":"execute_plan",
>> "class":"solr.ExecutePlanAction"}]},
>>   "node_added_trigger":{
>> "event":"nodeAdded",
>> "waitFor":20,
>> "enabled":true,
>> "actions":[
>>   {
>> "name":"compute_plan",
>> "class":"solr.ComputePlanAction"},
>>   {
>> "name":"execute_plan",
>> "class":"solr.ExecutePlanAction"}]}},
>> "listeners":{
>>   ".auto_add_replicas.system":{
>> "beforeAction":[],
>> "afterAction":[],
>> "stage":[
>>   "STARTED",
>>   "ABORTED",
>>   "SUCCEEDED",
>>   "FAILED",
>>   "BEFORE_ACTION",
>>   "AFTER_ACTION",
>>   "IGNORED"],
>> "trigger":".auto_add_replicas",
>> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener"},
>>   ".scheduled_maintenance.system":{
>> "beforeAction":[],
>> "afterAction":[],
>> "stage":[
>>   "STARTED",
>>   "ABORTED",
>>   "SUCCEEDED",
>>   "FAILED",
>>   "BEFORE_ACTION",
>>   "AFTER_ACTION",
>>   "IGNORED"],
>> "trigger":".scheduled_maintenance",
>> 

Re: Solr 8.3.1 - NullPointer during Autoscaling

[Anton Pfennig]

Hi Erick,

I found the issue with this NullPointer Issue.

If SolrCloud runs on Kubernetes, and if Kubernetes cluster is being scaled, new 
pods are created with solr nodes on it.
The DNS names of the new pods are propagated *AFTER* new solr node instance is 
created.

so the autoscaling work like this:
1. pod is being created
2. solr container starts
3. new solr node immediately calls Zookeeper and/or overseer and tells them new 
solr node url.
4. overseer immediately tries to fetch metrics from the new node but at this 
time the pod creation is still not complete. therefore, there is NO DNS entry 
of the new pod and metric call fails --> autoscaling fails.
5. pod creation is finished and the new solr node is reachable.

Is there any way to delay the autoscaling part / fetching metrics?

Br Anton




On 12.08.20, 14:05, "Erick Erickson"  wrote:

Autoscaling may be overkill, Is this a one-time thing or something you need 
automated?
Because for a one-time event, it’s simpler/faster just to specify 
createNodeSet with the CREATE command that lists the new machines you want 
the
collection to be placed on.

Note that there’s the special value EMPTY, if you use that then you can 
ADDREPLICA
to build out your shards with the “node” parameter to place each one exactly
where you want.

Best,
Erick

> On Aug 12, 2020, at 7:47 AM, Anton Pfennig  
wrote:
> 
> Hi guys,
> 
> in my solr setup as SolrCloud (8.3.1) I’m using 5 nodes for one 
collection (say “collection1”, one on each node.
> Now I would like to add a new collection on the same solr cluster but 
additionally the new collection (say “collection2”) should be only replicated 
on only nodes with sysprop.channel=mysysprop.
> 
> Whole setup runs on GKE (Kubernetes)
> 
> So If i scale and add additionally 5 nodes wich correctly sysprop.channel 
being set and cluster gets a new node, autoscaling dies with 
nullpointerexception trying to fetch metrics from new node. (see logs attached).
> 
> This is not an IO issue because the nodes and zookeeper can talk to each 
other.
> And if I call /metrics on these new nodes I also see the right properties.
> 
> Also I see no violations in the suggester.
> 
> Please help 
> 
> THX
> Anton
> 
> Attached files:
> 
>  *   
autoscaling.json
>  *   Logs
> 
> 
> 
> {
>  "policies":{"my-channel-policy[{
>"replica":"<100",
>"shard":"#EACH",
>"collection":"collection2",
>"nodeset":{"sysprop.channel":"mychannel"}
>}]},
>  "cluster-preferences":[
>{
>  "minimize":"cores",
>  "precision":1},
>{"maximize":"freedisk"}],
>  "triggers":{
>".auto_add_replicas":{
>  "name":".auto_add_replicas",
>  "event":"nodeLost",
>  "waitFor":120,
>  "enabled":true,
>  "actions":[
>{
>  "name":"auto_add_replicas_plan",
>  "class":"solr.AutoAddReplicasPlanAction"},
>{
>  "name":"execute_plan",
>  "class":"solr.ExecutePlanAction"}]},
>".scheduled_maintenance":{
>  "name":".scheduled_maintenance",
>  "event":"scheduled",
>  "startTime":"NOW",
>  "every":"+1DAY",
>  "enabled":true,
>  "actions":[
>{
>  "name":"inactive_shard_plan",
>  "class":"solr.InactiveShardPlanAction"},
>{
>  "name":"inactive_markers_plan",
>  "class":"solr.InactiveMarkersPlanAction"},
>{
>  "name":"execute_plan",
>  "class":"solr.ExecutePlanAction"}]},
>"node_added_trigger":{
>  "event":"nodeAdded",
>  "waitFor":20,
>  "enabled":true,
>  "actions":[
>{
>  "name":"compute_plan",
>  "class":"solr.ComputePlanAction"},
>{
>  "name":"execute_plan",
>  "class":"solr.ExecutePlanAction"}]}},
>  "listeners":{
>".auto_add_replicas.system":{
>  "beforeAction":[],
>  "afterAction":[],
>  "stage":[
>"STARTED",
>"ABORTED",
>"SUCCEEDED",
>"FAILED",
>"BEFORE_ACTION",
>"AFTER_ACTION",
>"IGNORED"],
>  "trigger":".auto_add_replicas",
>  "class":"org.apache.solr.cloud.autoscaling.SystemLogListener"},
>".scheduled_maintenance.system":{
>  "beforeAction":[],
>  "afterAction":[],
>  "stage":[
>"STARTED",
>"ABORTED",
>"SUCCEEDED",
>"FAILED",
>"BEFORE_ACTION",
>"AFTER_ACTION",
>"IGNORED"],
>  "trigger":".scheduled_maintenance",

Re: Force open a searcher in solr.

[Akshay Murarka]

So to make things clear, belows what I am expecting

I have a document with a unique id field lets say "uniqueID".
This document has both stored/indexed and not stored/ not indexed fields
Currently I have my pop values in external files but I will instead define
a new field in schema (popVal) which will not be stored or indexed and have
docValues=true.
I am also moving _version_ field to indexed=false and stored=false, since I
don't have any case where I retrieve it and use it for searching. Just
hoping doing this doesn't cause any issues with updates in general (I read
that keeping this as not stored and not indexed is recommended since solr 7)

Regards,
Akshay

On Thu, Aug 13, 2020 at 4:53 PM Erick Erickson 
wrote:

> Let us know how it works. I want to be sure I’m not confusing you
> though. There isn’t a “doc ID field”. The structure of an eff file is
> docid:value
>
> where docid is your . What updating numerics does is allow
> you to update a field in a doc that’s identified by . That
> field is any name you want as long as it’s defined respecting
> the limitations in that link.
>
> Best,
> Erick
>
> > On Aug 13, 2020, at 6:30 AM, Akshay Murarka  wrote:
> >
> > Hey Erick,
> >
> > Thanks for the information about the doc ID field.
> > So our external file values are single float value fields and we do use
> > them in functional queries in boost parameter, so based on the definition
> > the above should work.
> > So currently we use solr 5.4.0 but are in the process of upgrading our
> > systems so will try out this change.
> >
> > Regards,
> > Akshay
> >
> > On Mon, Aug 10, 2020 at 10:19 PM Erick Erickson  >
> > wrote:
> >
> >> Right, but you can use those with function queries. Assuming your eff
> >> entry is a doc ID plus single numeric, I was wondering if you can
> >> accomplish what you need to with function queries...
> >>
> >>> On Aug 10, 2020, at 11:30 AM, raj.yadav 
> wrote:
> >>>
> >>> Erick Erickson wrote
>  Ah, ok. That makes sense. I wonder if your use-case would be better
>  served, though, by “in place updates”, see:
> 
> >>
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
>  This has been around in since Solr 6.5…
> >>>
> >>> As per documentation `in place update` is only available for numeric
> >>> docValues (along with few more conditions). And here its external field
> >>> type.
> >>>
> >>> Regards,
> >>> Raj
> >>>
> >>>
> >>>
> >>> --
> >>> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >>
> >>
>
>

Re: Reaching max Filter cache limit increases the request latencies.

[Akshay Murarka]

Hey Erick,

So I am investigating the point where we can limit the values that are
cached using {!cache=false} (we already use it in some of our cases)
So in general there is 0 evictions on filter cache side but whenever we hit
this max limit there is a spike in evictions as well (which is expected)
As far as I remember not forcing enum on our side, but will definitely
verify that.
My filter cache hit ratio remains constant at around 97.5 %  and even
during this eviction the hit ratio doesn't go down
Regarding other operation there are  a few cases where indexing (80 to 150
docs) also happened during that time but there are also cases where index
happened 5- 10 min after that and the latencies remained high.

Regards,
Akshay


On Thu, Aug 13, 2020 at 5:08 PM Erick Erickson 
wrote:

> Well, when you hit the max capacity, cache entries get aged out and are
> eligible for GC, so GC
> activity increases. But for aging out filterCache entries to be
> noticeable, you have to be
> flushing a _lot_ of them out. Which, offhand, makes me wonder if you’re
> using the filterCache
> appropriately.
>
> Here’s what I’d investigate first: What kinds of fq clauses are you using
> and are they making
> best use of the filterCache? Consider an fq clause like
>
> fq=date_field:[* to NOW]
>
> That will consume
> an entry in the filterCache and never be re-used because NOW is the epoch
> time and will change a millisecond later.
>
> Similarly for fq clauses that contain a lot of values that may vary, for
> instance
>
> fq=id:(1 2 4 86 93 …)
>
> where the list of IDs is not likely to be repeated. Or even repeated in a
> different order.
>
> If you do identify patterns that you _know_ will not be repeated, just add
> fq={!cache=false}your_unrepeated_pattern
>
> What I’m guessing here is that if you’ve correctly identified that the
> filterCache filling up
> is increasing GC activity that much, you must be evicting a _lot_ of fq
> entries very rapidly
> which indicates you’re not repeating fq’s very often.
>
> I should add that the filterCache is also used for some other operations,
> particularly some
> kinds of faceting if you specify the enum method. Are you forcing that?
>
> All that said, I’m also wondering if this is coincidence and your slowdown
> is something
> else. Because given all the work a query does, the additional bookkeeping
> due to
> filterCache churn doesn’t really sound like the culprit. Prior to the
> filterCache filling up,
> what’s your hit ratio? The scenario I can see where the filterCache churn
> could cause
> your response times to go up is if, up until that point, you’re getting a
> high hit ratio that
> goes down after the cache starts aging out entries. I find this rather
> unlikely, but possible.
>
> Best,
> Erick
>
> > On Aug 13, 2020, at 3:19 AM, Akshay Murarka  wrote:
> >
> > Hey guys,
> >
> > So for quite some time we have been facing an issue where whenever the
> Used Filter Cache value reaches the maximum configured value we start
> seeing an increase in the query latencies on solr side.
> > During this time we also see an increase in our garbage collection and
> CPU as well.
> > When a commit happens with openSearcher=true then only the latencies
> value come back to normal.
> >
> > Is there any setting that can help us with this or will increasing the
> max configured value for filter cache help, because right now we can’t
> increase the commit frequency
> >
> > Thanks for the help.
> >
> > Regards,
> > Akshay
> >
> >
> > Below is the graph for request latency
> > 
> >
> >
> >
> >
> >
> > Below is the graph for the Filter cache values
> > 
>
>

Re: Solr and commits

[Erick Erickson]

Here’s a long explanation:
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

short explanation:
1> yes
2> when the hard commit setting in solrconfig.xml kicks in, regardless the 
openSearcher setting.

Best,
Erick

> On Aug 13, 2020, at 1:43 AM, Jayadevan Maymala  
> wrote:
> 
> Hi all,
> 
> A few doubts about commits.
> 
> 1)If no commit parameters are passed from a client (solarium) update, will
> the autoSoftCommit values automatically work?
> 2) When we are not committing from the client, when will the data actually
> be flushed to disk?
> 
> Regards,
> Jayadevan

Re: Reaching max Filter cache limit increases the request latencies.

[Erick Erickson]

Well, when you hit the max capacity, cache entries get aged out and are 
eligible for GC, so GC
activity increases. But for aging out filterCache entries to be noticeable, you 
have to be
flushing a _lot_ of them out. Which, offhand, makes me wonder if you’re using 
the filterCache
appropriately.

Here’s what I’d investigate first: What kinds of fq clauses are you using and 
are they making
best use of the filterCache? Consider an fq clause like 

fq=date_field:[* to NOW]

That will consume 
an entry in the filterCache and never be re-used because NOW is the epoch time 
and will change a millisecond later.

Similarly for fq clauses that contain a lot of values that may vary, for 
instance 

fq=id:(1 2 4 86 93 …)

where the list of IDs is not likely to be repeated. Or even repeated in a 
different order.

If you do identify patterns that you _know_ will not be repeated, just add
fq={!cache=false}your_unrepeated_pattern

What I’m guessing here is that if you’ve correctly identified that the 
filterCache filling up
is increasing GC activity that much, you must be evicting a _lot_ of fq entries 
very rapidly
which indicates you’re not repeating fq’s very often.

I should add that the filterCache is also used for some other operations, 
particularly some
kinds of faceting if you specify the enum method. Are you forcing that?

All that said, I’m also wondering if this is coincidence and your slowdown is 
something
else. Because given all the work a query does, the additional bookkeeping due 
to 
filterCache churn doesn’t really sound like the culprit. Prior to the 
filterCache filling up,
what’s your hit ratio? The scenario I can see where the filterCache churn could 
cause
your response times to go up is if, up until that point, you’re getting a high 
hit ratio that
goes down after the cache starts aging out entries. I find this rather 
unlikely, but possible.

Best,
Erick

> On Aug 13, 2020, at 3:19 AM, Akshay Murarka  wrote:
> 
> Hey guys,
> 
> So for quite some time we have been facing an issue where whenever the Used 
> Filter Cache value reaches the maximum configured value we start seeing an 
> increase in the query latencies on solr side.
> During this time we also see an increase in our garbage collection and CPU as 
> well.
> When a commit happens with openSearcher=true then only the latencies value 
> come back to normal.
> 
> Is there any setting that can help us with this or will increasing the max 
> configured value for filter cache help, because right now we can’t increase 
> the commit frequency
> 
> Thanks for the help.
> 
> Regards,
> Akshay
> 
> 
> Below is the graph for request latency
> 
> 
> 
> 
> 
> 
> Below is the graph for the Filter cache values
> 

Re: Force open a searcher in solr.

[Erick Erickson]

Let us know how it works. I want to be sure I’m not confusing you
though. There isn’t a “doc ID field”. The structure of an eff file is
docid:value

where docid is your . What updating numerics does is allow
you to update a field in a doc that’s identified by . That 
field is any name you want as long as it’s defined respecting
the limitations in that link.

Best,
Erick

> On Aug 13, 2020, at 6:30 AM, Akshay Murarka  wrote:
> 
> Hey Erick,
> 
> Thanks for the information about the doc ID field.
> So our external file values are single float value fields and we do use
> them in functional queries in boost parameter, so based on the definition
> the above should work.
> So currently we use solr 5.4.0 but are in the process of upgrading our
> systems so will try out this change.
> 
> Regards,
> Akshay
> 
> On Mon, Aug 10, 2020 at 10:19 PM Erick Erickson 
> wrote:
> 
>> Right, but you can use those with function queries. Assuming your eff
>> entry is a doc ID plus single numeric, I was wondering if you can
>> accomplish what you need to with function queries...
>> 
>>> On Aug 10, 2020, at 11:30 AM, raj.yadav  wrote:
>>> 
>>> Erick Erickson wrote
 Ah, ok. That makes sense. I wonder if your use-case would be better
 served, though, by “in place updates”, see:
 
>> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
 This has been around in since Solr 6.5…
>>> 
>>> As per documentation `in place update` is only available for numeric
>>> docValues (along with few more conditions). And here its external field
>>> type.
>>> 
>>> Regards,
>>> Raj
>>> 
>>> 
>>> 
>>> --
>>> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>> 
>> 

Re: Force open a searcher in solr.

[Akshay Murarka]

Hey Erick,

Thanks for the information about the doc ID field.
So our external file values are single float value fields and we do use
them in functional queries in boost parameter, so based on the definition
the above should work.
So currently we use solr 5.4.0 but are in the process of upgrading our
systems so will try out this change.

Regards,
Akshay

On Mon, Aug 10, 2020 at 10:19 PM Erick Erickson 
wrote:

> Right, but you can use those with function queries. Assuming your eff
> entry is a doc ID plus single numeric, I was wondering if you can
> accomplish what you need to with function queries...
>
> > On Aug 10, 2020, at 11:30 AM, raj.yadav  wrote:
> >
> > Erick Erickson wrote
> >> Ah, ok. That makes sense. I wonder if your use-case would be better
> >> served, though, by “in place updates”, see:
> >>
> https://lucene.apache.org/solr/guide/8_1/updating-parts-of-documents.html
> >> This has been around in since Solr 6.5…
> >
> > As per documentation `in place update` is only available for numeric
> > docValues (along with few more conditions). And here its external field
> > type.
> >
> > Regards,
> > Raj
> >
> >
> >
> > --
> > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
>

Re: Incorrect Insecure Settings Check in CoreContainer

[Jason Gerlowski]

Hey Mark,

I've fixed it for 8.7 as a part of this ticket here:
https://issues.apache.org/jira/browse/SOLR-14748. Thanks for reporting
this.

Jason

On Tue, Aug 11, 2020 at 3:19 PM Jason Gerlowski  wrote:
>
> Yikes, yeah it's hard to argue with that.
>
> I'm a little confused because I remember testing this, but maybe it
> snuck in at the last minute?  In any case, I'll reopen that jira to
> fix the check there.
>
> Sorry guys.
>
> Jason
>
>
> On Wed, Aug 5, 2020 at 9:22 AM Jan Høydahl  wrote:
> >
> > This seems to have been introduced in 
> > https://issues.apache.org/jira/browse/SOLR-13972 in 8.4
> > That test seems to be inverted for sure.
> >
> > Jason?
> >
> > Jan
> >
> > > 5. aug. 2020 kl. 13:15 skrev Mark Todd1 :
> > >
> > >
> > > I've configured SolrCloud (8.5) with both SSL and Authentication which is 
> > > working correctly. However, I get the following warning in the logs
> > >
> > > Solr authentication is enabled, but SSL is off. Consider enabling SSL to 
> > > protect user credentials and data with encryption
> > >
> > > Looking at the source code for SolrCloud there appears to be a bug
> > > if (authenticationPlugin !=null && 
> > > StringUtils.isNotEmpty(System.getProperty("solr.jetty.https.port"))) {
> > >
> > > log.warn("Solr authentication is enabled, but SSL is off.  Consider 
> > > enabling SSL to protect user credentials and data with encryption.");
> > >
> > > }
> > >
> > > Rather than checking for an empty system property (which would indicate 
> > > SLL is off) its checking for a populated one which is what you get when 
> > > SSL is on.
> > >
> > > Should I raise this as a Jira bug?
> > >
> > > Mark ToddUnless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with number 
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> > >
> >

Re: Multiple "df" fields

[Edward Turner]

Goodness me, woops, yes, it was a typo -- sorry fo the confusion. We're
indeed exploring qf, rather than pf! :). So far it's looking promising!

Thanks for your eagle-eye spotting!

Best,
Edd

Edward Turner


On Wed, 12 Aug 2020 at 13:15, Erick Erickson 
wrote:

> Probably a typo but I think you mean qf rather than pf?
>
> They’re both actually valid, but pf is “phrase field” which will give
> different results….
>
>  Best,
> Erick
>
> > On Aug 12, 2020, at 5:26 AM, Edward Turner  wrote:
> >
> > Many thanks for your suggestions.
> >
> > We do use edismax and bq fields to help with our result ranking, but we'd
> > never thought about using it for this purpose (we were stuck on the
> > copyfield pattern + df pattern). This is a good suggestion though thank
> you.
> >
> > We're now exploring the use of the pf field (thanks to Alexandre R. for
> > this) to automatically search on multiple fields, rather than relying on
> df.
> >
> > Kind regards,
> >
> > Edd
> > 
> > Edward Turner
> >
> >
> > On Tue, 11 Aug 2020 at 15:44, Erick Erickson 
> > wrote:
> >
> >> Have you explored edismax?
> >>
> >>> On Aug 11, 2020, at 10:34 AM, Alexandre Rafalovitch <
> arafa...@gmail.com>
> >> wrote:
> >>>
> >>> I can't remember if field aliasing works with df but it may be worth a
> >> try:
> >>>
> >>>
> >>
> https://lucene.apache.org/solr/guide/8_1/the-extended-dismax-query-parser.html#field-aliasing-using-per-field-qf-overrides
> >>>
> >>> Another example:
> >>>
> >>
> https://github.com/arafalov/solr-indexing-book/blob/master/published/languages/conf/solrconfig.xml
> >>>
> >>> Regards,
> >>>   Alex
> >>>
> >>> On Tue., Aug. 11, 2020, 9:59 a.m. Edward Turner, 
> >>> wrote:
> >>>
>  Hi all,
> 
>  Is it possible to have multiple "df" fields? (We think the answer is
> no
>  because our experiments did not work when adding multiple "df" values
> to
>  solrconfig.xml -- but we just wanted to double check with those who
> know
>  better.) The reason we would like to do this is that we have two main
> >> field
>  types (with different analyzers) and we'd like queries without a field
> >> to
>  be searched over both of them. We could also use copyfields, but this
> >> would
>  require us to have a common analyzer, which isn't exactly what we
> want.
> 
>  An alternative solution is to pre-process the query prior to sending
> it
> >> to
>  Solr, so that queries with no field are changed as follows:
> 
>  q=value -> q=(field1:value OR field2:value)
> 
>  ... however, we feel a bit uncomfortable doing this though via String
>  manipulation.
> 
>  Is there an obvious way we should tackle this problem that we are
> >> missing
>  (e.g., which would be cleaner/safer and perhaps works at the Query
> >> object
>  level)?
> 
>  Many thanks and best wishes,
> 
>  Edd
> 
> >>
> >>
>
>