Tuning solr for large index with rapid writes

Hello, I'm looking for some guidance on the best steps for tuning a solr cloud cluster which is heavy on writes. We are currently running a solr cloud fleet composed of one core, one shard, and three nodes. The cloud is hosted in AWS, and each solr node is on its own linux r3.2xl instance with 8 c

Re: Tuning solr for large index with rapid writes

you > need to figure out how many docs you expect to host in this collection > eventually > and have N/200M shards. At least. > > There are various strategies when the answer is "I don't know", you > might add new > collections when you max out and then use "

Re: Tuning solr for large index with rapid writes

Thanks for the good suggestions on read traffic. I have been simulating reads through parsing our elb logs and replaying them from a fleet of test servers acting as frontends using Siege . We are hoping to tune mostly based on exact use case, and so this seems th

Updating solr schema for a collection in place

Hello, I've recently set up a solr cloud using solr 6.0, and I've been having some trouble getting our collections to pick up schema updates. Following the docs on zkcli.sh and the collections API

Re: Updating solr schema for a collection in place

n Mon, Jun 20, 2016 at 7:01 PM, Stephen Lewis wrote: > Hello, > > I've recently set up a solr cloud using solr 6.0, and I've been having > some trouble getting our collections to pick up schema updates. Following > the docs on zkcli.sh > <https://cwiki.apache.org/conflu

Re: Updating solr schema for a collection in place

ing, etc). to consider deleting/recreating the > collection before tearing your hair out. > > Best, > Erick > > On Mon, Jun 20, 2016 at 10:37 PM, Stephen Lewis > wrote: > > I'm happy to say I figured out the issue. Looking through previous > > questions in this

Re: Updating solr schema for a collection in place

In my test environment noodling, I noticed that even when using a managed schema, I could update the solrconfig.xml through a reload. Is it generally safe to switch between schema factories through schema reloads, or is this getting on the "cavalier" side of things? :) On Mon, Jun 20, 2016 at

Error upgrading from 6.0 to 6.1

Hello, I have a question about updating a solr cloud cluster servers in place. I have a scripted method for updating a solr cloud in place, which works consistently to up/down grade between 6.0.0 and 6.0.1 (in our test environment), but hits an error consistently when going from either to solr 6.1

Re: Error upgrading from 6.0 to 6.1

: org.apache.solr.common.cloud.ZkStateReader.getClusterProps()Ljava/util/Map; On Mon, Aug 22, 2016 at 8:18 PM, Stephen Lewis wrote: > Oops, apologies for my confusing grammar and for missing the attachment. > The intro sentence should have read "I have a question about upgrading a > solr cloud cluster in place." I've a

Re: Error upgrading from 6.0 to 6.1

Oops, apologies for my confusing grammar and for missing the attachment. The intro sentence should have read "I have a question about upgrading a solr cloud cluster in place." I've actually attached the log below this time. Thanks again, Stephen On Mon, Aug 22, 2016 at 7:41 PM

Re: Error upgrading from 6.0 to 6.1

readPool.runJob( QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) ,code=500} On Tue, Aug 23, 2016 at 6:00 AM, Shawn Heisey wrote: > On 8/22/2016 9:18 PM, Stephen Lewis wrote: > > Oops, apologies for my c

Distributing nodes with the collections API RESTORE command

Hello, I have a solr cloud cluster in a test environment running 6.1 where I am looking at using the collections API BACKUP and RESTORE commands to manage data integrity. When restoring from a backup, I'm finding the same behavior occurs every time; after the restore command, all shards are being

Re: Distributing nodes with the collections API RESTORE command

quot;base_url" : "http://IP4:8983/solr";, "node_name" : "IP4:8983_solr", "state" : "recovering" } } } }, "router" : { "name" : "c

Re: Distributing nodes with the collections API RESTORE command

k and get back to you. > > Thanks > Hrishikesh > > On Fri, Sep 16, 2016 at 2:56 PM, Stephen Lewis wrote: > > > Hello, > > > > I've tried this on both solr 6.1 and 6.2, with the same result. You are > > right that the collections API offering collection level b

Re: Distributing nodes with the collections API RESTORE command

Hello Again, I've just submitted a patch on Jira to this issue in the branch "branch_6_2". This is my first time submitting a patch (or even building solr!), so please let me know if there is anything I should change to be more helpful. Thanks! On Mon, Sep 19, 2016 at 4:47 PM

Re: Challenges with new Solrcloud Backup/Restore functionality

Hi All, I have been experiencing error#1 too with the current branch_6_2 build. I started noticing after I applied my patch to that branch (on issue #2), but it appears to occur without the patch as well. I haven't seen this issue with solr 6.1.0 d

Performance Implications of Different Routing Schemes

Hello! I'm wondering what information you may be able to provide on performance implications of implicit routing VS composite ID routing. In particular, I'm curious what the horizontal scaling behavior may be of implicit routing or composite ID routing with and without the "/" param appended on.

Re: Performance Implications of Different Routing Schemes

d on, so I may have over generalized a bit :) One other question I have; what is the best way to route queries? I've seen both the _route_ param, as well as the shard.keys param. Is there a reason to choose one over the other? Thanks again so much for your thoughtful response! Looking forw

Re: Mysterious Solr crash

And early EOF in the middle of an impasse usually indicates a timeout either from client to server or between servers in an update chain. Try increasing the timeout. On Sun, Jun 3, 2018, 9:57 AM Andrea Gazzarini wrote: > Looking at the stack trace, which seem truncated, I would start from here >

Re: Mysterious Solr crash

*update, not impasse (apologies) On Sun, Jun 3, 2018, 12:19 PM Stephen Lewis wrote: > And early EOF in the middle of an impasse usually indicates a timeout > either from client to server or between servers in an update chain. Try > increasing the timeout. > > On Sun, Jun 3, 2018,

Atomic Update Failure With solr.UUID Field

Hello SOLR Community, I'm prototyping a collection on SOLR 6.6.3 with UUID fields, and I'm hitting some trouble with atomic updates. At a high level, here's the problem: suppose you have a schema with an optional field of type solr.UUID field, and a document with a value for that field. Any atomic

Re: Atomic Update Failure With solr.UUID Field

-td4332277.html Would love to work together to get this fixed. On Tue, Aug 21, 2018 at 6:50 PM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hello SOLR Community, > > I'm prototyping a collection on SOLR 6.6.3 with UUID fields, and I'm > hitting some tro

Re: Atomic Update Failure With solr.UUID Field

a byte array? Cheers, Stephen On Wed, Aug 22, 2018 at 8:53 AM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hello again! I found a thread which seems relevant. It looks like someone > else found this occurred as well, but did not follow up with repro steps. >

Increasing Fault Tolerance of SOLR Cloud and Zookeeper

Hello SOLR Community! I have a SOLR cluster which recently hit this error (full error below). ""Cannot talk to ZooKeeper - Updates are disabled."" I'm running solr 6.6.2 and zookeeper 3.4.6. The first time this happened, we replaced a node within our cluster. The second time, we followed the advi

Re: Increasing Fault Tolerance of SOLR Cloud and Zookeeper

ances need to be > reachable. > > On that topic, did you bounce your ZK servers or change them in any > other way? There's a known ZK issue when you reconfigure live ZK > ensembles, see: https://issues.apache.org/jira/browse/SOLR-12727 > > Fourth: > See above. > > HTH

Re: Increasing Fault Tolerance of SOLR Cloud and Zookeeper

n't be more help > Erick > On Thu, Dec 13, 2018 at 3:15 PM Stephen Lewis Bianamara > wrote: > > > > Thanks for the help Erick. > > > > This is an external zookeeper, running on three separate AWS instances > > separate from the instances hosting SOLR. I th

Sorting on an empty filter

Hi SOLR Community, I've observed while hosting a SOLR 6.6 installation that if I issue a query where the filter query returns an empty set but there is a sort associated with it, the query takes an extended time to execute over the query with only the filter query and no sort associated to it. Thi

Re: Sorting on an empty filter

Hi Folks, Has anyone had a chance to consider this question yet? Thanks! Stephen On Tue, Mar 3, 2020 at 2:28 PM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hi SOLR Community, > > I've observed while hosting a SOLR 6.6 installation that if I issue a >

GC1 and StringDeduplication

Hi SOLR Community, I've been looking at performance tuning solr's GC lately. I found this helpful article on the matter. https://cwiki.apache.org/confluence/display/SOLR/ShawnHeisey One thing the article does not address is the GC1's ability to use string deduplication: https://blog.gceasy.io/201

What is the logical order of applying sorts in SOLR?

Hi SOLR Community, What is the order of operations which SOLR applies to sorting? I've observed many times and across SOLR versions that a restrictive filter with a sort takes an extremely long time to return, suggesting to me that the SORT is applied before the filter. An example situation is qu

Re: What is the logical order of applying sorts in SOLR?

Hi Paras, I'm not sure I follow. How would one replace sort by id with boosting? Thanks, Stephen On Fri, May 15, 2020, 7:43 AM Paras Lehana wrote: > As a workaround, can you try field boosting? > > On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara < > stephen.bianam

Re: What is the logical order of applying sorts in SOLR?

low.com/questions/45150856/how-to-know-when-to-use-solr-bq-vs-bf-and-how-to-apply-query-logic > > On Sat, May 16, 2020 at 6:27 PM Stephen Lewis Bianamara < > stephen.bianam...@gmail.com> wrote: > > > Hi Paras, > > > > I'm not sure I follow. How would one rep

Re: What is the logical order of applying sorts in SOLR?

where without sort, each node is asked > for their "top X" matches and others are never even sent. That would > be my working theory anyway, I am not deep into milti-path mode the > cluster code does. > > Regards, >Alex. > > On Mon, 11 May 2020 at 15:1

Japanese Query Unexpectedly Misses

Hi SOLR Community, I have an example of a basic Japanese indexing/recall scenario which I am trying to support, but cannot get to work. The scenario is: I would like for 日本人 (Japanese Person) to be matched by either 日本 (Japan) or 人 (Person). Currently, I am not seeing this work. My Japanese text

Re: Japanese Query Unexpectedly Misses

result having the lowest cost. > Using nBest, JapaneseTokenizer becomes to return the lowest and some other > results. > However, this can affect the result not only the case you want to solve, > but also the others. > > And, both way require you to re-indexing all documents w

Caffeine Cache and Filter Cache in 8.3

Hi SOLR Community, I have a question about cache implementations based on some seemingly inconsistent documentation I'm looking at. I'm currently inquiring about 8.3, but more generally about solr version 8 too for upgrade planning. In the description in the docs for cache implementations says

Re: Caffeine Cache and Filter Cache in 8.3

che> and here <https://github.com/apache/lucene-solr/blob/e732f943a4df1f2c0692ab475e136f4acc271f59/solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml#L430-L432> )? On Mon, Feb 22, 2021 at 1:16 PM Shawn Heisey wrote: > On 2/22/2021 1:50 PM, Stephen Lewis Bianamara

What controls field cache size and eviction rates?

Hi SOLR Community, I've been trying to understand how the field cache in SOLR manages its evictions, and it is not easily readable from the code or documentation the simple question of when and how something gets evicted from the field cache. This cache also doesn't show hit ratio, total hits, evi

Caffeine Cache Metrics Broken?

Hi SOLR Community, I'm investigating a weird behavior I've observed in the admin page for caffeine cache metrics. It looks to me like on the older caches, warm-up queries were not counted toward hit/miss ratios, which of course makes sense, but on Caffeine cache it looks like they are. I'm using s

Re: Caffeine Cache Metrics Broken?

this more and get back to you if that doesn't end up making sense based on what I observe. Thanks again, Stephen On Tue, Mar 2, 2021 at 6:35 PM Shawn Heisey wrote: > On 3/2/2021 3:47 PM, Stephen Lewis Bianamara wrote: > > I'm investigating a weird behavior I've o

Re: What controls field cache size and eviction rates?

speak to how the field cache handles evictions? Best, Stephen On Wed, Feb 24, 2021 at 4:43 PM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hi SOLR Community, > > I've been trying to understand how the field cache in SOLR manages > its evictions, and it is n

Re: What controls field cache size and eviction rates?

Should say -- Can anyone confirm if it's right *still*, since the article is 10 years old :) On Fri, Mar 5, 2021 at 10:36 AM Stephen Lewis Bianamara < stephen.bianam...@gmail.com> wrote: > Hi SOLR Community, > > Just following up here with an update. I found this article whi

Investigating Seeming Deadlock

Hi SOLR Community, I'm investigating a node on solr 8.3.1 running in cloud mode which appears to have deadlocked, and I'm trying to figure out if this is a known issue or not, and looking for some guidance in understanding both (a) whether this is a resolved issue in future releases or needs a bug