Re: Is 8.8.x going be stabilized and finalized?

[Shawn Heisey]

On 2/16/2021 7:57 PM, Subhajit Das wrote:

I am planning to use 8.8 line-up for production use.

But recently, a lot of people are complaining on 8.7 and 8.8. Also, there is a 
clearly known issue on 8.8 as well.

Following trends of earlier versions (5.x, 6.x and 7.x), will 8.8 will also be 
finalized?
For 5.x, 5.5.x was last version. For 6.x, 6.6.x was last version. For 7.x, 
7.7.x was last version. It would match the pattern, it seems.
And 9.x is already planned and under development.
And it seems, we require some stability.


All released versions are considered stable.  Sometimes problems are 
uncovered after release.  Sometimes BIG problems.  We try our very best 
to avoid bugs, but achieving that kind of perfection is nearly 
impossible for any software project.


8.8.0 is the most current release.  The 8.8.1 release is underway, but 
there's no way I can give you a concrete date.  The announcement MIGHT 
come in the next few days, but it's always possible it could get pushed 
back.  At this time, the changelog for 8.8.1 has five bugfixes 
mentioned.  It should be more stable than 8.8.0, but it's impossible for 
me to tell you whether you will have any problems with it.


On the dev list, the project is discussing the start of work on the 9.0 
release, but that work has not yet begun.  Even if it started tomorrow, 
it would be several weeks, maybe even a few months, before 9.0 is 
actually released.  On top of the "normal" headaches involved in any new 
major version release, there are some other things going on that might 
further delay 9.0 and future 8.x versions:


* Solr is being promoted from a subproject of Lucene to it's own 
top-level project at Apache.  This involves a LOT of work.  Much of that 
work is administrative in nature, which is going to occupy us and take 
away from time that we might spend working on the code and new releases.
* The build system for the master branch, which is currently versioned 
as 9.0.0-SNAPSHOT, was recently switched from Ant+Ivy to Gradle.  It's 
going to take some time to figure out all the fallout from that migration.
* Some of the devs have been involved in an effort to greatly simplify 
and rewrite how SolrCloud does internal management of a cluster.  The 
intent is much better stability and better performance.  You might have 
seen public messages referring to a "reference implementation."  At this 
time, it is unclear how much of that work will make it into 9.0 and how 
much will be revealed in later releases.  We would like very much to 
include at least the first phase in 9.0 if we can.


From what I have seen over the last several years as one of the 
developers on this project, it is likely that 8.9 and possibly even 8.10 
and 8.11 will be released before we see 9.0.  Releases are NOT made on a 
specific schedule, so I cannot tell you which versions you will see or 
when they might happen.


I am fully aware that despite typing quite a lot of text here, that I 
provided almost nothing in the way of concrete information that you can 
use.  Sorry about that.


Thanks,
Shawn

Is 8.8.x going be stabilized and finalized?

[Subhajit Das]

Hi there,

I am planning to use 8.8 line-up for production use.

But recently, a lot of people are complaining on 8.7 and 8.8. Also, there is a 
clearly known issue on 8.8 as well.

Following trends of earlier versions (5.x, 6.x and 7.x), will 8.8 will also be 
finalized?
For 5.x, 5.5.x was last version. For 6.x, 6.6.x was last version. For 7.x, 
7.7.x was last version. It would match the pattern, it seems.
And 9.x is already planned and under development.
And it seems, we require some stability.

Thanks in advance.

Re: tlog keeps growing

[mmb1234]

Erik,

Looks like we're also running into this issue.
https://www.mail-archive.com/solr-user@lucene.apache.org/msg153798.html

Is there any think we can do to remedy this besides a node restart, which
causes leader re-election on the good shards which causes them to also
become un-operational?





--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Elevation in dataDir in Solr Cloud

[Chris Hostetter]

: Of course, here is the full stack trace (collection 'techproducts' with
: just one core to make it easier):

Ah yeah ... see -- this looks like a mistake introduced at some point...

: Caused by: org.apache.solr.core.SolrResourceNotFoundException: Can't
: find resource 'elevate.xml' in classpath or
: '/configs/techproductsConfExp', cwd=/usr/share/solr-7.7.2/server
:   at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:130)
:   at 
org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:362)
:   at org.apache.solr.core.Config.(Config.java:120)
:   at org.apache.solr.core.Config.(Config.java:90)
:   at 
org.apache.solr.handler.component.QueryElevationComponent.loadElevationProvider(QueryElevationComponent.java:366)

...this bit of code is *expecting* to be able to init a Config object from 
the SolrResourceLoader, even thought this bit of code...

:   at 
org.apache.solr.handler.component.QueryElevationComponent.getElevationProvider(QueryElevationComponent.java:321)
:   at 
org.apache.solr.handler.component.QueryElevationComponent.loadElevationConfiguration(QueryElevationComponent.java:259)

...has already established that there is no "Config" file available from 
the resource loader, and we should be initializing an ElevationProvider 
that can raed from the data dir.  9and this code seems to be unchanged on 
branch_8x)

Can you please file a jira pointing out that this doesn't work along with 
the full stack trace, and then add a comment copy/pasting my comments here 
that the code makes no sense?


I'm not sure if/when someone who understands the code well enough will be 
able to help fix this (and write a test for it) ... was the experiment 
/ work around i suggested viable? ...


: > I don't know if it will work, but one thing you might want to experiment
: > with is putting your elevate.xml back the configset in zk, and updating it
: > on the fly in zk -- then see if it gets reloaded by each core the next
: > time the index changes (NOTE that there will almost certainly need to be
: > an index change for it to re-load, since I don't see any indication that
: > it's watching for changes in zk)
: >
: > FWIW: the way most people seem to be using QEC these days is to have an
: > empty elevate.xml file, and then have their application use some other
: > key/val store, or more complex matching logic, to decide which documents
: > to elevate, and then use the "elevateIds" param to pass that info to solr.


-Hoss
http://www.lucidworks.com/

Re: Down Replica is elected as Leader (solr v8.7.0)

[mmb1234]

> Are yours growing always, on all nodes, forever?  Or is it one or two who
ends up in a bad state?

Randomly on some of the shards and some of the followers in the collection.
Then whichever tlog was open on follower when it was the leader, that one
doesn't stops growing. And that shard had active ingestion at a high rate.

If we now add more shards, or do a cluster rebalance, the collection is
unsusable and causes a production query and ingest outage. Very painful
manual restoration twice a day.




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Down Replica is elected as Leader (solr v8.7.0)

[matthew sporleder]

I've run into this (or similar) issues in the past (solr6? I don't
remember exactly) where tlogs get stuck either growing indefinitely
and/or refusing to commit on restart.

What I ended up doing was writing a monitor to check for the number of
tlogs and alert if they got over some limit (100 or whatever) and then
I could stay ahead of the issue by rebuilding individual nodes
as-needed.

Are yours growing always, on all nodes, forever?  Or is it one or two
who ends up in a bad state?

On Tue, Feb 16, 2021 at 3:57 PM mmb1234  wrote:
>
> Looks like the problem is related to tlog rotation on the follower shard.
>
> We did the following for a specific shard.
>
> 0. start solr cloud
> 1. solr-0 (leader), solr-1, solr-2
> 2. rebalance to make solr-1 as preferred leader
> 3. solr-0, solr-1 (leader), solr-2
>
> The tlog file on solr-0 kept on growing infinitely (100s of GBs) until we
> shut the cluster and dropped all shards (manually).
>
> Only way to "restart" tlog rotation on solr-0 (follower) was to issue
> /admin/cores/action=RELOAD=x atleast twice when the tlog size was
> small (in MBs).
>
> Also if rebalance is is issued to select solr-0 as a leader, leader election
> never completes.
>
> solr-0 output after step (3) above.
>
> solr-0
> 2140856 ./data2/mydata_0_e000-/tlog
> 2140712 ./data2/mydata_0_e000-/tlog/tlog.021
>
> solr-1 (leader)
> 35268   ./data2/mydata_0_e000-/tlog
> 35264   ./data2/mydata_0_e000-/tlog/tlog.055
>
> solr-2
> 35256   ./data2/mydata_0_e000-/tlog
> 35252   ./data2/mydata_0_e000-/tlog/tlog.054
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Down Replica is elected as Leader (solr v8.7.0)

[mmb1234]

Looks like the problem is related to tlog rotation on the follower shard.

We did the following for a specific shard.

0. start solr cloud
1. solr-0 (leader), solr-1, solr-2
2. rebalance to make solr-1 as preferred leader
3. solr-0, solr-1 (leader), solr-2

The tlog file on solr-0 kept on growing infinitely (100s of GBs) until we
shut the cluster and dropped all shards (manually).

Only way to "restart" tlog rotation on solr-0 (follower) was to issue
/admin/cores/action=RELOAD=x atleast twice when the tlog size was
small (in MBs).

Also if rebalance is is issued to select solr-0 as a leader, leader election
never completes.

solr-0 output after step (3) above.

solr-0
2140856 ./data2/mydata_0_e000-/tlog
2140712 ./data2/mydata_0_e000-/tlog/tlog.021

solr-1 (leader)
35268   ./data2/mydata_0_e000-/tlog
35264   ./data2/mydata_0_e000-/tlog/tlog.055

solr-2
35256   ./data2/mydata_0_e000-/tlog
35252   ./data2/mydata_0_e000-/tlog/tlog.054



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Down Replica is elected as Leader (solr v8.7.0)

[mmb1234]

Looks like the problem is related to tlog rotation on the follower shard.

We did the following for a specific shard.

0. start solr cloud
1. solr-0 (leader), solr-1, solr-2
2. rebalance to make solr-1 as preferred leader
3. solr-0, solr-1 (leader), solr-2

The tlog file on solr-0 kept on growing infinitely (100s of GBs) until we
shut the cluster and dropped all shards (manually).

Only way to "restart" tlog rotation on solr-0 (follower) was to issue
/admin/cores/action=RELOAD=x atleast twice when the tlog size was
small (in MBs).

Also if rebalance is is issued to select solr-0 as a leader, leader election
never completes.

solr-0 output after step (3) above.

solr-0
2140856 ./data2/mydata_0_e000-/tlog
2140712 ./data2/mydata_0_e000-/tlog/tlog.021

solr-1 (leader)
35268   ./data2/mydata_0_e000-/tlog
35264   ./data2/mydata_0_e000-/tlog/tlog.055

solr-2
35256   ./data2/mydata_0_e000-/tlog
35252   ./data2/mydata_0_e000-/tlog/tlog.054




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SolrJ: SolrInputDocument.addField()

[Shawn Heisey]

On 2/15/2021 10:17 AM, Steven White wrote:

Yes, I have managed schema enabled like so:

   
 true
 cp-schema.xml
   

The reason why I enabled it is so that I can dynamically customize the
schema based on what's in the DB.  So that I can add fields to the schema
dynamically.


A managed/mutable schema is a configuration detail that's separate from 
(and required by) the update processor that guesses unknown fields.  It 
has been the default schema factory used in out-of-the box 
configurations for quite a while.



I guess a better question, to meet my need, is this: how do I tell Solr, in
schema-less mode, to use *my* defined field-type whenever it needs to
create a new field?


The config for that is described here:

https://lucene.apache.org/solr/guide/8_6/schemaless-mode.html#enable-field-class-guessing

It is a bad idea to rely on field guessing for a production index.  Even 
the most carefully designed configuration cannot get it right every 
time.  You're very likely to run into situations where the software's 
best guess turns out to be wrong for your needs.  And then you're forced 
into what you should have done in the first place -- manually fixing the 
definition for that field, which usually also requires reindexing from 
scratch.


One counter-argument to what I stated in the last paragraph that 
frequently comes up is "my data is very well curated and consistent." 
But if that is the case, then you will know what fields and types are 
required *in advance* and you can easily construct a schema yourself 
before sending any data for indexing -- no guessing required.


Thanks,
Shawn

Re: Meaning of "Index" flag under properties and schema

[Shawn Heisey]

On 2/16/2021 9:16 AM, ufuk yılmaz wrote:

I didn’t realise that, sorry. The table is like:

Flags   Indexed Tokenized   Stored  UnInvertible

Properties  YesYesYes Yes
Schema  YesYesYes Yes
Index   YesYesYes NO

Problematic collection has a Index row under Schema row. No other collection 
has it. I was asking about what the “Index” meant


I am not completely sure, but I think that row means the field was found 
in the actual Lucene index.


In the original message you mentioned "weird exceptions" but didn't 
include any information about them.  Can you give us those exceptions, 
and the requests that caused them?


Thanks,
Shawn

Re: Significant terms expression giving error "length needs to be >= 1"

[Joel Bernstein]

Can you include the stack trace from the logs?


Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Feb 15, 2021 at 3:53 PM ufuk yılmaz 
wrote:

> We have a SolrCloud cluster, version 8.4
>
> At the customer’s site there’s a collection with very few documents,
> around 12. We usually have collections with hundreds of millions of
> documents, so that collection is a bit of an exception.
>
> When I send a significantTerms streaming expression it immediately gets a
> “IllegalArgumentException("length needs to be >= 1")” from that
> collection’s shard. I took a look at it, but it doesn’t seem to have
> anything different in it from other collections. We also don’t get that
> exception in our own cluster, which is very similar to customer’s.
>
> I found the exception log in
> “lucene-solr\lucene\core\src\java\org\apache\lucene\util\SparseFixedBitSet.java”
> but I don’t have enough knowlegde on the inner workings of the streaming
> expression to interpret it.
>
> What may cause this?
>
> --ufuk
>
> Sent from Mail for Windows 10
>
>

RE: Meaning of "Index" flag under properties and schema

[ufuk yılmaz]

I didn’t realise that, sorry. The table is like:

Flags   Indexed Tokenized   Stored  UnInvertible

Properties  YesYesYes Yes
Schema  YesYesYes Yes
Index   YesYesYes NO


Problematic collection has a Index row under Schema row. No other collection 
has it. I was asking about what the “Index” meant

-ufuk

Sent from Mail for Windows 10

From: Charlie Hull
Sent: 16 February 2021 18:48
To: solr-user@lucene.apache.org
Subject: Re: Meaning of "Index" flag under properties and schema

This list strips attachments so you'll have to figure out another way to 
show the difference,

Cheers

Charlie

On 16/02/2021 15:16, ufuk yılmaz wrote:
>
> There’s a collection at our customer’s site giving weird exceptions 
> when a particular field is involved (asked another question detailing 
> that).
>
> When I inspected it, there’s only one difference between it and other 
> dozens of fine working collections, which is,
>
> A text_general field in all other collections has the above 
> configuration without my artsy paint edits, but only that problematic 
> collection has an “index” flag with indexed tokenized and stored 
> checked. I never saw this “Index” flag before. What does it mean?
>
> Sent from Mail  for 
> Windows 10
>

-- 
Charlie Hull - Managing Consultant at OpenSource Connections Limited 

Founding member of The Search Network  
and co-author of Searching the Enterprise 

tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828

Re: Meaning of "Index" flag under properties and schema

[Charlie Hull]

This list strips attachments so you'll have to figure out another way to 
show the difference,


Cheers

Charlie

On 16/02/2021 15:16, ufuk yılmaz wrote:


There’s a collection at our customer’s site giving weird exceptions 
when a particular field is involved (asked another question detailing 
that).


When I inspected it, there’s only one difference between it and other 
dozens of fine working collections, which is,


A text_general field in all other collections has the above 
configuration without my artsy paint edits, but only that problematic 
collection has an “index” flag with indexed tokenized and stored 
checked. I never saw this “Index” flag before. What does it mean?


Sent from Mail  for 
Windows 10




--
Charlie Hull - Managing Consultant at OpenSource Connections Limited 

Founding member of The Search Network  
and co-author of Searching the Enterprise 


tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828

Meaning of "Index" flag under properties and schema

[ufuk yılmaz]

There’s a collection at our customer’s site giving weird exceptions when a 
particular field is involved (asked another question detailing that).

When I inspected it, there’s only one difference between it and other dozens of 
fine working collections, which is,


A text_general field in all other collections has the above configuration 
without my artsy paint edits, but only that problematic collection has an 
“index” flag with indexed tokenized and stored checked. I never saw this 
“Index” flag before. What does it mean?




Sent from Mail for Windows 10

Re: SolrJ: SolrInputDocument.addField()

[Jimi Hullegård]

Hi Steven,

Just a thought, from someone who never have used schema-less mode: Have you 
considered using a regular schema file, with a bunch of dynamicField 
definitions? Then you can for example define a dynamic boolean field like this:



Then, when you index the data, you can append "_b" to the field name for all 
boolean values. So if you for example want to index searchable: true, then you 
send that data with the fieldname "searchable_b" and solr will index it as a 
Boolean field.

/Jimi

Steven White wrote:
>
> Hi Shawn,
>
> Yes, I have managed schema enabled like so:
>
>  
> true
> cp-schema.xml
>   
>
> The reason why I enabled it is so that I can dynamically customize the schema 
> based on what's in the DB.  So that I can add fields to the schema 
> dynamically.
>
> I didn't know about the field "guessing" part.  Now that I know I see this in 
> my solrconfig.xml file:
>
>default="${update.autoCreateFields:true}"
>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
> 
> 
> 
>   
>
> If I remove this block, what will happen?
>
> I guess a better question, to meet my need, is this: how do I tell Solr, in 
> schema-less mode, to use *my* defined field-type whenever it needs to create 
> a new field?
>
> I'm on Solr 8.6.1 and the link at
> https://lucene.apache.org/solr/guide/8_6/schema-factory-definition-in-solrconfig.html#schema-factory-definition-in-solrconfig
> doesn't offer much help.
>
> Thanks
>
> Steven
Svenskt Näringsliv är företagsamhetens röst i Sverige. Vi samverkar med 50 
arbetsgivar- och branschorganisationer och är den gemensamma rösten för 60 000 
företag med nästan 2 miljoner medarbetare. Vår uppgift är att tala för alla 
företag och branscher, även de som ännu inte finns men som kan uppstå om 
förutsättningarna är de rätta. Ett bättre företagsklimat för ett bättre 
Sverige. Det är vårt uppdrag.

Svenskt Näringsliv behandlar dina personuppgifter i enlighet med GDPR. Här kan 
du läsa mer om vår behandling och dina rättigheter, 
Integritetspolicy