Atomic Update Failures with Nested Schema and Lazy Field Loading
Hi, I've encountered another issue that might be related to nested schema. Not always, but many times atomic updates fails for some shards with the message "TransactionLog doesn't know how to serialize class org.apache.lucene.document.LazyDocument$LazyField". The client retrieves documents, constructs bulk of input documents (id and changed field), adds the bulk and finally sends an explicit commit. I checked both options: 1. Set false. 2. Set true but removed child documents. In both cases atomic update worked without any errors. This might suggest that there is an issue with this combination. Thanks in advance, Ronen. This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.
Re: No Live server exception: Solr Cloud 6.6.6
Hi Guys, Any update. On Tue, 5 Jan 2021 at 18:06, Ritvik Sharma wrote: > Hi Guys > > Happy New Year. > > We are trying to move to solr cloud 6.6.6 as we are using same version > master-slave arch. > > solr cloud: 6.6.6 > zk: 3.4.10 > > We are facing few errors > 1. Every time we upload a model-store using curl XPUT command , it is > showing at that time but after reloading collection , it is removed > automatically. > > 2.While querying the data, we are getting below exception, > > "msg": "org.apache.solr.client.solrj.SolrServerException: No live > SolrServers available to handle this request:[ > http://x.x.x.x:8983/solr/solrcollection_shard1_replica2, > http://x.x.x.y:8983/solr/solrcollection_shard1_replica1]","trace": > "org.apache.solr.common.SolrException: > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > available to handle this request:[ > http://x.x.x.x:8983/solr/solrcollection_shard1_replica2, > http://x.x.x.y:8983/solr/solrcollection_shard1_replica1]\n\tat > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:416)\n\tat > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)\n\tat > org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)\n\tat > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)\n\tat > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)\n\tat > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)\n\tat > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)\n\tat > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)\n\tat > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat > > > > >
How pull replica works
I want to know how pull replica replicate from leader in real? Does internally admin API get data from the leader in form of batches? Regards, Abhishek
Re: maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
On 1/5/2021 8:26 AM, dinesh naik wrote:
Hi all,
I want to update the maxBooleanClauses to 2048 (from default value 1024).
Below are the steps tried:
1. updated solrconfig.xml :
${solr.max.booleanClauses:2048}
You need to update EVERY solrconfig.xml that the JVM is loading for this
to actually work.
maxBooleanClauses is an odd duck. At the Lucene level, where this
matters, it is a global (JVM-wide) variable. So whenever Solr sets this
value, it applies to ALL of the Lucene indexes that are being accessed
by that JVM.
When you havet multiple Solr cores, the last core that was loaded will
set the max clauses value for ALL cores. If any of your solrconfig.xml
files don't have that config, then it will be set to the default of 1024
when that core is loaded or reloaded. Leaving the config out is not a
solution.
So if any of your configs don't have the setting or set it to something
lower than you need, you run the risk of having the max value
incorrectly set across the board.
Here are the ways that I think this could be fixed:
1) Make the value per-index in Lucene, (or maybe even per-query) instead
of global.
2) Have Solr only change the global Lucene value if the config is
*higher* than the current global value.
3) Eliminate the limit entirely. Remove the config option from Solr and
have Solr hard-set it to the maximum value.
4) Move the maxBooleanClauses config to solr.xml instead of solrconfig.xml
I think that option 1 is the best way to do it, but this problem has
been around for many years, so it's probably not easy to do. I don't
think it's going to happen. There are a number of existing issues in
the Solr bug tracker for changing how the limit is configured.
2. updated solr.xml :
${solr.max.booleanClauses:2048}
I don't think it's currently possible to set the value with solr.xml.
Thanks,
Shawn
Re: maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
I experienced the same thing in solr-8.7 , it worked for me using system
property.
Set system property in solr.in.sh file
On Tue, Jan 5, 2021 at 8:58 PM dinesh naik
wrote:
> Hi all,
> I want to update the maxBooleanClauses to 2048 (from default value 1024).
> Below are the steps tried:
> 1. updated solrconfig.xml :
> ${solr.max.booleanClauses:2048}
>
> 2. updated solr.xml :
> ${solr.max.booleanClauses:2048}
>
> 3. Restarted the solr nodes.
>
> 4. Tried query with more than 2000 OR clauses and getting below waring
> message in solr logs:
>
> 2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
> o.a.s.c.SolrConfig solrconfig.xml: of 2048 is greater
> than global limit of 1024 and will have no effect
>
> 2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
> o.a.s.c.SolrConfig set 'maxBooleanClauses' in solr.xml to increase global
> limit
>
> Note: In 7.6.1 version we just need to change the solrconfig.xml and it
> works.
>
> Kindly let me know if i am missing something for making it work in 8.4.1
> version.
> --
> Best Regards,
> Dinesh Naik
>
Re: Identifying open segments.
Are you trying to copy the index by an external process not running in the Solr JVM? I believe this is risky if the Solr JVM is running at the same time. For example segments can be deleted by Solr. There might also be closed segments that you do not need but that are still on the disk (no longer part of the current commit point). You could look at backup options in Solr, I believe they basically do what you need (I'm not familiar with what's available but I'm sure you can find the info). Ilan On Tue, Jan 5, 2021 at 12:46 PM Jacob Ward wrote: > > Hello, > > I am looking for a way to identify the open segment files in a lucene > index, so that I can export only the closed segments (and the segmentsN > file). My current ideas are: > > - Ignore any segment files newer than the segmentsN file. > OR > - Open the segmentsN file using Lucene core's SegmentInfos class (which I > presume would allow me to identify which are the closed segments). > > Could anyone provide suggestions on how to do this best? Ideally I'd like > to do this without the SegmentInfos class if there is a suitable method. > > Thanks. > > -- > > Jacob Ward|Graduate Data Infrastructure Engineer > > jw...@brandwatch.com > > > NEW YORK | BOSTON | BRIGHTON | LONDON | BERLIN | STUTTGART | > PARIS | SINGAPORE | SYDNEY
Re:Possible bug on LTR when using solr 8.6.3 - index out of bounds DisiPriorityQueue.add(DisiPriorityQueue.java:102)
Hello Florin Babes,
Thanks for this detailed report! I agree you experiencing
ArrayIndexOutOfBoundsException during SolrFeature computation sounds like a
bug, would you like to open a SOLR JIRA issue for it?
Here's some investigative ideas I would have, in no particular order:
Reproducibility: if a failed query is run again, does it also fail second time
around (when some caches may be used)?
Data as a factor: is your setup single-sharded or multi-sharded? in a
multi-sharded setup if the same query fails on some shards but succeeds on
others (and all shards have some documents that match the query) then this
could support a theory that a certain combination of data and features leads to
the exception.
Feature vs. Model: you mention use of a MultipleAdditiveTrees model, if the
same features are used in a LinearModel instead, do the same errors happen? or
if no model is used but only feature extraction is done, does that give errors?
Identification of the troublesome feature(s): narrowing down to a single
feature or a small combination of features could make it easier to figure out
the problem. assuming the existing logging doesn't identify the features,
replacing the org.apache.solr.ltr.feature.SolrFeature with a
com.mycompany.solr.ltr.feature.MySolrFeature containing instrumentation could
provide insights e.g. the existing code [2] logs feature names for
UnsupportedOperationException and if it also caught
ArrayIndexOutOfBoundsException then it could log the feature name before
rethrowing the exception.
Based on your detail below and this [3] conditional in the code probably at
least two features will be necessary to hit the issue, but for investigative
purposes two features could still be simplified potentially to effectively one
feature e.g. if one feature is a SolrFeature and the other is a ValueFeature or
if featureA and featureB are both SolrFeature features with _identical_
parameters but different names.
Hope that helps.
Regards,
Christine
[1]
https://lucene.apache.org/solr/guide/8_6/learning-to-rank.html#extracting-features
[2]
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.3/solr/contrib/ltr/src/java/org/apache/solr/ltr/feature/SolrFeature.java#L243
[3]
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.3/solr/contrib/ltr/src/java/org/apache/solr/ltr/LTRScoringQuery.java#L520-L525
From: solr-user@lucene.apache.org At: 01/04/21 17:31:44To:
solr-user@lucene.apache.org
Subject: Possible bug on LTR when using solr 8.6.3 - index out of bounds
DisiPriorityQueue.add(DisiPriorityQueue.java:102)
Hello,
We are trying to update Solr from 8.3.1 to 8.6.3. On Solr 8.3.1 we are
using LTR in production using a MultipleAdditiveTrees model. On Solr 8.6.3
we receive an error when we try to compute some SolrFeatures. We didn't
find any pattern of the queries that fail.
Example:
We have the following query raw parameters:
q=lg cx 4k oled 120 hz -> just of many examples
term_dq=lg cx 4k oled 120 hz
rq={!ltr model=model reRankDocs=1000 store=feature_store
efi.term=${term_dq}}
defType=edismax,
mm=2<75%
The features are something like this:
{
"name":"similarity_query_fileld_1",
"class":"org.apache.solr.ltr.feature.SolrFeature",
"params":{"q":"{!dismax qf=query_field_1 mm=1}${term}"},
"store":"feature_store"
},
{
"name":"similarity_query_field_2",
"class":"org.apache.solr.ltr.feature.SolrFeature",
"params":{"q":"{!dismax qf=query_field_2 mm=5}${term}"},
"store":"feature_store"
}
We are testing ~6300 production queries and for about 1% of them we receive
that following error message:
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","java.lang.ArrayIndexOutOfBoundsException"],
"msg":"java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds
for length 2",
The stacktrace is :
org.apache.solr.common.SolrException:
java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
at org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:154)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:159
9)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1413
)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
at
org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryC
omponent.java:1513)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:403
)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.
java:360)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java
:214)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2627)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:795)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:568)
at
maxBooleanClauses change in solr.xml not reflecting in solr 8.4.1
Hi all,
I want to update the maxBooleanClauses to 2048 (from default value 1024).
Below are the steps tried:
1. updated solrconfig.xml :
${solr.max.booleanClauses:2048}
2. updated solr.xml :
${solr.max.booleanClauses:2048}
3. Restarted the solr nodes.
4. Tried query with more than 2000 OR clauses and getting below waring
message in solr logs:
2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
o.a.s.c.SolrConfig solrconfig.xml: of 2048 is greater
than global limit of 1024 and will have no effect
2021-01-05 14:03:59.603 WARN (qtp1545077099-27) x:col1_shard1_replica_n3
o.a.s.c.SolrConfig set 'maxBooleanClauses' in solr.xml to increase global
limit
Note: In 7.6.1 version we just need to change the solrconfig.xml and it
works.
Kindly let me know if i am missing something for making it work in 8.4.1
version.
--
Best Regards,
Dinesh Naik
No Live server exception: Solr Cloud 6.6.6
Hi Guys Happy New Year. We are trying to move to solr cloud 6.6.6 as we are using same version master-slave arch. solr cloud: 6.6.6 zk: 3.4.10 We are facing few errors 1. Every time we upload a model-store using curl XPUT command , it is showing at that time but after reloading collection , it is removed automatically. 2.While querying the data, we are getting below exception, "msg": "org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[ http://x.x.x.x:8983/solr/solrcollection_shard1_replica2, http://x.x.x.y:8983/solr/solrcollection_shard1_replica1]","trace": "org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[ http://x.x.x.x:8983/solr/solrcollection_shard1_replica2, http://x.x.x.y:8983/solr/solrcollection_shard1_replica1]\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:416)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat
Identifying open segments.
Hello, I am looking for a way to identify the open segment files in a lucene index, so that I can export only the closed segments (and the segmentsN file). My current ideas are: - Ignore any segment files newer than the segmentsN file. OR - Open the segmentsN file using Lucene core's SegmentInfos class (which I presume would allow me to identify which are the closed segments). Could anyone provide suggestions on how to do this best? Ideally I'd like to do this without the SegmentInfos class if there is a suitable method. Thanks. -- Jacob Ward|Graduate Data Infrastructure Engineer jw...@brandwatch.com NEW YORK | BOSTON | BRIGHTON | LONDON | BERLIN | STUTTGART | PARIS | SINGAPORE | SYDNEY
Sending compressed (gzip) UpdateRequest with SolrJ
Hello, I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ. Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default? - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin) - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 By the way, is there any documentation about javabin? I could only find one on the "old wiki". Thanks, Gaël
