Sort on internal lucene ID led to OOM

[sanjay dutt]

Hello,
We were investigating one HEAP DUMP in which FielsCaceImpl has occupied around 
5GB. Upon further investigation, I got to know that 
FieldCacheImpl$SortedDocValues occupies 90% of the memory where 
FieldCacheImpl$CacheKey is "id".
When I checked the logs I was trying to find any query in which there is any 
sort on id field. But instead I find few queries in which we are sorting on 
_docid_ lucene internal id.
Can sort on _docid_ cause OOM?
And if I will enable docValues for uniqueKey(id) will that solve my problem?
Regards,SanjaySent from Yahoo Mail on Android

Slow query response from SOLR 5.4.1

[Abhijit Pawar]

Hello,

I am seeing a performance issue in querying in one of the SOLR servers -
instance version 5.4.1.
Total number of documents indexed are 20K plus.
Data returned for this particular query is just as less as 22 documents
however it takes almost 2 minutes to get the results back.

Is there a way to improve on performance of query - in general the query
response time is slow..

I have most of the fields which are stored and indexed both.I can take off
some fields which are just needed to be indexed however those are not many
fields.

Can I do something solrconfig.xml in terms of cache or something else?

Any suggestions?

Thanks!!

Re: How to route requests to a specific core of a node hosting multiple shards?

[Jason J Baik]

Thanks for looking into this @Erick Erickson.
What'd be the proper way to get David Smiley's attention on this issue? A
JIRA ticket?

As for the performance difference, we haven't had a chance to test it.
We're still in the dev phase for migrating to solr 8, so we'll run our
benchmarks afterward, and try to see if it's a serious problem.

On Mon, Jul 20, 2020 at 10:43 AM Erick Erickson 
wrote:

> Hmm, ok.
>
> I’d have to defer to David Smiley about whether that was an intended
> change.
>
> I’m curious whether you can actually measure the difference in
> performance. If
> you can then that changes the urgency. Of course it’ll be a little more
> expensive
> for the replica serving shard2 on that machine to forward it to the replica
> serving shard1, but since it’s not going across the network IDK if it’s a
> consequential difference.
>
> Best,
> Erick
>
> > On Jul 20, 2020, at 10:04 AM, Jason J Baik 
> wrote:
> >
> > Our use case here is that we want to highlight a single document (against
> > user-provided keywords), and we know the document's unique key already.
> > So this is really not a distributed query, but more of a get by id, but
> we
> > use SolrClient.query() for highlighting capabilities.
> > And since we know the unique key, for speed gains, we've been making use
> of
> > the "_route_" param to limit the request to the shard containing the
> > document.
> >
> > Our use case aside, SOLR-11444
> >  generally seems to
> be at
> > odds with the advertised use of the "_route_" param
> >
> https://lucene.apache.org/solr/guide/7_5/solrcloud-query-routing-and-read-tolerance.html#_route_-parameter
> > .
> > Solr is routing the request to the correct "node", but it no longer
> routes
> > to the correct "shard" on that node?
> >
> >
> > On Mon, Jul 20, 2020 at 9:33 AM Erick Erickson 
> > wrote:
> >
> >> First I want to check if this is an XY problem. Why do you want to do
> this?
> >>
> >> If you’re using CloudSolrClient, requests are automatically load
> balanced.
> >> And
> >> even if you send a top-level request (assuming you do NOT set
> >> distrib=false),
> >> then the request may be forwarded to another Solr node anyway. This is
> to
> >> handle the case where people are sending requests to a specific node,
> you
> >> don’t
> >> really want that node doing all the aggregating.
> >>
> >> Of course if you’re using an external load balancer, you can avoid all
> >> that.
> >>
> >> I’m not sure what the value is of sending a general request to a
> specific
> >> core in the same JVM. A “node” is really Solr running in a JVM, so there
> >> may be multiple of these on a particular machine, but the resolution
> >> takes that into account.
> >>
> >> If you have reason to ping a specific replica _only_ (I’ve often done
> this
> >> for
> >> troubleshooting), address the full replica and add “distrib=false”, i.e.
> >> http://…../solr/collection1_shard1_replica1?q=*:*=false
> >>
> >> Best,
> >> Erick
> >>
> >>> On Jul 20, 2020, at 9:02 AM, Jason J Baik 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> After upgrading from Solr 6.6.2 to 7.6.0, we're seeing an issue with
> >>> request routing in CloudSolrClient. It seems that we've lost the
> ability
> >> to
> >>> route a request to a specific core of a node.
> >>>
> >>> For example, if a host is serving shard 1 core 1, and shard 2 core
> >>> 1, @6.6.2, adding a "_route_="
> >>> param was sufficient for CloudSolrClient to figure out the request
> should
> >>> go to shard 1 core 1, but @7.6.0, the request is routed to one of them
> >>> randomly.
> >>>
> >>> It seems the core-level url resolution has been removed from
> >>> CloudSolrClient at commit e001f352895c83652c3cf31e3c724d29a46bb721
> around
> >>> L1053, as part of SOLR-11444
> >>> . The url the
> request
> >> is
> >>> sent to is now constructed only to the node level, and no longer to the
> >>> core level.
> >>>
> >>> There's a related issue for this at SOLR-10695
> >>> , and SOLR-9063
> >>>  but not quite the
> >> same.
> >>> Can somebody please advise what the new way to achieve this nowadays
> is?
> >>
> >>
>
>

Re: org.apache.lucene.util.fst.FST taking up lot of Java Heap Memory

[Erick Erickson]

Possibly https://issues.apache.org/jira/browse/LUCENE-9286? I’m not quite sure 
whether this only affects 8.5 or not.

Also: https://issues.apache.org/jira/browse/LUCENE-8920 and 
https://issues.apache.org/jira/browse/LUCENE-9398 might be of interest.

I have no idea whether these are relevant or whether you just need more heap…

Best,
Erick

> On Aug 3, 2020, at 9:42 PM, sanjay dutt  
> wrote:
> 
> Hello Solr community,
> On our Production SolrCloud Server, OutOfMemory has been occurring on lot of 
> instances. When I download the HEAP DUMP and analyzed it. I got to know that 
> in multiple HEAP DUMPS there are lots of instances of 
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader  which has the 
> highest retained heap memory and further I have checked the 
> outgoing-reference for those objects, the  org.apache.lucene.util.fst.FST is 
> the one which occupy 90% of the heap memory.
> it's like 
> Production HEAP memory :- 12GBout of which  
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader total retained heap 
> :- 7-8 GB(vary from instance to instance)and org.apache.lucene.util.fst.FST 
> total retained heap :- 6-7 GB
> Upon further looking I have calculated the total retained heap for 
> FieldReader.fieldInfo.name="my_field" is around 7GB. Now this is the same 
> reader which also contains reference to org.apache.lucene.util.fst.FST.
> Now "my_field" is the field on which we are performing spatial searches. Is 
> spatial searches use FST internally and hence we are seeing lot of heap 
> memory used by FST.l only.
> IS there any way we can optimize the spatial searches so that it take less 
> memory.
> Can someone please give me any pointer that from where Should I start looking 
> to debug the above issue. 
> Thanks and Regards,Sanjay Dutt
> Sent from Yahoo Mail on Android

Re: solr classloader and cores

[Vincenzo D'Amore]

Hi all,

sorry for not being clear in the first instance.
What I'm trying to say is, what happens when a core loads a library,
are the classes present in the library even available for other cores? I
think no.
But if there is some sort of encapsulation this should be clear in the
documentation.
In this way one can deploy different versions of a plugin/extension for
each core without being worried of collisions.


On Tue, Aug 4, 2020 at 12:12 PM Vincenzo D'Amore  wrote:

> Hi all,
>
> I know that Solr allows loading classes by defining the  directive
> into the solrconfig.xml.
> What it's not clear is if there is a separate (private?) classloader for
> each core.
> Is this correct?
>
> Best regards,
> Vincenzo
>
>
> --
> Vincenzo D'Amore
>
>

-- 
Vincenzo D'Amore

solr classloader and cores

[Vincenzo D'Amore]

Hi all,

I know that Solr allows loading classes by defining the  directive
into the solrconfig.xml.
What it's not clear is if there is a separate (private?) classloader for
each core.
Is this correct?

Best regards,
Vincenzo


-- 
Vincenzo D'Amore

Solr-7 CorruptIndexException: checksum failed (hardware problem?)

[slly]

Hello everyone, I use the version of Solr-7.7.3. 


The following error occurred during the index write phase, but after restarting 
the Solr service, the file was deleted, and access to the index has also been 
restored.


Has anyone ever encountered this mistake?




Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is 
closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694)
at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1613)
at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608)
at 
org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:969)
at 
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341)
at 
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288)
at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235)
... 67 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed 
(hardware problem?) : expected=dd0fcc0f actual=2317b057 
(resource=BufferedChecksumIndexInput(_2g_Lucene50_0.tim))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
at 
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526)
at 
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:309)
at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:339)
at 
org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterFieldsProducer.checkIntegrity(PerFieldMergeState.java:271)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
at 
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:231)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
at 
org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:224)
at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)


Thanks.