10 billion documents on 12 cores is over 800M documents/shard at best.
This is _very_ aggressive for a shard. Could you give more information
about your setup?
I've seen 250M docs fit in 12G memory. I've also seen 10M documents
strain 32G of memory. Details matter a lot. The only way I've been
Hello,
ENV: solrcloud 6.3
3*dell server
128G 12cores 4.3T /server
3 solr node /server
20G /node (with parameter �Cm 20G)
10 billlion documents totle
Problem:
When we start solrcloud ,the cached index will make memory 98% or
more used . And if we continue to index document
Subject: Re: Solr Caching (documentCache) not working
On 8/17/2015 7:04 AM, Maulin Rathod wrote:
We have observed that Intermittently querying become slower when
documentCache become empty. The documentCache is getting flushed whenever
new document added to the collection
On 8/18/2015 2:30 AM, Daniel Collins wrote:
I think this is expected. As Shawn mentioned, your hard commits have
openSearcher=false, so they flush changes to disk, but don't force a
re-open of the active searcher.
By contrast softCommit, sets openSearcher=true, the point of softCommit is
to
On Mon, Aug 17, 2015 at 4:36 PM, Daniel Collins danwcoll...@gmail.com wrote:
we had to turn off
ALL the Solr caches (warming is useless at that kind of frequency
Warming and caching are related, but different. Caching still
normally makes sense without warming, and Solr is generally written
On 8/17/2015 7:04 AM, Maulin Rathod wrote:
We have observed that Intermittently querying become slower when
documentCache become empty. The documentCache is getting flushed whenever new
document added to the collection.
Is there any way by which we can ensure that newly added documents are
Hi,
We are using solr cloud 5.2 version.
We have observed that Intermittently querying become slower when documentCache
become empty. The documentCache is getting flushed whenever new document added
to the collection.
Is there any way by which we can ensure that newly added documents are
[mailto:apa...@elyograg.org]
Sent: 17 August 2015 19:02
To: solr-user@lucene.apache.org
Subject: Re: Solr Caching (documentCache) not working
On 8/17/2015 7:04 AM, Maulin Rathod wrote:
We have observed that Intermittently querying become slower when
documentCache become empty. The documentCache
On Mon, Aug 17, 2015 at 11:36 PM, Daniel Collins danwcoll...@gmail.com
wrote:
Just to open the can of worms, it *can* be possible to have very low commit
times, we have 250ms currently and are in production with that. But it
does come with pain (no such thing as a free lunch!), we had to turn
Just to open the can of worms, it *can* be possible to have very low commit
times, we have 250ms currently and are in production with that. But it
does come with pain (no such thing as a free lunch!), we had to turn off
ALL the Solr caches (warming is useless at that kind of frequency, it will
Thanks Shawn,
Can you please re-direct me to any wiki which describes (in detail) the
differences between MMapDirectoryFactory and NRTCachingDirectoryFactory? I
found this blog
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html very
helpful which describes about
On 12/8/2014 2:42 AM, Manohar Sripada wrote:
Can you please re-direct me to any wiki which describes (in detail) the
differences between MMapDirectoryFactory and NRTCachingDirectoryFactory? I
found this blog
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html very
helpful
Hi,
I am working on implementing Solr in my product. I have a few questions on
caching.
1. Does posting-list and term-list of the index reside in the memory? If
not, how to load this to memory. I don't want to load entire data, like
using DocumentCache. Either I want to use RAMDirectoryFactory
Hi, Manohar,
1. Does posting-list and term-list of the index reside in the memory? If
not, how to load this to memory. I don't want to load entire data, like
using DocumentCache. Either I want to use RAMDirectoryFactory as the data
will be lost if you restart
If you use MMapDirectory, Lucene
Thanks Micheal for the response.
If you use MMapDirectory, Lucene will map the files into memory off heap
and the OS's disk cache will cache the files in memory for you. Don't use
RAMDirectory, it's not better than MMapDirectory for any use I'm aware of.
Will that mean it will cache the
On 12/4/2014 10:06 PM, Manohar Sripada wrote:
If you use MMapDirectory, Lucene will map the files into memory off heap
and the OS's disk cache will cache the files in memory for you. Don't use
RAMDirectory, it's not better than MMapDirectory for any use I'm aware of.
Will that mean it will
Manuel:
First off, anything that Mike McCandless says about low-level
details should override anything I say. The memory savings
he's talking about there are actually something he tutored me
in once on a chat.
The savings there, as I understand it, aren't huge. For large
sets I think it's a 25%
Great explanation and article.
Yes, this buffer for merges seems very small, and still optimized. Thats
impressive.
Alright, thanks Erick. For the question about memory usage of merges, taken
from Mike McCandless Blog
The big thing that stays in RAM is a logical int[] mapping old docIDs to
new docIDs, but in more recent versions of Lucene (4.x) we use a much more
efficient structure than a simple int[] ...
Inline
On Thu, Jul 11, 2013 at 8:36 AM, Manuel Le Normand
manuel.lenorm...@gmail.com wrote:
Hello,
As a result of frequent java OOM exceptions, I try to investigate more into
the solr jvm memory heap usage.
Please correct me if I am mistaking, this is my understanding of usages for
the heap
Hello,
As a result of frequent java OOM exceptions, I try to investigate more into
the solr jvm memory heap usage.
Please correct me if I am mistaking, this is my understanding of usages for
the heap (per replica on a solr instance):
1. Buffers for indexing - bounded by ramBufferSize
2. Solr
I've just started to read about Solr caching. I want to learn one thing.
Let's assume that I have given 4 GB RAM into my Solr application and I have
10 GB RAM. When Solr caching mechanism starts to work, does it use memory
from that 4 GB part or lets operating system to cache it from 6 GB part
On Apr 17, 2013, at 3:09 PM, Furkan KAMACI wrote:
I've just started to read about Solr caching. I want to learn one thing.
Let's assume that I have given 4 GB RAM into my Solr application and I have
10 GB RAM. When Solr caching mechanism starts to work, does it use memory
from that 4 GB part
4.0 is significantly more efficient memory-wise, both in the usage and
number of objects allocated. See:
http://searchhub.org/dev/2012/04/06/memory-comparisons-between-solr-3x-and-trunk/
Erick
On Sun, Sep 30, 2012 at 12:25 AM, varun srivastava
varunmail...@gmail.com wrote:
Hi Erick,
You
Greetings,
I've recently moved to running some of our Solr (3.6.1) instances
using JDK 7u7 with the G1 GC (playing with max pauses in the 20 to
100ms range). By and large, it has been working well (or, perhaps I
should say that without requiring much tuning it works much better in
general than my
Well, I haven't had experience with JDK7, so I'll skip that part...
But about caches. First, as far as memory is concerned, be
sure to read Uwe's blog about MMapDirectory here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
As to the caches.
Be a little careful here.
Hi Erick,
You mentioned for 4.0 memory pattern is much difference than 3.X . Can you
elaborate whether its worse or better ? Does 4.0 tend to use more memory
for similar index size as compared to 3.X ?
Thanks
Varun
On Sat, Sep 29, 2012 at 1:58 PM, Erick Erickson erickerick...@gmail.comwrote:
. that the reason i am not able
to caching my data.
please advise me if anything need to be done in configuration or in tomcat
configuration.
this will be very help full for me.
-
Regards,
Suneel Pandey
Sr. Software Developer
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-caching
On 3/31/2012 4:30 AM, Suneel wrote:
Hello friends,
I am using DIH for solr indexing. I have 60 million records in SQL which
need to upload on solr. i started caching its smoothly working and memory
consumption is normal, But after some time incrementally memory consumption
going high and
. that the reason i am not able
to caching my data.
please advise me if anything need to be done in configuration or in tomcat
configuration.
this will be very help full for me.
-
Regards,
Suneel Pandey
Sr. Software Developer
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr
There are now two excellent books: Lucene In Action 2 and Solr 1.4
Enterprise Search Server the describe the inners workings of these
technologies and how they fit together.
Otherwise Solr and Lucene knowledge are only available in a fragmented
form across many wiki pages, bug reports and email
Is there any way to analyze or see that which documents are getting cached
by documentCache -
documentCache
class=solr.LRUCache
size=512
initialSize=512
autowarmCount=0/
On Wed, Sep 23, 2009 at 8:10 AM, satya tosatyaj...@gmail.com wrote:
First of all , thanks a lot for
I configured filter cache in solrconfig.xml as here under :
filterCache
class=solr.FastLRUCache
size=16384
initialSize=4096
autowarmCount=4096/
useFilterForSortedQuerytrue/useFilterForSortedQuery
as per
http://wiki.apache.org/solr/SolrCaching#head-b6a7d51521d55fa0c89f2b576b2659f297f9
And
Solr's caches should be transparent - they should only speed up
queries, not change the result of queries.
-Yonik
http://www.lucidimagination.com
On Tue, Sep 22, 2009 at 9:45 AM, satyasundar jena tosatyaj...@gmail.com wrote:
I configured filter cache in solrconfig.xml as here under :
1)Then do you mean , if we delete a perticular doc ,then that is going to be
deleted from
cache also.
2)In solr,is cache storing the entire document in memory or only the
references to
documents in memory.
And how to test this caching after all.
I ll be thankful upon getting an elaboration.
1)Then do you mean , if we delete a perticular doc ,then that is going to
be
deleted from
cache also.
When you delete document, and then COMMIT your changes, new caches will be
warmed up (and prepopulated by some key-value pairs from old instances),
etc:
!-- documentCache caches Lucene
First of all , thanks a lot for the clarification.Is there any way to see,
how this cache is working internally and what are the objects being stored
and how much memory its consuming,so that we can get a clear picture in
mind.And how to test the performance through cache.
On Tue, Sep 22, 2009 at
this message in context:
http://www.nabble.com/Contributions-Needed%3A-Faceting-Performance%2C-SOLR-Caching-tp20058987p20058987.html
Sent from the Solr - User mailing list archive at Nabble.com.
38 matches
Mail list logo
