bq: We are indexing with autocommit at 30 minutes
OK, check the size of your tlogs. What this means is that all the
updates accumulate for 30 minutes in a single tlog. That tlog will be
closed when autocommit happens and a new one opened for the
next 30 minutes. The first tlog won't be purged unti
One option:
First you may purge all documents before full-reindex that you don't need
to run optimize unless you need the data to serve queries same time.
i think you are running into out of space because your 43 million may be
consuming 30% of total disk space and when you re-index the total dis
We are having an issue with running out of space when trying to do a
full re-index.
We are indexing with autocommit at 30 minutes.
We have it set to only optimize at the end of an indexing cycle.
On 12/12/2016 02:43 PM, Erick Erickson wrote:
First off, optimize is actually rarely necessary.
How much difference between below two parameters from your Solr stats
screen. For e.g. in our case we have very frequent updates which results
into max docs = num docs x2 over the period of time and in that case I have
seen optimization helps in query performance. Unless you have huge
difference,
First off, optimize is actually rarely necessary. I wouldn't bother
unless you have measurements to prove that it's desirable.
I would _certainly_ not call optimize every 10M docs. If you must call
it at all call it exactly once when indexing is complete. But see
above.
As far as the commit, I'd
Halp!
I need to reindex over 43 millions documents, when optimized the
collection is currently < 30% of disk space, we tried it over this
weekend and it ran out of space during the reindexing.
I'm thinking for the best solution for what we are trying to do is to
call commit/optimize every 10
