Hi,
Thanks for your reply.
I will work on your suggestion for using only one solr instance.
I tried to merge the 15 indexes again, and I found out that the new merged
index (without opitmization) size was about 351 GB , but when I optimize it
the size return back to 411 GB, Why?
I thought that optimization would decrease or at least be equal to the same
index size before optimization
Funtick wrote:
>
> Hi,
>
> Can you try to use single SOLR instance with heavy RAM (so that
> ramBufferSizeMB=8192 for instance) and mergeFactor=10? Single SOLR
> instance
> is fast enough (> 100 client threads of Tomcat; configurable) - I usually
> prefer single instance for single "writable" box with heavy RAM allocation
> and good I/O.
>
> Merging 15 indexes and 4-times larger size could happen, for instance,
> because of differences in SOLR Schema and Lucene; ensure that schema is
> the
> same (using Luke for instance). SOLR 1.4 has some new powerful features
> such
> as document->term cache stored somewhere (uninverted index) (Yonik), term
> vectors, stored=true, copyField, etc.
>
> Do not do commit per 100; do it once at the end...
>
>
>
> -Original Message-
> From: engy.ali [mailto:omeshm...@hotmail.com]
> Sent: August-25-09 3:31 PM
> To: solr-user@lucene.apache.org
> Subject: Solr index - Size and indexing speed
>
>
> Summary
> ===
>
> I had about 120,000 object of total size 71.2 GB, those objects are
> already
> indexed using Lucene. The index size is about 111 GB.
>
> I tried to use solr 1.4 nightly build to index the same collection. I
> divided collection on three servers, each server had 5 solr instances (not
> solr cores) up and running.
>
> After collection had been indexed, i merge the 15 indexes.
>
> Problems
> ==
>
> 1. The new merged index size is about 411 GB (i.e: 4 times larger than old
> index using lucene)
>
> I tried to index only on object using lucene and same object using solr to
> verify the size and the result was that the new index is about twice size
> of
> old index.
>
> DO you have any idea what might be the reason?
>
>
> 2. the indexing speed is slow, 100 object on single solr instance were
> indexed in 1 hour so i estimated that 1000 on single instance can be done
> in
> 10 hours, but that was not the case, the indexing time exceeds estimated
> time by about 12 hour.
>
> is that might be related to the growth of index?if not, so what might be
> the
> reason.
>
> Note: I do a commit/100 object and an optimize by the end of the whole
> operation. I also changed the mergeFactor from 10 to 15.
>
>
> 3. I google and found out that solr is using an inverted index, but I
> want
> to know what is the internal structure of solr index,for example if i have
> a
> word and its stems, how it will be store in the index
>
> Thanks,
> Engy
> --
> View this message in context:
> http://www.nabble.com/Solr-index---Size-and-indexing-speed-tp25140702p251407
> 02.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
--
View this message in context:
http://www.nabble.com/Solr-index---Size-and-indexing-speed-tp25140702p25201981.html
Sent from the Solr - User mailing list archive at Nabble.com.
