Indexing speed comes down to a lot of factors. The settings as talked about above, VM settings, the size of the documents, how many are sent at a time, how active you can keep the indexer (i.e. one thread sending documents lets the indexer relax whereas N threads keeps pressure on the indexer), how often you commit and of course the hardware you are running on. Disk I/O is a big factor along with having enough cores and memory to buffer and process the documents.
Comparing two sets of numbers is tough. We have indexes that range from indexing a few million an hour up through 18-20M per hour in a indexing cluster for distributed search. --j Jack Godwin wrote: > > 20+ hours? I index 3 million records in 3 hours. Is your auto commit > causing a snapshot? What do you have listed in the events. > > Jack > > On 5/14/09, Gargate, Siddharth <sgarg...@ptc.com> wrote: >> Hi all, >> I am also facing the same issue where autocommit blocks all >> other requests. I having around 1,00,000 documents with average size of >> 100K each. It took more than 20 hours to index. >> I have currently set autocommit maxtime to 7 seconds, mergeFactor to 25. >> Do I need more configuration changes? >> Also I see that memory usage goes to peak level of heap specified(6 GB >> in my case). Looks like Solr spends most of the time in GC. >> According to my understanding, fix for Solr-1155 would be that commit >> will run in background and new documents will be queued in the memory. >> But I am afraid of the memory consumption by this queue if commit takes >> much longer to complete. >> >> Thanks, >> Siddharth >> > > -- View this message in context: http://www.nabble.com/Autocommit-blocking-adds---AutoCommit-Speedup--tp23435224p23540643.html Sent from the Solr - User mailing list archive at Nabble.com.
