Hi Francois, If I got your numbers right, you are indexing on a single server and indexing rate is ~31 doc/s. I would first check if something is wrong with indexing logic. You check where the bottleneck is: do you read documents from DB fast enough, do you batch documents… Assuming you cannot have better rate than 30 doc/s and that bottleneck is Solr, in order to finish it in 6h, you need to parallelise indexing on Solr by splitting index to ~6 servers and have overall indexing rate of 180 doc/s.
Thanks, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 19 Jul 2018, at 09:59, servus01 <francois.rein...@sportcast.de> wrote: > > Would like to ask what your recommendations are for a new performant Solr > architecture. > > SQL DB 4M documents with up to 5000 metadata fields each document [2xXeon > 2.1Ghz, 32GB RAM] > Actual Solr: 1 Core version 4.6, 3.8M documents, schema has 300 metadata > fields to import, size 3.6GB [2xXeon 2.4Ghz, 32GB RAM] > (atm we need 35h to build the index and about 24h for a mass update which > affects the production) > > Building the index should be less than 6h. Sometimes we change some of the > Metadata fields which affects most of the documents and therefore a > massupdate / reindex is necessary. Reindex is ok also for about 6h (night) > but should not have an impact to user queries. Anyway, every faster indexing > is very welcome. We will have max. 20 - 30 CCUser. > > So i asked myself. How many nodes, Lshards, replicas ect. Could someone > please give me recommendation for a fast working architecture. > > really appreciate this, best > > Francois > > > > > > > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
