Hi Daniel, Great background information about your setup! I've got just a few more questions:
- Can you describe the process that queries the DB and sends records to Solr? - Is it a SolrJ-based application? - If it is, which client package are you using? - How many documents do you send at once? - Are you sending your indexing or query traffic through a load balancer? If you're sending documents to each replica as fast as they can take them, you might be seeing a bottleneck at the shard leaders. The SolrJ CloudSolrClient finds out from Zookeeper which nodes are the shard leaders and sends docs directly to them. -Scott On Tue, Aug 22, 2017 at 2:16 PM, Daniel Ortega <danielortegauf...@gmail.com> wrote: > *Main Problems* > > > We are involved in a migration from Solr Master/Slave infrastructure to > SolrCloud infrastructure. > > > > The main problems that we have now are: > > > > - Excessive resources consumption: Currently we have 5 instances with 80 > processors/768 GB RAM each instance using SSD Hard Disk Drives that > doesn't > support the load that we have in the other architecture. In our > Master-Slave architecture we have only 7 Virtual Machines with lower > specs > (4 processors and 16 GB each instance using SSD Hard Disk Drives too). > So, > at the moment our SolrCloud infrastructure is wasting several dozen > times > more resources than our Solr Master/Slave infrastructure. > - Despite spending more resources we have worst query times (compared to > Solr in master/slave architecture) > > > *Search infrastructure (SolrCloud infrastructure)* > > > > As we cannot use DIH Handler (which is what we use in Solr Master/Slave), > we > have developed an application which reads every transaction from Oracle, > builds a document collection searching in the database and sends the result > to the */update* handler every 200 milliseconds using SolrJ client. This > application tries to delete the possible duplicates in each update window, > but we are using solr’s de-duplication techniques > <https://emea01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay% > 2Fsolr%2FDe-Duplication&data=02%7C01%7Cdortega%40idealista.com% > 7Cb169ea024abc4954927208d4bc6868eb%7Cd78b7929c2a34897ae9a7d8f8dc1 > a1cf%7C0%7C0%7C636340604697721266&sdata=WEhzoHC1Bf77K706% > 2Fj2wIWOw5gzfOgsP1IPQESvMsqQ%3D&reserved=0> > too. > > > > We are indexing ~100 documents per second (with peaks of ~1000 documents > per second). > > > > Every search query is centralized in other application which exposes a DSL > behind a REST API and uses SolrJ client too to perform queries. We have > peaks of 2000 QPS. > > *Cluster structure **(SolrCloud infrastructure)* > > > > At the moment, the cluster has 30 SolrCloud instances with the same specs > (Same physical hosts, same JVM Settings, etc.). > > > > *Main collection* > > > > In our use case we are using this collection as a NoSQL database basically. > Our document is composed of about 300 fields that represents an advert, and > is a denormalization of its relational representation in Oracle. > > > We are using all our nodes to store the collection in 3 shards. So, each > shard has 10 replicas. > > > At the moment, we are only indexing a subset of the adverts stored in > Oracle, but our goal is to store all the ads that we have in the DB (a few > tens of millions of documents). We have NRT requirements, so we need to > index every document as soon as posible once it’s changed in Oracle. > > > > We have defined the properties of each field (if it’s stored/indexed or > not, if should be defined as DocValue, etc…) considering the use of that > field. > > > > *Index size **(SolrCloud infrastructure)* > > > > The index size is currently above 6 GB, storing 1.300.000 documents in each > shard. So, we are storing 3.900.000 documents and the total index size is > 18 GB. > > > > *Indexation **(SolrCloud infrastructure)* > > > > The commits *aren’t* triggered by the application described before. The > hardcommit/softcommit interval are configured in Solr: > > > > - *HardCommit:* every 15 minutes (with opensearcher = false) > - *SoftCommit:* every 5 seconds > > > > *Apache Solr Version* > > > > We are currently using the last version of Solr (6.6.0) under an Oracle VM > (Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Oracle (64 bits)) in > both deployments. > > > The question is... What is wrong here?!?!?! > -- Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC | 434.409.2780 http://www.opensourceconnections.com
