I tend to separate the search quality requirements from the search performance requirements. For example, often in environments where data changes frequently, and I’m doing search quality testing, then I’ll use a “date frozen” data set. If recency is a part of my algorithm then I’ll set the NOW parameter to a specific date, so that when I run my evaluations they aren’t changing from day to day (https://solr.apache.org/guide/solr/latest/indexing-guide/date-formatting-math.html#now). I’ll update that frozen data set on a periodic basis, say once a quarter. This makes it a lot easier to use human eval for search quality.
For the performance use case, well you can use a tool like https://goreplay.org/ to pick up all the traffic from Prod and replay it on your test environment. Log the search and indexing requests from production, and then replay them in your test to simulate load while you run your performance tools. > On Nov 6, 2023, at 12:30 PM, Saksham Gupta > <saksham.gu...@indiamart.com.INVALID> wrote: > > Hi Eric, > Thanks for the valuable inputs. > > A big one is, are you testing query/indexing performance or are you testing > search quality? > Mostly we are testing search quality but once in a while query/indexing > performance. > The major problem here is how to keep the data on test shards updated as we > have a high number of updates on the production server. > > Same arguments for separate pipeline. If dev is where you build new > pipelines and stage is where you stage them, then don’t use the same. > Again, depends on what you are trying to do. > I was trying to enquire about how to perform indexing on my dev/ stage solr > setup, can you please elaborate your point. > > On Fri, Nov 3, 2023 at 7:58 PM Eric Pugh <ep...@opensourceconnections.com > <mailto:ep...@opensourceconnections.com>> > wrote: > >> I think there are a lot of ways of setting this. A big one is, are you >> testing query/indexing performance or are you testing search quality? >> >> If it’s search quality, you need a representative data set…. The easiest >> way to do that is to have a copy of production data in your test >> environment. But you don’t need the same scale to support testing search >> quality…. If you are testing query performance, well then, you either >> need a model that lets you confirm your smaller test environment will match >> the larger prod environment…. Or you just have test == prod ;-). >> >> >>> On Nov 3, 2023, at 12:26 AM, Saksham Gupta >>> <saksham.gu...@indiamart.com.INVALID> >> wrote: >>> >>> Hi Solr Developers, >>> >>> Reaching out to inquire about the best practices to set up staging and >> dev >>> environments for solr cloud. >>> >>> We are using solr cloud with a cluster of 8 nodes ~25 gb of data(15-16 >>> million docs) present on each shard. The resources used are optimized to >>> server heavy indexing/searching. On a daily basis, approximately 4 >> million >>> search requests are served and more than 50 million documents are updated >>> daily. >>> >>> While setting up the solr for testing, got perplexed about multiple >> points >>> like: >>> >>> 1. How to maintain dev and stage environments? Should we maintain >> separate >>> infra for both? >> Separate infra makes it easier to test upgrades of Solr.. you can have >> 9.4 in dev and 9.3 in stage and prod ;-). Dev can often be smaller…. >> Unless you are testing search quality, and then you want dev to mimic prod >> from a volume and variety of data ;-). >> >>> >>> 2. Should we set up indexing separately for dev and stage as well or is >>> there a much simpler way to replicate data from production replicas onto >>> these replicas? >> Same arguments for separate pipeline. If dev is where you build new >> pipelines and stage is where you stage them, then don’t use the same. >> Again, depends on what you are trying to do. >>> >>> 3. How much resources to allocate? Should it be the same as on >> production, >>> or is there a better way here to save the cost on resources? If yes, what >>> are the minimum resources that should be allocated? >> If you can scale up and down easily, then it means you can spin up stage >> when you are doing perf testing and then spin down when you are done. >>> >>> 4. Each production node contains a single shard, but on the testing >> infra, >>> should we keep more than 1 shard on each node? >>> >> >> Use what ever load testing tools you use for the rest of your stack. >> But.. do make sure you are running a variety of production queries.. >> don’t just issue one query over and over! BTW, I like GoReplay for >> replaying real traffic from prod against my test environment. >>> 5. How to run performance/ load testing to check the performance related >>> impact? How to measure the impact as accurately as possible, so that we >> get >>> the expected impact on production as well? >>> >>> Please help me get through these questions. Thanks in advance!! >> >> _______________________ >> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | >> http://www.opensourceconnections.com < >> http://www.opensourceconnections.com/> | My Free/Busy < >> http://tinyurl.com/eric-cal> >> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed < >> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> >> >> This e-mail and all contents, including attachments, is considered to be >> Company Confidential unless explicitly stated otherwise, regardless of >> whether attachments are marked as such. _______________________ Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com <http://www.opensourceconnections.com/> | My Free/Busy <http://tinyurl.com/eric-cal> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
