Hi all,
We are trying to roughly double our sustained indexing throughput so we can
safely absorb approximately 2× tomorrow's incoming update volume without the
indexing pipeline falling behind.
I am looking primarily for advice on the write/indexing path (merge policy,
merge scheduler, commit strategy, indexing buffers, replication cost, and
client-side batching/concurrency).
Architecture
*
SolrCloud 9.6.1
*
Hosted on GCP Compute Engine
*
10 Solr nodes
*
Mixed memory sizes (~48 GB on 8 Nodes and ~94 GB RAM on 2 Nodes)
*
Solr heap: 12 GB/node
*
Default GC (no explicit tuning)
*
CPU:
*
16 logical cores on 8 Nodes
*
24 logical cores on 2 Nodes
*
3-node ZooKeeper
Collections:
*
Products
*
~180M documents
*
~62 shards
*
Total Index Size: 245 GB
*
Leads
*
~70M documents
*
~62 shards
*
Total Index Size: 200 GB
Routing is custom (implicit routing, not compositeId).
Replication factor is mostly 1, with 3 shards currently at RF=2.
Indexing Jobs Writing Flow
Products collection — two-stage pipeline (DB → CSV → Solr)
1.
DB → CSV: Two batch jobs each read up to 300k changed rows from a changelog
table per run (600k rows/cycle combined) and write pipe-delimited CSV files
containing 5,000 rows each.
2.
CSV → Solr: A separate batch job drains up to 120 CSV files per run and indexes
them via SolrJ, sending update batches of 4,000 documents per request.
Leads collection — single-stage pipeline (DB → Solr)
1.
One batch job reads up to 600k changed rows directly from the database (no CSV
intermediate) and indexes them via SolrJ, sending update batches of 3,000
documents per request on a near-continuous (every-minute) schedule.
Indexing Configuration
The following indexing configuration is currently in use for both collections:
Setting
Value
ramBufferSizeMB
200
Merge Policy
TieredMergePolicyFactory
maxMergeAtOnce
5
segmentsPerTier
5
floorSegmentMB
10
deletesPctAllowed
20
noCFSRatio
0
Merge Scheduler
Default ConcurrentMergeScheduler
useCompoundFile
FALSE
Hard Commit
Every 30 minutes (openSearcher=false)
Soft Commit
Every 15 minutes
Update Log
Enabled (numVersionBuckets=65536)
Directory Factory
StandardDirectoryFactory (MMapDirectory)
Update Request Processor
Default update request processor chain
Indexing Client Configuration
Setting
Value
Client
SolrJ 8.10.0
Server Version
Solr 9.6.1
Workload
Most indexing is performed by Spring Batch + SolrJ using large batched update
requests rather than individual document writes.
Daily incoming update volume
*
Products collection: ~20 million documents/day
*
Leads collection: ~45 million documents/day
Current indexing capacity
*
Products collection: ~25 million documents/day
*
Leads collection: ~25 million documents/day
We also have live incoming search requests on the two collections which have a
peak request count of 6 Lakhs search request per hour.
The Products indexing pipeline currently has sufficient headroom, while the
Leads pipeline is the bottleneck. We need to increase sustained indexing
throughput to comfortably handle approximately 45 million documents/day for the
Leads collection (with additional operational headroom) without the indexing
jobs falling behind.
Questions
If your goal were approximately 2x to 3x sustained indexing throughput with
this workload, what would you prioritize first?
1.
Increasing ramBufferSizeMB or adjusting TieredMergePolicy?
2.
Explicitly tuning ConcurrentMergeScheduler instead of relying on Lucene
defaults?
3.
Changing the 30-minute hard commit interval (balancing tlog growth versus
commit overhead)?
4.
Using more TLOG replicas (or different replica types) to reduce write-path cost?
5.
Increasing client-side concurrency versus increasing batch size?
6.
Any known indexing bottlenecks with StandardDirectoryFactory/MMapDirectory on
GCP persistent disks at this scale?
If there are other areas you'd investigate first before touching merge policy
or commit settings, I'd appreciate any guidance.
Happy to provide additional configuration or metrics if useful.
Thanks!
Govind Kumar
