Hi, Cool with helpful tools like this. Have you thought of writing blog posts about them and publishing on solr's blog?
Idea - could there perhaps be some synergy between our new solr-orbit tool and your solr-datagen tool? I mean, we currently have two workloads available at https://github.com/apache/solr-orbit-workloads - could perhaps the datagen be a new workload type, but instead of pulling a static dataset, it generates one? I suppose it can generate with a seed so you get same output each time? And your tool is python, so perhaps it can be adapted as a workload, i.e. the data generation logic only, not the ingest logic which is handled by orbit itself? Jan > 22. juni 2026 kl. 17:51 skrev Rahul Goswami <[email protected]>: > > Hello, > I commonly find myself needing to cook up data for Solr or > (not-so-commonly) needing to validate data integrity (detect > corruption/data deviation scenarios). To that effect, I recently vibe coded > a couple of repos over several iterations of testing and feature-additions. > > solr-datagen <https://github.com/rahulgoswami/solr-datagen> : Generate and > index realistic documents into Apache Solr at scale, or reindex existing > documents in-place. > Given a Solr URL (with collection/core name), solr-datagen can either > generate synthetic documents with realistic data across all field types and > index them in parallel batches (index mode), or read existing documents > from the collection and write them back in-place (reindex mode) > > index_validator <https://github.com/rahulgoswami/index_validator> : Detect > field-level data loss and value corruption by comparing two Solr indexes > (or a pre-operation snapshot against a live index) document-by-document. > > I have found them useful for scale testing and data validation and so > thought of sharing with the community if it helps anyone. These originated > as scratch-work tools and gradually evolved to handle scale and support > more features as needed. So I'm sure there are a lot of ways these can be > improved (eg: solr-datagen doesn't necessarily have to be specific to Solr > and can be decoupled to work with other engines). Sharing here in their > present form nevertheless if it helps others. > > Best, > Rahul --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
