Hi all, I'd like your thoughts on adding a new data discovery handler to Solr. The design would revolve around two principles, simplicity and safety. It should be easy for anyone to use, and it should have protections built in so that exploring your data never puts the cluster at risk. In some cases, that may mean returning approximate results instead of exact ones. Questions, comments and concerns are all welcome.
Problem People often come to Solr with a question but no idea how the data is structured. Which fields exist, which ones are populated, and what do they contain? Developers, analysts and (increasingly) LLM agents all run into this, and Solr makes these questions hard to answer. Additionally, dynamic fields mean the schema doesn't show which fields actually exist, and documents in the same collection can often have different fields. Performing a very naive data discovery has the potential to execute very expensive Solr queries (e.g. faceting on a high cardinality field, returning a significant number of rows, etc…). What exists today *Schema XML/Schema API shows the fields that are defined, but not the dynamic fields that actually exist in the index. It also has no statistics, so clients are unable to tell which fields have data. *Luke (/admin/luke) has per-field stats, but you can't narrow them down with filter queries. Newer versions of Solr can run it across shards, but the detailed term stats are still per-shard. Finally, it's also an advanced tool and hard for beginners to use. *Facets can answer the question, but only if you already know how to write a good Solr query. Every client has to build a facet for each field and batch them, which isn't obvious to someone new to Solr. The concern of expensive queries is directly related to utilizing facets. Proposal A new DiscoveryHandler that takes filter queries (fq) and supports: *Field coverage: how many matching documents have each field *Field statistics/facets: distinct count, top values, and min/max for numbers and dates. For low cardinality fields, the handler would return the top facets. Where an exact answer would be too expensive, these may be provided as approximate values. *Pivot field: optionally compute stats and coverage by the values of a field, such as the document type. Each value-group then gets its own numbers. Future Plans Once the basic stats are available, there's a lot more the handler could return: *Field relationships: field coverage inter-relationships, such as fields that only appear when another field is present or has a certain value. *Visualization: the Admin UI could eventually be updated to display this information. How exactly is out of scope for this proposal. The aim is for discovery to eventually grow from "what fields are here" into a real picture of how a user’s data is structured. Thank you, Srikanth
