(disclaimer: I worked with Srikanth on this proposal) I've seen a number of situations where we fumble in the absence of this or things are built similar outside of Solr. The experts here may not fully appreciate this as we are experts; Solr newbies and sometimes agents, are not. I think there is growing utility for this with LLM-based interrogation of what's in Solr, and I could definitely see Solr's MCP using this.
Safety: This handler is only intended for use by a single person/LLM at a time to protect the "production" load assumed to be happening concurrently on the cluster. Of course this aspect could be toggled off if desired (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe even rate limit to no more than one or two requests at a time (Solr has code for this). There will be a slippery slope & tension on what /discover should do versus leave to existing/standard Solr methods. In my view, /discover shouldn't return entire documents -- just use /select if that is needed. I think it'd be helpful for the response structure to mirror /select where it fits. Certainly for facets & stats, which is mostly what /discover exposes. Much of the logic of /discover is just applying safe defaults on what facet/stats to return (possibly blocking "unsafe" requests), plus a rather new field coverage component. That's powered by "facet.query=field:*" ultimately, but still. In order to determine if it's "safe" to get the facet values for a field, an approximate cardinality check may be necessary first. On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 PARK) via dev <[email protected]> wrote: > Hi all, > > I'd like your thoughts on adding a new data discovery handler to Solr. The > design would revolve around two principles, simplicity and safety. It > should be easy for anyone to use, and it should have protections built in > so that exploring your data never puts the cluster at risk. In some cases, > that may mean returning approximate results instead of exact ones. > Questions, comments and concerns are all welcome. > > Problem > People often come to Solr with a question but no idea how the data is > structured. Which fields exist, which ones are populated, and what do they > contain? Developers, analysts and (increasingly) LLM agents all run into > this, and Solr makes these questions hard to answer. Additionally, dynamic > fields mean the schema doesn't show which fields actually exist, and > documents in the same collection can often have different fields. > Performing a very naive data discovery has the potential to execute very > expensive Solr queries (e.g. faceting on a high cardinality field, > returning a significant number of rows, etc…). > > What exists today > *Schema XML/Schema API shows the fields that are defined, but not the > dynamic fields that actually exist in the index. It also has no statistics, > so clients are unable to tell which fields have data. > *Luke (/admin/luke) has per-field stats, but you can't narrow them down > with filter queries. Newer versions of Solr can run it across shards, but > the detailed term stats are still per-shard. Finally, it's also an advanced > tool and hard for beginners to use. > *Facets can answer the question, but only if you already know how to write > a good Solr query. Every client has to build a facet for each field and > batch them, which isn't obvious to someone new to Solr. The concern of > expensive queries is directly related to utilizing facets. > > Proposal > A new DiscoveryHandler that takes filter queries (fq) and supports: > *Field coverage: how many matching documents have each field > *Field statistics/facets: distinct count, top values, and min/max for > numbers and dates. For low cardinality fields, the handler would return the > top facets. Where an exact answer would be too expensive, these may be > provided as approximate values. > *Pivot field: optionally compute stats and coverage by the values of a > field, such as the document type. Each value-group then gets its own > numbers. > Future Plans > Once the basic stats are available, there's a lot more the handler could > return: > *Field relationships: field coverage inter-relationships, such as fields > that only appear when another field is present or has a certain value. > *Visualization: the Admin UI could eventually be updated to display this > information. How exactly is out of scope for this proposal. > The aim is for discovery to eventually grow from "what fields are here" > into a real picture of how a user’s data is structured. > > Thank you, > Srikanth > > >
