This same problem existed when I was working on the sql handler. Basically how do you handle `*` or other fields that are in the "schema" but are not directly referenced (dynamic fields).
https://issues.apache.org/jira/browse/SOLR-8847 and some of the linked jiras highlighted the problems around not knowing what fields were actually indexed. As a side note from implementing a similar lookup in the past, certain field types have these counts easily available and others don't. So the implementation of this can be problematic from a performance standpoint. A field count query is not necessarily fast if it has to iterate and count all the matching documents. It can be quick if there are stats available. Luke can do a lot of this since it ignores deletes. Docvalues can help a lot here too. Its just not always available on say "stored" only fields without indexed or docvalues. I don't think this is a deal breaker, but just a limitation that needs to be called out somewhere. Kevin Risden On Mon, Nov 16, 2026 at 3:50 PM David Smiley via dev <[email protected]> wrote: > Amrit summarized this really well! > > So I'm not seeing any resistance. We could create a JIRA. > > I'm inclined to suggest a first stage that just lists the (populated) > fields with type info and doc-counts (applying filters). Maybe > "/discovery/fields"? I don't think this would be combined with subsequent > detailed statistics per field on the same request; thus, the requests are > fundamentally different/bifurcated. I could see it taking a field regex > (default none -- list all), on top of any "fq" to filter (if any). > Safety: This might still be rather expensive if there are many fields. At > least value-existence is exposed as a DocIdSetIterator (DISI) at the Lucene > layer for almost all index structures (albeit nothing for > stored/term-vectors). A field that is stored-only (or TV-only) will have > no stats for this handler to return. Adding field paging sounds more > awkward than it's worth; we want this to be easy to use. It can be added > later, too. > > A client could then later request statistics per-field by listing them > explicitly, not unlike "facet.field", and with the "f.<fieldname>..." > override potential if needed to tune what's needed. Maybe I'm thinking of > this in a very classic Solr API v1 way... I know JSON Facets and to some > extent V2 likes JSON input. > > > On Sun, Nov 15, 2026 at 5:26 PM Amrit Sarkar <[email protected]> > wrote: > > > Hi Srikanth, David, > > > > +1, I like this. One concrete gap it would close: solr-mcp's get-schema > > < > > > https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248 > > > > > only returns defined fields, so dynamic fields that actually hold data > are > > invisible to an agent today. > > > > A few use cases I'd add: > > > > - Onboarding a collection someone else built, which is routine for > > platform teams like the one at my company that run search for many > > product teams. The same goes for multi-tenant collections where doc > > types > > carry different fields > > - Data quality after ingestion: a field's fill rate dropping from 98% > to > > 40% usually means a broken pipeline. I deal with this almost weekly > with > > other data stores, and a cheap way to see it would help a lot > > - Auditing dynamic field sprawl before a schema cleanup or migration > > > > On David's point about checking cardinality first, Druid is a useful > > reference. I've used it for metrics OLAP, and its segmentMetadata > > <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/> > > query > > reports per-column cardinality and min/max from metadata it already > stores, > > without scanning data. If "/discover" can answer "is this field safe to > > facet" from stats the index already keeps, the safety gate stays cheap > and > > only borderline fields need the heavier approximate check. > > > > Will closely follow this development. > > > > Amrit > > > > On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected]> > > wrote: > > > > > > > > > > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote: > > > > Hi > > > > > > > > Sounds like something that could be useful, although it would be a > bit > > > uncommon > > > > to have a bunch of data in Solr already without already knowing your > > > fields? > > > > > > > > The proposal reminds me of OpenRefine https://openrefine.org which > is > > a > > > great > > > > tool to use *before* you start indexing a dataset to Solr. It will > eat > > a > > > CSV, XMl or > > > > whatever, and let you discover what's in the data, cardinality, data > > > quality etc. > > > > > > > > A different use case, but just wanted to x-ref it. > > > > > > > > Jan > > > > > > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev < > > > [email protected]>: > > > > > > > > > > (disclaimer: I worked with Srikanth on this proposal) > > > > > > > > > > I've seen a number of situations where we fumble in the absence of > > > this or > > > > > things are built similar outside of Solr. The experts here may not > > > fully > > > > > appreciate this as we are experts; Solr newbies and sometimes > agents, > > > are > > > > > not. I think there is growing utility for this with LLM-based > > > > > interrogation of what's in Solr, and I could definitely see Solr's > > MCP > > > > > using this. > > > > > > > > > > Safety: This handler is only intended for use by a single > person/LLM > > > at a > > > > > time to protect the "production" load assumed to be happening > > > concurrently > > > > > on the cluster. Of course this aspect could be toggled off if > > desired > > > > > (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe > even > > > rate > > > > > limit to no more than one or two requests at a time (Solr has code > > for > > > > > this). > > > > > > > > > > There will be a slippery slope & tension on what /discover should > do > > > versus > > > > > leave to existing/standard Solr methods. In my view, /discover > > > shouldn't > > > > > return entire documents -- just use /select if that is needed. > > > > > > > > > > I think it'd be helpful for the response structure to mirror > /select > > > where > > > > > it fits. Certainly for facets & stats, which is mostly what > > /discover > > > > > exposes. Much of the logic of /discover is just applying safe > > > defaults on > > > > > what facet/stats to return (possibly blocking "unsafe" requests), > > plus > > > a > > > > > rather new field coverage component. That's powered by > > > > > "facet.query=field:*" ultimately, but still. In order to determine > > if > > > it's > > > > > "safe" to get the facet values for a field, an approximate > > cardinality > > > > > check may be necessary first. > > > > > > > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 > PARK) > > > via > > > > > dev <[email protected]> wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> I'd like your thoughts on adding a new data discovery handler to > > > Solr. The > > > > >> design would revolve around two principles, simplicity and safety. > > It > > > > >> should be easy for anyone to use, and it should have protections > > > built in > > > > >> so that exploring your data never puts the cluster at risk. In > some > > > cases, > > > > >> that may mean returning approximate results instead of exact ones. > > > > >> Questions, comments and concerns are all welcome. > > > > >> > > > > >> Problem > > > > >> People often come to Solr with a question but no idea how the data > > is > > > > >> structured. Which fields exist, which ones are populated, and what > > do > > > they > > > > >> contain? Developers, analysts and (increasingly) LLM agents all > run > > > into > > > > >> this, and Solr makes these questions hard to answer. Additionally, > > > dynamic > > > > >> fields mean the schema doesn't show which fields actually exist, > and > > > > >> documents in the same collection can often have different fields. > > > > >> Performing a very naive data discovery has the potential to > execute > > > very > > > > >> expensive Solr queries (e.g. faceting on a high cardinality field, > > > > >> returning a significant number of rows, etc…). > > > > >> > > > > >> What exists today > > > > >> *Schema XML/Schema API shows the fields that are defined, but not > > the > > > > >> dynamic fields that actually exist in the index. It also has no > > > statistics, > > > > >> so clients are unable to tell which fields have data. > > > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow them > > > down > > > > >> with filter queries. Newer versions of Solr can run it across > > shards, > > > but > > > > >> the detailed term stats are still per-shard. Finally, it's also an > > > advanced > > > > >> tool and hard for beginners to use. > > > > >> *Facets can answer the question, but only if you already know how > to > > > write > > > > >> a good Solr query. Every client has to build a facet for each > field > > > and > > > > >> batch them, which isn't obvious to someone new to Solr. The > concern > > of > > > > >> expensive queries is directly related to utilizing facets. > > > > >> > > > > >> Proposal > > > > >> A new DiscoveryHandler that takes filter queries (fq) and > supports: > > > > >> *Field coverage: how many matching documents have each field > > > > >> *Field statistics/facets: distinct count, top values, and min/max > > for > > > > >> numbers and dates. For low cardinality fields, the handler would > > > return the > > > > >> top facets. Where an exact answer would be too expensive, these > may > > be > > > > >> provided as approximate values. > > > > >> *Pivot field: optionally compute stats and coverage by the values > > of a > > > > >> field, such as the document type. Each value-group then gets its > own > > > > >> numbers. > > > > >> Future Plans > > > > >> Once the basic stats are available, there's a lot more the handler > > > could > > > > >> return: > > > > >> *Field relationships: field coverage inter-relationships, such as > > > fields > > > > >> that only appear when another field is present or has a certain > > value. > > > > >> *Visualization: the Admin UI could eventually be updated to > display > > > this > > > > >> information. How exactly is out of scope for this proposal. > > > > >> The aim is for discovery to eventually grow from "what fields are > > > here" > > > > >> into a real picture of how a user’s data is structured. > > > > >> > > > > >> Thank you, > > > > >> Srikanth > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > This lines up with the idea that in the future, we actually expose (in > a > > > safe-ish manner) all the capabilities of our search environment to the > > > agents so they build a "perfect" one off query. Isntead of a narrow > "we > > > accept just a query and maybe some filters" and then expand everything > > out, > > > we flip that scirpt and say "here are all the tools, build your perfect > > > query" and we then provide all the validation/guard rails/workload > > > management to support that approach! > > > > > > I've looked at https://llmstxt.org/ and the idea of having a > > > http://localhost:8983/llms.txt that told you about the cluster, what > > > collections are available based on who you are and your permissions. > And > > > then imagine we had https://localhost:8983/solr/techproducts/llms.txt > > > that was specific to that collection? > > > > > > I think there is some overlap here with what solr-mcp might provide in > > > being a "safer" way of exposing primitives to Agents.... > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
