Solr faceting supports "field.exists" boolean optimization when you don't want the count (a nominal '1' is returned instead). That sounds perfect for /discovery/fields since we want that to be cheap. Make counts opt-in. That said, facet.exists only applies/works on a facet.field (terms faceting) and not facet.query. And even then, just sometimes -- when facet.method is enum. :-( I confirmed this; sadly our ref guide doesn't communicate this limitation. Any way, we can tangentially work on fixing/optimizing that further. I think the underlying plumbing to return early is present. I filed https://issues.apache.org/jira/browse/SOLR-18666 for that.
On Mon, Nov 16, 2026 at 9:47 PM Aditya Parikh <[email protected]> wrote: > Hi Srikanth, David, Amrit, Kevin, all, > > +1 from the solr-mcp side. An agent pointed at a Solr instance starts > every conversation knowing nothing about the collection, so it has to > discover what's there before it can search it. > > What solr-mcp has today (including a PR in flight): > > 1.Tool: *get-schema*, Resource: *solr://{collection}/schema* > Uses: *GET /solr/{collection}/schema,* which returns the entire schema > ( > https://solr.apache.org/guide/solr/latest/indexing-guide/schema-api.html#retrieve-the-entire-schema > ) > Used for: seeing which field types and analyzer chains exist before > adding fields or field types. > Shortcomings: it describes the configset, not the data: no counts, > declared-but-empty fields are listed, and a dynamic field in use shows only > as its pattern. > > 2. Tool: list-fields, Resource: solr://{collection}/fields ( > https://github.com/apache/solr-mcp/pull/288) > Uses: /admin/luke, which returns the fields that occur in the index, > each with its type, flags and dynamicBase, and a doc count. > Used for: what an agent should read before choosing fields for q, fq, > faceting and sorting. > Shortcomings: the doc count comes from terms, so numeric/date point > fields and docValues-only fields report 0, meaning not counted rather than > empty. It also includes deleted > documents and can't take an fq. > > How solr-mcp would use /discover: > - *list-fields* and *solr://{collection}/fields* would call > */discovery/fields* on Solr versions that have it, keeping Luke as the > fallback. > - A new tool, *get-field-stats*: distinct count, top values and min/max > for the fields an agent picks under the same fq. > > On Kevin's point about calling out the limitation: for an agent, the place > to call it out is the response itself. If each count says how it was > obtained (exact, an index statistic, approximate, or not available), an > agent won't read "not counted" as "empty". > > Aditya > > On Mon, Nov 16, 2026 at 4:40 PM Kevin Risden via dev <[email protected]> > wrote: > >> This same problem existed when I was working on the sql handler. Basically >> how do you handle `*` or other fields that are in the "schema" but are not >> directly referenced (dynamic fields). >> >> https://issues.apache.org/jira/browse/SOLR-8847 and some of the linked >> jiras highlighted the problems around not knowing what fields were >> actually >> indexed. >> >> As a side note from implementing a similar lookup in the past, certain >> field types have these counts easily available and others don't. So the >> implementation of this can be problematic from a performance standpoint. A >> field count query is not necessarily fast if it has to iterate and count >> all the matching documents. It can be quick if there are stats available. >> Luke can do a lot of this since it ignores deletes. Docvalues can help a >> lot here too. Its just not always available on say "stored" only fields >> without indexed or docvalues. I don't think this is a deal breaker, but >> just a limitation that needs to be called out somewhere. >> >> Kevin Risden >> >> >> >> On Mon, Nov 16, 2026 at 3:50 PM David Smiley via dev <[email protected] >> > >> wrote: >> >> > Amrit summarized this really well! >> > >> > So I'm not seeing any resistance. We could create a JIRA. >> > >> > I'm inclined to suggest a first stage that just lists the (populated) >> > fields with type info and doc-counts (applying filters). Maybe >> > "/discovery/fields"? I don't think this would be combined with >> subsequent >> > detailed statistics per field on the same request; thus, the requests >> are >> > fundamentally different/bifurcated. I could see it taking a field regex >> > (default none -- list all), on top of any "fq" to filter (if any). >> > Safety: This might still be rather expensive if there are many >> fields. At >> > least value-existence is exposed as a DocIdSetIterator (DISI) at the >> Lucene >> > layer for almost all index structures (albeit nothing for >> > stored/term-vectors). A field that is stored-only (or TV-only) will >> have >> > no stats for this handler to return. Adding field paging sounds more >> > awkward than it's worth; we want this to be easy to use. It can be >> added >> > later, too. >> > >> > A client could then later request statistics per-field by listing them >> > explicitly, not unlike "facet.field", and with the "f.<fieldname>..." >> > override potential if needed to tune what's needed. Maybe I'm thinking >> of >> > this in a very classic Solr API v1 way... I know JSON Facets and to some >> > extent V2 likes JSON input. >> > >> > >> > On Sun, Nov 15, 2026 at 5:26 PM Amrit Sarkar <[email protected]> >> > wrote: >> > >> > > Hi Srikanth, David, >> > > >> > > +1, I like this. One concrete gap it would close: solr-mcp's >> get-schema >> > > < >> > > >> > >> https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248 >> > > > >> > > only returns defined fields, so dynamic fields that actually hold data >> > are >> > > invisible to an agent today. >> > > >> > > A few use cases I'd add: >> > > >> > > - Onboarding a collection someone else built, which is routine for >> > > platform teams like the one at my company that run search for many >> > > product teams. The same goes for multi-tenant collections where doc >> > > types >> > > carry different fields >> > > - Data quality after ingestion: a field's fill rate dropping from >> 98% >> > to >> > > 40% usually means a broken pipeline. I deal with this almost weekly >> > with >> > > other data stores, and a cheap way to see it would help a lot >> > > - Auditing dynamic field sprawl before a schema cleanup or >> migration >> > > >> > > On David's point about checking cardinality first, Druid is a useful >> > > reference. I've used it for metrics OLAP, and its segmentMetadata >> > > <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/> >> > > query >> > > reports per-column cardinality and min/max from metadata it already >> > stores, >> > > without scanning data. If "/discover" can answer "is this field safe >> to >> > > facet" from stats the index already keeps, the safety gate stays cheap >> > and >> > > only borderline fields need the heavier approximate check. >> > > >> > > Will closely follow this development. >> > > >> > > Amrit >> > > >> > > On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected] >> > >> > > wrote: >> > > >> > > > >> > > > >> > > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote: >> > > > > Hi >> > > > > >> > > > > Sounds like something that could be useful, although it would be a >> > bit >> > > > uncommon >> > > > > to have a bunch of data in Solr already without already knowing >> your >> > > > fields? >> > > > > >> > > > > The proposal reminds me of OpenRefine https://openrefine.org >> which >> > is >> > > a >> > > > great >> > > > > tool to use *before* you start indexing a dataset to Solr. It will >> > eat >> > > a >> > > > CSV, XMl or >> > > > > whatever, and let you discover what's in the data, cardinality, >> data >> > > > quality etc. >> > > > > >> > > > > A different use case, but just wanted to x-ref it. >> > > > > >> > > > > Jan >> > > > > >> > > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev < >> > > > [email protected]>: >> > > > > > >> > > > > > (disclaimer: I worked with Srikanth on this proposal) >> > > > > > >> > > > > > I've seen a number of situations where we fumble in the absence >> of >> > > > this or >> > > > > > things are built similar outside of Solr. The experts here may >> not >> > > > fully >> > > > > > appreciate this as we are experts; Solr newbies and sometimes >> > agents, >> > > > are >> > > > > > not. I think there is growing utility for this with LLM-based >> > > > > > interrogation of what's in Solr, and I could definitely see >> Solr's >> > > MCP >> > > > > > using this. >> > > > > > >> > > > > > Safety: This handler is only intended for use by a single >> > person/LLM >> > > > at a >> > > > > > time to protect the "production" load assumed to be happening >> > > > concurrently >> > > > > > on the cluster. Of course this aspect could be toggled off if >> > > desired >> > > > > > (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe >> > even >> > > > rate >> > > > > > limit to no more than one or two requests at a time (Solr has >> code >> > > for >> > > > > > this). >> > > > > > >> > > > > > There will be a slippery slope & tension on what /discover >> should >> > do >> > > > versus >> > > > > > leave to existing/standard Solr methods. In my view, /discover >> > > > shouldn't >> > > > > > return entire documents -- just use /select if that is needed. >> > > > > > >> > > > > > I think it'd be helpful for the response structure to mirror >> > /select >> > > > where >> > > > > > it fits. Certainly for facets & stats, which is mostly what >> > > /discover >> > > > > > exposes. Much of the logic of /discover is just applying safe >> > > > defaults on >> > > > > > what facet/stats to return (possibly blocking "unsafe" >> requests), >> > > plus >> > > > a >> > > > > > rather new field coverage component. That's powered by >> > > > > > "facet.query=field:*" ultimately, but still. In order to >> determine >> > > if >> > > > it's >> > > > > > "safe" to get the facet values for a field, an approximate >> > > cardinality >> > > > > > check may be necessary first. >> > > > > > >> > > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 >> > PARK) >> > > > via >> > > > > > dev <[email protected]> wrote: >> > > > > > >> > > > > >> Hi all, >> > > > > >> >> > > > > >> I'd like your thoughts on adding a new data discovery handler >> to >> > > > Solr. The >> > > > > >> design would revolve around two principles, simplicity and >> safety. >> > > It >> > > > > >> should be easy for anyone to use, and it should have >> protections >> > > > built in >> > > > > >> so that exploring your data never puts the cluster at risk. In >> > some >> > > > cases, >> > > > > >> that may mean returning approximate results instead of exact >> ones. >> > > > > >> Questions, comments and concerns are all welcome. >> > > > > >> >> > > > > >> Problem >> > > > > >> People often come to Solr with a question but no idea how the >> data >> > > is >> > > > > >> structured. Which fields exist, which ones are populated, and >> what >> > > do >> > > > they >> > > > > >> contain? Developers, analysts and (increasingly) LLM agents all >> > run >> > > > into >> > > > > >> this, and Solr makes these questions hard to answer. >> Additionally, >> > > > dynamic >> > > > > >> fields mean the schema doesn't show which fields actually >> exist, >> > and >> > > > > >> documents in the same collection can often have different >> fields. >> > > > > >> Performing a very naive data discovery has the potential to >> > execute >> > > > very >> > > > > >> expensive Solr queries (e.g. faceting on a high cardinality >> field, >> > > > > >> returning a significant number of rows, etc…). >> > > > > >> >> > > > > >> What exists today >> > > > > >> *Schema XML/Schema API shows the fields that are defined, but >> not >> > > the >> > > > > >> dynamic fields that actually exist in the index. It also has no >> > > > statistics, >> > > > > >> so clients are unable to tell which fields have data. >> > > > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow >> them >> > > > down >> > > > > >> with filter queries. Newer versions of Solr can run it across >> > > shards, >> > > > but >> > > > > >> the detailed term stats are still per-shard. Finally, it's >> also an >> > > > advanced >> > > > > >> tool and hard for beginners to use. >> > > > > >> *Facets can answer the question, but only if you already know >> how >> > to >> > > > write >> > > > > >> a good Solr query. Every client has to build a facet for each >> > field >> > > > and >> > > > > >> batch them, which isn't obvious to someone new to Solr. The >> > concern >> > > of >> > > > > >> expensive queries is directly related to utilizing facets. >> > > > > >> >> > > > > >> Proposal >> > > > > >> A new DiscoveryHandler that takes filter queries (fq) and >> > supports: >> > > > > >> *Field coverage: how many matching documents have each field >> > > > > >> *Field statistics/facets: distinct count, top values, and >> min/max >> > > for >> > > > > >> numbers and dates. For low cardinality fields, the handler >> would >> > > > return the >> > > > > >> top facets. Where an exact answer would be too expensive, these >> > may >> > > be >> > > > > >> provided as approximate values. >> > > > > >> *Pivot field: optionally compute stats and coverage by the >> values >> > > of a >> > > > > >> field, such as the document type. Each value-group then gets >> its >> > own >> > > > > >> numbers. >> > > > > >> Future Plans >> > > > > >> Once the basic stats are available, there's a lot more the >> handler >> > > > could >> > > > > >> return: >> > > > > >> *Field relationships: field coverage inter-relationships, such >> as >> > > > fields >> > > > > >> that only appear when another field is present or has a certain >> > > value. >> > > > > >> *Visualization: the Admin UI could eventually be updated to >> > display >> > > > this >> > > > > >> information. How exactly is out of scope for this proposal. >> > > > > >> The aim is for discovery to eventually grow from "what fields >> are >> > > > here" >> > > > > >> into a real picture of how a user’s data is structured. >> > > > > >> >> > > > > >> Thank you, >> > > > > >> Srikanth >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- >> > > > > To unsubscribe, e-mail: [email protected] >> > > > > For additional commands, e-mail: [email protected] >> > > > > >> > > > > >> > > > >> > > > This lines up with the idea that in the future, we actually expose >> (in >> > a >> > > > safe-ish manner) all the capabilities of our search environment to >> the >> > > > agents so they build a "perfect" one off query. Isntead of a >> narrow >> > "we >> > > > accept just a query and maybe some filters" and then expand >> everything >> > > out, >> > > > we flip that scirpt and say "here are all the tools, build your >> perfect >> > > > query" and we then provide all the validation/guard rails/workload >> > > > management to support that approach! >> > > > >> > > > I've looked at https://llmstxt.org/ and the idea of having a >> > > > http://localhost:8983/llms.txt that told you about the cluster, >> what >> > > > collections are available based on who you are and your permissions. >> > And >> > > > then imagine we had >> https://localhost:8983/solr/techproducts/llms.txt >> > > > that was specific to that collection? >> > > > >> > > > I think there is some overlap here with what solr-mcp might provide >> in >> > > > being a "safer" way of exposing primitives to Agents.... >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: [email protected] >> > > > For additional commands, e-mail: [email protected] >> > > > >> > > > >> > > >> > >> > > > -- > Thanks, > Aditya >
