Hi Srikanth, David, Amrit, Kevin, all,
+1 from the solr-mcp side. An agent pointed at a Solr instance starts every
conversation knowing nothing about the collection, so it has to discover
what's there before it can search it.
What solr-mcp has today (including a PR in flight):
1.Tool: *get-schema*, Resource: *solr://{collection}/schema*
Uses: *GET /solr/{collection}/schema,* which returns the entire schema (
https://solr.apache.org/guide/solr/latest/indexing-guide/schema-api.html#retrieve-the-entire-schema
)
Used for: seeing which field types and analyzer chains exist before
adding fields or field types.
Shortcomings: it describes the configset, not the data: no counts,
declared-but-empty fields are listed, and a dynamic field in use shows only
as its pattern.
2. Tool: list-fields, Resource: solr://{collection}/fields (
https://github.com/apache/solr-mcp/pull/288)
Uses: /admin/luke, which returns the fields that occur in the index,
each with its type, flags and dynamicBase, and a doc count.
Used for: what an agent should read before choosing fields for q, fq,
faceting and sorting.
Shortcomings: the doc count comes from terms, so numeric/date point
fields and docValues-only fields report 0, meaning not counted rather than
empty. It also includes deleted
documents and can't take an fq.
How solr-mcp would use /discover:
- *list-fields* and *solr://{collection}/fields* would call
*/discovery/fields* on Solr versions that have it, keeping Luke as the
fallback.
- A new tool, *get-field-stats*: distinct count, top values and min/max for
the fields an agent picks under the same fq.
On Kevin's point about calling out the limitation: for an agent, the place
to call it out is the response itself. If each count says how it was
obtained (exact, an index statistic, approximate, or not available), an
agent won't read "not counted" as "empty".
Aditya
On Mon, Nov 16, 2026 at 4:40 PM Kevin Risden via dev <[email protected]>
wrote:
> This same problem existed when I was working on the sql handler. Basically
> how do you handle `*` or other fields that are in the "schema" but are not
> directly referenced (dynamic fields).
>
> https://issues.apache.org/jira/browse/SOLR-8847 and some of the linked
> jiras highlighted the problems around not knowing what fields were actually
> indexed.
>
> As a side note from implementing a similar lookup in the past, certain
> field types have these counts easily available and others don't. So the
> implementation of this can be problematic from a performance standpoint. A
> field count query is not necessarily fast if it has to iterate and count
> all the matching documents. It can be quick if there are stats available.
> Luke can do a lot of this since it ignores deletes. Docvalues can help a
> lot here too. Its just not always available on say "stored" only fields
> without indexed or docvalues. I don't think this is a deal breaker, but
> just a limitation that needs to be called out somewhere.
>
> Kevin Risden
>
>
>
> On Mon, Nov 16, 2026 at 3:50 PM David Smiley via dev <[email protected]>
> wrote:
>
> > Amrit summarized this really well!
> >
> > So I'm not seeing any resistance. We could create a JIRA.
> >
> > I'm inclined to suggest a first stage that just lists the (populated)
> > fields with type info and doc-counts (applying filters). Maybe
> > "/discovery/fields"? I don't think this would be combined with
> subsequent
> > detailed statistics per field on the same request; thus, the requests are
> > fundamentally different/bifurcated. I could see it taking a field regex
> > (default none -- list all), on top of any "fq" to filter (if any).
> > Safety: This might still be rather expensive if there are many fields.
> At
> > least value-existence is exposed as a DocIdSetIterator (DISI) at the
> Lucene
> > layer for almost all index structures (albeit nothing for
> > stored/term-vectors). A field that is stored-only (or TV-only) will have
> > no stats for this handler to return. Adding field paging sounds more
> > awkward than it's worth; we want this to be easy to use. It can be added
> > later, too.
> >
> > A client could then later request statistics per-field by listing them
> > explicitly, not unlike "facet.field", and with the "f.<fieldname>..."
> > override potential if needed to tune what's needed. Maybe I'm thinking
> of
> > this in a very classic Solr API v1 way... I know JSON Facets and to some
> > extent V2 likes JSON input.
> >
> >
> > On Sun, Nov 15, 2026 at 5:26 PM Amrit Sarkar <[email protected]>
> > wrote:
> >
> > > Hi Srikanth, David,
> > >
> > > +1, I like this. One concrete gap it would close: solr-mcp's get-schema
> > > <
> > >
> >
> https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248
> > > >
> > > only returns defined fields, so dynamic fields that actually hold data
> > are
> > > invisible to an agent today.
> > >
> > > A few use cases I'd add:
> > >
> > > - Onboarding a collection someone else built, which is routine for
> > > platform teams like the one at my company that run search for many
> > > product teams. The same goes for multi-tenant collections where doc
> > > types
> > > carry different fields
> > > - Data quality after ingestion: a field's fill rate dropping from
> 98%
> > to
> > > 40% usually means a broken pipeline. I deal with this almost weekly
> > with
> > > other data stores, and a cheap way to see it would help a lot
> > > - Auditing dynamic field sprawl before a schema cleanup or migration
> > >
> > > On David's point about checking cardinality first, Druid is a useful
> > > reference. I've used it for metrics OLAP, and its segmentMetadata
> > > <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/>
> > > query
> > > reports per-column cardinality and min/max from metadata it already
> > stores,
> > > without scanning data. If "/discover" can answer "is this field safe to
> > > facet" from stats the index already keeps, the safety gate stays cheap
> > and
> > > only borderline fields need the heavier approximate check.
> > >
> > > Will closely follow this development.
> > >
> > > Amrit
> > >
> > > On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected]>
> > > wrote:
> > >
> > > >
> > > >
> > > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote:
> > > > > Hi
> > > > >
> > > > > Sounds like something that could be useful, although it would be a
> > bit
> > > > uncommon
> > > > > to have a bunch of data in Solr already without already knowing
> your
> > > > fields?
> > > > >
> > > > > The proposal reminds me of OpenRefine https://openrefine.org which
> > is
> > > a
> > > > great
> > > > > tool to use *before* you start indexing a dataset to Solr. It will
> > eat
> > > a
> > > > CSV, XMl or
> > > > > whatever, and let you discover what's in the data, cardinality,
> data
> > > > quality etc.
> > > > >
> > > > > A different use case, but just wanted to x-ref it.
> > > > >
> > > > > Jan
> > > > >
> > > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev <
> > > > [email protected]>:
> > > > > >
> > > > > > (disclaimer: I worked with Srikanth on this proposal)
> > > > > >
> > > > > > I've seen a number of situations where we fumble in the absence
> of
> > > > this or
> > > > > > things are built similar outside of Solr. The experts here may
> not
> > > > fully
> > > > > > appreciate this as we are experts; Solr newbies and sometimes
> > agents,
> > > > are
> > > > > > not. I think there is growing utility for this with LLM-based
> > > > > > interrogation of what's in Solr, and I could definitely see
> Solr's
> > > MCP
> > > > > > using this.
> > > > > >
> > > > > > Safety: This handler is only intended for use by a single
> > person/LLM
> > > > at a
> > > > > > time to protect the "production" load assumed to be happening
> > > > concurrently
> > > > > > on the cluster. Of course this aspect could be toggled off if
> > > desired
> > > > > > (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe
> > even
> > > > rate
> > > > > > limit to no more than one or two requests at a time (Solr has
> code
> > > for
> > > > > > this).
> > > > > >
> > > > > > There will be a slippery slope & tension on what /discover should
> > do
> > > > versus
> > > > > > leave to existing/standard Solr methods. In my view, /discover
> > > > shouldn't
> > > > > > return entire documents -- just use /select if that is needed.
> > > > > >
> > > > > > I think it'd be helpful for the response structure to mirror
> > /select
> > > > where
> > > > > > it fits. Certainly for facets & stats, which is mostly what
> > > /discover
> > > > > > exposes. Much of the logic of /discover is just applying safe
> > > > defaults on
> > > > > > what facet/stats to return (possibly blocking "unsafe" requests),
> > > plus
> > > > a
> > > > > > rather new field coverage component. That's powered by
> > > > > > "facet.query=field:*" ultimately, but still. In order to
> determine
> > > if
> > > > it's
> > > > > > "safe" to get the facet values for a field, an approximate
> > > cardinality
> > > > > > check may be necessary first.
> > > > > >
> > > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120
> > PARK)
> > > > via
> > > > > > dev <[email protected]> wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I'd like your thoughts on adding a new data discovery handler to
> > > > Solr. The
> > > > > >> design would revolve around two principles, simplicity and
> safety.
> > > It
> > > > > >> should be easy for anyone to use, and it should have protections
> > > > built in
> > > > > >> so that exploring your data never puts the cluster at risk. In
> > some
> > > > cases,
> > > > > >> that may mean returning approximate results instead of exact
> ones.
> > > > > >> Questions, comments and concerns are all welcome.
> > > > > >>
> > > > > >> Problem
> > > > > >> People often come to Solr with a question but no idea how the
> data
> > > is
> > > > > >> structured. Which fields exist, which ones are populated, and
> what
> > > do
> > > > they
> > > > > >> contain? Developers, analysts and (increasingly) LLM agents all
> > run
> > > > into
> > > > > >> this, and Solr makes these questions hard to answer.
> Additionally,
> > > > dynamic
> > > > > >> fields mean the schema doesn't show which fields actually exist,
> > and
> > > > > >> documents in the same collection can often have different
> fields.
> > > > > >> Performing a very naive data discovery has the potential to
> > execute
> > > > very
> > > > > >> expensive Solr queries (e.g. faceting on a high cardinality
> field,
> > > > > >> returning a significant number of rows, etc…).
> > > > > >>
> > > > > >> What exists today
> > > > > >> *Schema XML/Schema API shows the fields that are defined, but
> not
> > > the
> > > > > >> dynamic fields that actually exist in the index. It also has no
> > > > statistics,
> > > > > >> so clients are unable to tell which fields have data.
> > > > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow
> them
> > > > down
> > > > > >> with filter queries. Newer versions of Solr can run it across
> > > shards,
> > > > but
> > > > > >> the detailed term stats are still per-shard. Finally, it's also
> an
> > > > advanced
> > > > > >> tool and hard for beginners to use.
> > > > > >> *Facets can answer the question, but only if you already know
> how
> > to
> > > > write
> > > > > >> a good Solr query. Every client has to build a facet for each
> > field
> > > > and
> > > > > >> batch them, which isn't obvious to someone new to Solr. The
> > concern
> > > of
> > > > > >> expensive queries is directly related to utilizing facets.
> > > > > >>
> > > > > >> Proposal
> > > > > >> A new DiscoveryHandler that takes filter queries (fq) and
> > supports:
> > > > > >> *Field coverage: how many matching documents have each field
> > > > > >> *Field statistics/facets: distinct count, top values, and
> min/max
> > > for
> > > > > >> numbers and dates. For low cardinality fields, the handler would
> > > > return the
> > > > > >> top facets. Where an exact answer would be too expensive, these
> > may
> > > be
> > > > > >> provided as approximate values.
> > > > > >> *Pivot field: optionally compute stats and coverage by the
> values
> > > of a
> > > > > >> field, such as the document type. Each value-group then gets its
> > own
> > > > > >> numbers.
> > > > > >> Future Plans
> > > > > >> Once the basic stats are available, there's a lot more the
> handler
> > > > could
> > > > > >> return:
> > > > > >> *Field relationships: field coverage inter-relationships, such
> as
> > > > fields
> > > > > >> that only appear when another field is present or has a certain
> > > value.
> > > > > >> *Visualization: the Admin UI could eventually be updated to
> > display
> > > > this
> > > > > >> information. How exactly is out of scope for this proposal.
> > > > > >> The aim is for discovery to eventually grow from "what fields
> are
> > > > here"
> > > > > >> into a real picture of how a user’s data is structured.
> > > > > >>
> > > > > >> Thank you,
> > > > > >> Srikanth
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: [email protected]
> > > > > For additional commands, e-mail: [email protected]
> > > > >
> > > > >
> > > >
> > > > This lines up with the idea that in the future, we actually expose
> (in
> > a
> > > > safe-ish manner) all the capabilities of our search environment to
> the
> > > > agents so they build a "perfect" one off query. Isntead of a narrow
> > "we
> > > > accept just a query and maybe some filters" and then expand
> everything
> > > out,
> > > > we flip that scirpt and say "here are all the tools, build your
> perfect
> > > > query" and we then provide all the validation/guard rails/workload
> > > > management to support that approach!
> > > >
> > > > I've looked at https://llmstxt.org/ and the idea of having a
> > > > http://localhost:8983/llms.txt that told you about the cluster, what
> > > > collections are available based on who you are and your permissions.
> > And
> > > > then imagine we had
> https://localhost:8983/solr/techproducts/llms.txt
> > > > that was specific to that collection?
> > > >
> > > > I think there is some overlap here with what solr-mcp might provide
> in
> > > > being a "safer" way of exposing primitives to Agents....
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > > >
> > >
> >
>
--
Thanks,
Aditya