Solr faceting supports "field.exists" boolean optimization when you don't
want the count (a nominal '1' is returned instead).  That sounds perfect
for /discovery/fields since we want that to be cheap.  Make counts opt-in.
That said, facet.exists only applies/works on a facet.field (terms
faceting) and not facet.query.  And even then, just sometimes -- when
facet.method is enum. :-(  I confirmed this; sadly our ref guide doesn't
communicate this limitation.  Any way, we can tangentially work on
fixing/optimizing that further.  I think the underlying plumbing to return
early is present.  I filed https://issues.apache.org/jira/browse/SOLR-18666
for that.

On Mon, Nov 16, 2026 at 9:47 PM Aditya Parikh <[email protected]>
wrote:

> Hi Srikanth, David, Amrit, Kevin, all,
>
> +1 from the solr-mcp side. An agent pointed at a Solr instance starts
> every conversation knowing nothing about the collection, so it has to
> discover what's there before it can search it.
>
> What solr-mcp has today (including a PR in flight):
>
> 1.Tool: *get-schema*, Resource: *solr://{collection}/schema*
>    Uses: *GET /solr/{collection}/schema,* which returns the entire schema
> (
> https://solr.apache.org/guide/solr/latest/indexing-guide/schema-api.html#retrieve-the-entire-schema
> )
>    Used for: seeing which field types and analyzer chains exist before
> adding fields or field types.
>    Shortcomings: it describes the configset, not the data: no counts,
> declared-but-empty fields are listed, and a dynamic field in use shows only
> as its pattern.
>
> 2. Tool: list-fields, Resource: solr://{collection}/fields (
> https://github.com/apache/solr-mcp/pull/288)
>    Uses: /admin/luke, which returns the fields that occur in the index,
> each with its type, flags and dynamicBase, and a doc count.
>    Used for: what an agent should read before choosing fields for q, fq,
> faceting and sorting.
>    Shortcomings: the doc count comes from terms, so numeric/date point
> fields and docValues-only fields report 0, meaning not counted rather than
> empty. It also includes deleted
>    documents and can't take an fq.
>
> How solr-mcp would use /discover:
> - *list-fields* and *solr://{collection}/fields* would call
> */discovery/fields* on Solr versions that have it, keeping Luke as the
> fallback.
> - A new tool, *get-field-stats*: distinct count, top values and min/max
> for the fields an agent picks under the same fq.
>
> On Kevin's point about calling out the limitation: for an agent, the place
> to call it out is the response itself. If each count says how it was
> obtained (exact, an index statistic, approximate, or not available), an
> agent won't read "not counted" as "empty".
>
> Aditya
>
> On Mon, Nov 16, 2026 at 4:40 PM Kevin Risden via dev <[email protected]>
> wrote:
>
>> This same problem existed when I was working on the sql handler. Basically
>> how do you handle `*` or other fields that are in the "schema" but are not
>> directly referenced (dynamic fields).
>>
>> https://issues.apache.org/jira/browse/SOLR-8847 and some of the linked
>> jiras highlighted the problems around not knowing what fields were
>> actually
>> indexed.
>>
>> As a side note from implementing a similar lookup in the past, certain
>> field types have these counts easily available and others don't. So the
>> implementation of this can be problematic from a performance standpoint. A
>> field count query is not necessarily fast if it has to iterate and count
>> all the matching documents. It can be quick if there are stats available.
>> Luke can do a lot of this since it ignores deletes. Docvalues can help a
>> lot here too. Its just not always available on say "stored" only fields
>> without indexed or docvalues. I don't think this is a deal breaker, but
>> just a limitation that needs to be called out somewhere.
>>
>> Kevin Risden
>>
>>
>>
>> On Mon, Nov 16, 2026 at 3:50 PM David Smiley via dev <[email protected]
>> >
>> wrote:
>>
>> > Amrit summarized this really well!
>> >
>> > So I'm not seeing any resistance.  We could create a JIRA.
>> >
>> > I'm inclined to suggest a first stage that just lists the (populated)
>> > fields with type info and doc-counts (applying filters).  Maybe
>> > "/discovery/fields"?  I don't think this would be combined with
>> subsequent
>> > detailed statistics per field on the same request; thus, the requests
>> are
>> > fundamentally different/bifurcated.  I could see it taking a field regex
>> > (default none -- list all), on top of any "fq" to filter (if any).
>> > Safety:  This might still be rather expensive if there are many
>> fields.  At
>> > least value-existence is exposed as a DocIdSetIterator (DISI) at the
>> Lucene
>> > layer for almost all index structures (albeit nothing for
>> > stored/term-vectors).  A field that is stored-only (or TV-only) will
>> have
>> > no stats for this handler to return.  Adding field paging sounds more
>> > awkward than it's worth; we want this to be easy to use.  It can be
>> added
>> > later, too.
>> >
>> > A client could then later request statistics per-field by listing them
>> > explicitly, not unlike "facet.field", and with the "f.<fieldname>..."
>> > override potential if needed to tune what's needed.  Maybe I'm thinking
>> of
>> > this in a very classic Solr API v1 way... I know JSON Facets and to some
>> > extent V2 likes JSON input.
>> >
>> >
>> > On Sun, Nov 15, 2026 at 5:26 PM Amrit Sarkar <[email protected]>
>> > wrote:
>> >
>> > > Hi Srikanth, David,
>> > >
>> > > +1, I like this. One concrete gap it would close: solr-mcp's
>> get-schema
>> > > <
>> > >
>> >
>> https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248
>> > > >
>> > > only returns defined fields, so dynamic fields that actually hold data
>> > are
>> > > invisible to an agent today.
>> > >
>> > > A few use cases I'd add:
>> > >
>> > >    - Onboarding a collection someone else built, which is routine for
>> > >    platform teams like the one at my company that run search for many
>> > >    product teams. The same goes for multi-tenant collections where doc
>> > > types
>> > >    carry different fields
>> > >    - Data quality after ingestion: a field's fill rate dropping from
>> 98%
>> > to
>> > >    40% usually means a broken pipeline. I deal with this almost weekly
>> > with
>> > >    other data stores, and a cheap way to see it would help a lot
>> > >    - Auditing dynamic field sprawl before a schema cleanup or
>> migration
>> > >
>> > > On David's point about checking cardinality first, Druid is a useful
>> > > reference. I've used it for metrics OLAP, and its segmentMetadata
>> > > <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/>
>> > > query
>> > > reports per-column cardinality and min/max from metadata it already
>> > stores,
>> > > without scanning data. If "/discover" can answer "is this field safe
>> to
>> > > facet" from stats the index already keeps, the safety gate stays cheap
>> > and
>> > > only borderline fields need the heavier approximate check.
>> > >
>> > > Will closely follow this development.
>> > >
>> > > Amrit
>> > >
>> > > On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected]
>> >
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote:
>> > > > > Hi
>> > > > >
>> > > > > Sounds like something that could be useful, although it would be a
>> > bit
>> > > > uncommon
>> > > > > to have a bunch of data in Solr already without already knowing
>> your
>> > > > fields?
>> > > > >
>> > > > > The proposal reminds me of OpenRefine https://openrefine.org
>> which
>> > is
>> > > a
>> > > > great
>> > > > > tool to use *before* you start indexing a dataset to Solr. It will
>> > eat
>> > > a
>> > > > CSV, XMl or
>> > > > > whatever, and let you discover what's in the data, cardinality,
>> data
>> > > > quality etc.
>> > > > >
>> > > > > A different use case, but just wanted to x-ref it.
>> > > > >
>> > > > > Jan
>> > > > >
>> > > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev <
>> > > > [email protected]>:
>> > > > > >
>> > > > > > (disclaimer: I worked with Srikanth on this proposal)
>> > > > > >
>> > > > > > I've seen a number of situations where we fumble in the absence
>> of
>> > > > this or
>> > > > > > things are built similar outside of Solr.  The experts here may
>> not
>> > > > fully
>> > > > > > appreciate this as we are experts; Solr newbies and sometimes
>> > agents,
>> > > > are
>> > > > > > not.  I think there is growing utility for this with LLM-based
>> > > > > > interrogation of what's in Solr, and I could definitely see
>> Solr's
>> > > MCP
>> > > > > > using this.
>> > > > > >
>> > > > > > Safety: This handler is only intended for use by a single
>> > person/LLM
>> > > > at a
>> > > > > > time to protect the "production" load assumed to be happening
>> > > > concurrently
>> > > > > > on the cluster.  Of course this aspect could be toggled off if
>> > > desired
>> > > > > > (e.g. a "dev" cluster).  Assume a timeAllowed/cpuAllowed.  Maybe
>> > even
>> > > > rate
>> > > > > > limit to no more than one or two requests at a time (Solr has
>> code
>> > > for
>> > > > > > this).
>> > > > > >
>> > > > > > There will be a slippery slope & tension on what /discover
>> should
>> > do
>> > > > versus
>> > > > > > leave to existing/standard Solr methods.  In my view, /discover
>> > > > shouldn't
>> > > > > > return entire documents -- just use /select if that is needed.
>> > > > > >
>> > > > > > I think it'd be helpful for the response structure to mirror
>> > /select
>> > > > where
>> > > > > > it fits.  Certainly for facets & stats, which is mostly what
>> > > /discover
>> > > > > > exposes.  Much of the logic of /discover is just applying safe
>> > > > defaults on
>> > > > > > what facet/stats to return (possibly blocking "unsafe"
>> requests),
>> > > plus
>> > > > a
>> > > > > > rather new field coverage component.  That's powered by
>> > > > > > "facet.query=field:*" ultimately, but still.  In order to
>> determine
>> > > if
>> > > > it's
>> > > > > > "safe" to get the facet values for a field, an approximate
>> > > cardinality
>> > > > > > check may be necessary first.
>> > > > > >
>> > > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120
>> > PARK)
>> > > > via
>> > > > > > dev <[email protected]> wrote:
>> > > > > >
>> > > > > >> Hi all,
>> > > > > >>
>> > > > > >> I'd like your thoughts on adding a new data discovery handler
>> to
>> > > > Solr. The
>> > > > > >> design would revolve around two principles, simplicity and
>> safety.
>> > > It
>> > > > > >> should be easy for anyone to use, and it should have
>> protections
>> > > > built in
>> > > > > >> so that exploring your data never puts the cluster at risk. In
>> > some
>> > > > cases,
>> > > > > >> that may mean returning approximate results instead of exact
>> ones.
>> > > > > >> Questions, comments and concerns are all welcome.
>> > > > > >>
>> > > > > >> Problem
>> > > > > >> People often come to Solr with a question but no idea how the
>> data
>> > > is
>> > > > > >> structured. Which fields exist, which ones are populated, and
>> what
>> > > do
>> > > > they
>> > > > > >> contain? Developers, analysts and (increasingly) LLM agents all
>> > run
>> > > > into
>> > > > > >> this, and Solr makes these questions hard to answer.
>> Additionally,
>> > > > dynamic
>> > > > > >> fields mean the schema doesn't show which fields actually
>> exist,
>> > and
>> > > > > >> documents in the same collection can often have different
>> fields.
>> > > > > >> Performing a very naive data discovery has the potential to
>> > execute
>> > > > very
>> > > > > >> expensive Solr queries (e.g. faceting on a high cardinality
>> field,
>> > > > > >> returning a significant number of rows, etc…).
>> > > > > >>
>> > > > > >> What exists today
>> > > > > >> *Schema XML/Schema API shows the fields that are defined, but
>> not
>> > > the
>> > > > > >> dynamic fields that actually exist in the index. It also has no
>> > > > statistics,
>> > > > > >> so clients are unable to tell which fields have data.
>> > > > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow
>> them
>> > > > down
>> > > > > >> with filter queries. Newer versions of Solr can run it across
>> > > shards,
>> > > > but
>> > > > > >> the detailed term stats are still per-shard. Finally, it's
>> also an
>> > > > advanced
>> > > > > >> tool and hard for beginners to use.
>> > > > > >> *Facets can answer the question, but only if you already know
>> how
>> > to
>> > > > write
>> > > > > >> a good Solr query. Every client has to build a facet for each
>> > field
>> > > > and
>> > > > > >> batch them, which isn't obvious to someone new to Solr. The
>> > concern
>> > > of
>> > > > > >> expensive queries is directly related to utilizing facets.
>> > > > > >>
>> > > > > >> Proposal
>> > > > > >> A new DiscoveryHandler that takes filter queries (fq) and
>> > supports:
>> > > > > >> *Field coverage: how many matching documents have each field
>> > > > > >> *Field statistics/facets: distinct count, top values, and
>> min/max
>> > > for
>> > > > > >> numbers and dates. For low cardinality fields, the handler
>> would
>> > > > return the
>> > > > > >> top facets. Where an exact answer would be too expensive, these
>> > may
>> > > be
>> > > > > >> provided as approximate values.
>> > > > > >> *Pivot field: optionally compute stats and coverage by the
>> values
>> > > of a
>> > > > > >> field, such as the document type. Each value-group then gets
>> its
>> > own
>> > > > > >> numbers.
>> > > > > >> Future Plans
>> > > > > >> Once the basic stats are available, there's a lot more the
>> handler
>> > > > could
>> > > > > >> return:
>> > > > > >> *Field relationships: field coverage inter-relationships, such
>> as
>> > > > fields
>> > > > > >> that only appear when another field is present or has a certain
>> > > value.
>> > > > > >> *Visualization: the Admin UI could eventually be updated to
>> > display
>> > > > this
>> > > > > >> information. How exactly is out of scope for this proposal.
>> > > > > >> The aim is for discovery to eventually grow from "what fields
>> are
>> > > > here"
>> > > > > >> into a real picture of how a user’s data is structured.
>> > > > > >>
>> > > > > >> Thank you,
>> > > > > >> Srikanth
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail: [email protected]
>> > > > > For additional commands, e-mail: [email protected]
>> > > > >
>> > > > >
>> > > >
>> > > > This lines up with the idea that in the future, we actually expose
>> (in
>> > a
>> > > > safe-ish manner) all the capabilities of our search environment to
>> the
>> > > > agents so they build a "perfect" one off query.   Isntead of a
>> narrow
>> > "we
>> > > > accept just a query and maybe some filters" and then expand
>> everything
>> > > out,
>> > > > we flip that scirpt and say "here are all the tools, build your
>> perfect
>> > > > query" and we then provide all the validation/guard rails/workload
>> > > > management to support that approach!
>> > > >
>> > > > I've looked at https://llmstxt.org/ and the idea of having a
>> > > > http://localhost:8983/llms.txt that told you about the cluster,
>> what
>> > > > collections are available based on who you are and your permissions.
>> > And
>> > > > then imagine we had
>> https://localhost:8983/solr/techproducts/llms.txt
>> > > > that was specific to that collection?
>> > > >
>> > > > I think there is some overlap here with what solr-mcp might provide
>> in
>> > > > being a "safer" way of exposing primitives to Agents....
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: [email protected]
>> > > > For additional commands, e-mail: [email protected]
>> > > >
>> > > >
>> > >
>> >
>>
>
>
> --
> Thanks,
> Aditya
>

Reply via email to