This same problem existed when I was working on the sql handler. Basically
how do you handle `*` or other fields that are in the "schema" but are not
directly referenced (dynamic fields).

https://issues.apache.org/jira/browse/SOLR-8847 and some of the linked
jiras highlighted the problems around not knowing what fields were actually
indexed.

As a side note from implementing a similar lookup in the past, certain
field types have these counts easily available and others don't. So the
implementation of this can be problematic from a performance standpoint. A
field count query is not necessarily fast if it has to iterate and count
all the matching documents. It can be quick if there are stats available.
Luke can do a lot of this since it ignores deletes. Docvalues can help a
lot here too. Its just not always available on say "stored" only fields
without indexed or docvalues. I don't think this is a deal breaker, but
just a limitation that needs to be called out somewhere.

Kevin Risden



On Mon, Nov 16, 2026 at 3:50 PM David Smiley via dev <[email protected]>
wrote:

> Amrit summarized this really well!
>
> So I'm not seeing any resistance.  We could create a JIRA.
>
> I'm inclined to suggest a first stage that just lists the (populated)
> fields with type info and doc-counts (applying filters).  Maybe
> "/discovery/fields"?  I don't think this would be combined with subsequent
> detailed statistics per field on the same request; thus, the requests are
> fundamentally different/bifurcated.  I could see it taking a field regex
> (default none -- list all), on top of any "fq" to filter (if any).
> Safety:  This might still be rather expensive if there are many fields.  At
> least value-existence is exposed as a DocIdSetIterator (DISI) at the Lucene
> layer for almost all index structures (albeit nothing for
> stored/term-vectors).  A field that is stored-only (or TV-only) will have
> no stats for this handler to return.  Adding field paging sounds more
> awkward than it's worth; we want this to be easy to use.  It can be added
> later, too.
>
> A client could then later request statistics per-field by listing them
> explicitly, not unlike "facet.field", and with the "f.<fieldname>..."
> override potential if needed to tune what's needed.  Maybe I'm thinking of
> this in a very classic Solr API v1 way... I know JSON Facets and to some
> extent V2 likes JSON input.
>
>
> On Sun, Nov 15, 2026 at 5:26 PM Amrit Sarkar <[email protected]>
> wrote:
>
> > Hi Srikanth, David,
> >
> > +1, I like this. One concrete gap it would close: solr-mcp's get-schema
> > <
> >
> https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248
> > >
> > only returns defined fields, so dynamic fields that actually hold data
> are
> > invisible to an agent today.
> >
> > A few use cases I'd add:
> >
> >    - Onboarding a collection someone else built, which is routine for
> >    platform teams like the one at my company that run search for many
> >    product teams. The same goes for multi-tenant collections where doc
> > types
> >    carry different fields
> >    - Data quality after ingestion: a field's fill rate dropping from 98%
> to
> >    40% usually means a broken pipeline. I deal with this almost weekly
> with
> >    other data stores, and a cheap way to see it would help a lot
> >    - Auditing dynamic field sprawl before a schema cleanup or migration
> >
> > On David's point about checking cardinality first, Druid is a useful
> > reference. I've used it for metrics OLAP, and its segmentMetadata
> > <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/>
> > query
> > reports per-column cardinality and min/max from metadata it already
> stores,
> > without scanning data. If "/discover" can answer "is this field safe to
> > facet" from stats the index already keeps, the safety gate stays cheap
> and
> > only borderline fields need the heavier approximate check.
> >
> > Will closely follow this development.
> >
> > Amrit
> >
> > On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected]>
> > wrote:
> >
> > >
> > >
> > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote:
> > > > Hi
> > > >
> > > > Sounds like something that could be useful, although it would be a
> bit
> > > uncommon
> > > > to have a bunch of data in Solr already without already knowing your
> > > fields?
> > > >
> > > > The proposal reminds me of OpenRefine https://openrefine.org which
> is
> > a
> > > great
> > > > tool to use *before* you start indexing a dataset to Solr. It will
> eat
> > a
> > > CSV, XMl or
> > > > whatever, and let you discover what's in the data, cardinality, data
> > > quality etc.
> > > >
> > > > A different use case, but just wanted to x-ref it.
> > > >
> > > > Jan
> > > >
> > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev <
> > > [email protected]>:
> > > > >
> > > > > (disclaimer: I worked with Srikanth on this proposal)
> > > > >
> > > > > I've seen a number of situations where we fumble in the absence of
> > > this or
> > > > > things are built similar outside of Solr.  The experts here may not
> > > fully
> > > > > appreciate this as we are experts; Solr newbies and sometimes
> agents,
> > > are
> > > > > not.  I think there is growing utility for this with LLM-based
> > > > > interrogation of what's in Solr, and I could definitely see Solr's
> > MCP
> > > > > using this.
> > > > >
> > > > > Safety: This handler is only intended for use by a single
> person/LLM
> > > at a
> > > > > time to protect the "production" load assumed to be happening
> > > concurrently
> > > > > on the cluster.  Of course this aspect could be toggled off if
> > desired
> > > > > (e.g. a "dev" cluster).  Assume a timeAllowed/cpuAllowed.  Maybe
> even
> > > rate
> > > > > limit to no more than one or two requests at a time (Solr has code
> > for
> > > > > this).
> > > > >
> > > > > There will be a slippery slope & tension on what /discover should
> do
> > > versus
> > > > > leave to existing/standard Solr methods.  In my view, /discover
> > > shouldn't
> > > > > return entire documents -- just use /select if that is needed.
> > > > >
> > > > > I think it'd be helpful for the response structure to mirror
> /select
> > > where
> > > > > it fits.  Certainly for facets & stats, which is mostly what
> > /discover
> > > > > exposes.  Much of the logic of /discover is just applying safe
> > > defaults on
> > > > > what facet/stats to return (possibly blocking "unsafe" requests),
> > plus
> > > a
> > > > > rather new field coverage component.  That's powered by
> > > > > "facet.query=field:*" ultimately, but still.  In order to determine
> > if
> > > it's
> > > > > "safe" to get the facet values for a field, an approximate
> > cardinality
> > > > > check may be necessary first.
> > > > >
> > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120
> PARK)
> > > via
> > > > > dev <[email protected]> wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I'd like your thoughts on adding a new data discovery handler to
> > > Solr. The
> > > > >> design would revolve around two principles, simplicity and safety.
> > It
> > > > >> should be easy for anyone to use, and it should have protections
> > > built in
> > > > >> so that exploring your data never puts the cluster at risk. In
> some
> > > cases,
> > > > >> that may mean returning approximate results instead of exact ones.
> > > > >> Questions, comments and concerns are all welcome.
> > > > >>
> > > > >> Problem
> > > > >> People often come to Solr with a question but no idea how the data
> > is
> > > > >> structured. Which fields exist, which ones are populated, and what
> > do
> > > they
> > > > >> contain? Developers, analysts and (increasingly) LLM agents all
> run
> > > into
> > > > >> this, and Solr makes these questions hard to answer. Additionally,
> > > dynamic
> > > > >> fields mean the schema doesn't show which fields actually exist,
> and
> > > > >> documents in the same collection can often have different fields.
> > > > >> Performing a very naive data discovery has the potential to
> execute
> > > very
> > > > >> expensive Solr queries (e.g. faceting on a high cardinality field,
> > > > >> returning a significant number of rows, etc…).
> > > > >>
> > > > >> What exists today
> > > > >> *Schema XML/Schema API shows the fields that are defined, but not
> > the
> > > > >> dynamic fields that actually exist in the index. It also has no
> > > statistics,
> > > > >> so clients are unable to tell which fields have data.
> > > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow them
> > > down
> > > > >> with filter queries. Newer versions of Solr can run it across
> > shards,
> > > but
> > > > >> the detailed term stats are still per-shard. Finally, it's also an
> > > advanced
> > > > >> tool and hard for beginners to use.
> > > > >> *Facets can answer the question, but only if you already know how
> to
> > > write
> > > > >> a good Solr query. Every client has to build a facet for each
> field
> > > and
> > > > >> batch them, which isn't obvious to someone new to Solr. The
> concern
> > of
> > > > >> expensive queries is directly related to utilizing facets.
> > > > >>
> > > > >> Proposal
> > > > >> A new DiscoveryHandler that takes filter queries (fq) and
> supports:
> > > > >> *Field coverage: how many matching documents have each field
> > > > >> *Field statistics/facets: distinct count, top values, and min/max
> > for
> > > > >> numbers and dates. For low cardinality fields, the handler would
> > > return the
> > > > >> top facets. Where an exact answer would be too expensive, these
> may
> > be
> > > > >> provided as approximate values.
> > > > >> *Pivot field: optionally compute stats and coverage by the values
> > of a
> > > > >> field, such as the document type. Each value-group then gets its
> own
> > > > >> numbers.
> > > > >> Future Plans
> > > > >> Once the basic stats are available, there's a lot more the handler
> > > could
> > > > >> return:
> > > > >> *Field relationships: field coverage inter-relationships, such as
> > > fields
> > > > >> that only appear when another field is present or has a certain
> > value.
> > > > >> *Visualization: the Admin UI could eventually be updated to
> display
> > > this
> > > > >> information. How exactly is out of scope for this proposal.
> > > > >> The aim is for discovery to eventually grow from "what fields are
> > > here"
> > > > >> into a real picture of how a user’s data is structured.
> > > > >>
> > > > >> Thank you,
> > > > >> Srikanth
> > > > >>
> > > > >>
> > > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: [email protected]
> > > > For additional commands, e-mail: [email protected]
> > > >
> > > >
> > >
> > > This lines up with the idea that in the future, we actually expose (in
> a
> > > safe-ish manner) all the capabilities of our search environment to the
> > > agents so they build a "perfect" one off query.   Isntead of a narrow
> "we
> > > accept just a query and maybe some filters" and then expand everything
> > out,
> > > we flip that scirpt and say "here are all the tools, build your perfect
> > > query" and we then provide all the validation/guard rails/workload
> > > management to support that approach!
> > >
> > > I've looked at https://llmstxt.org/ and the idea of having a
> > > http://localhost:8983/llms.txt that told you about the cluster, what
> > > collections are available based on who you are and your permissions.
> And
> > > then imagine we had https://localhost:8983/solr/techproducts/llms.txt
> > > that was specific to that collection?
> > >
> > > I think there is some overlap here with what solr-mcp might provide in
> > > being a "safer" way of exposing primitives to Agents....
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [email protected]
> > > For additional commands, e-mail: [email protected]
> > >
> > >
> >
>

Reply via email to