On 2026/11/14 17:25:37 Jan Høydahl via dev wrote:
> Hi
> 
> Sounds like something that could be useful, although it would be a bit 
> uncommon
> to have a bunch of data in Solr already without already knowing your fields?
> 
> The proposal reminds me of OpenRefine https://openrefine.org which is a great
> tool to use *before* you start indexing a dataset to Solr. It will eat a CSV, 
> XMl or
> whatever, and let you discover what's in the data, cardinality, data quality 
> etc.
> 
> A different use case, but just wanted to x-ref it.
> 
> Jan
> 
> > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev <[email protected]>:
> > 
> > (disclaimer: I worked with Srikanth on this proposal)
> > 
> > I've seen a number of situations where we fumble in the absence of this or
> > things are built similar outside of Solr.  The experts here may not fully
> > appreciate this as we are experts; Solr newbies and sometimes agents, are
> > not.  I think there is growing utility for this with LLM-based
> > interrogation of what's in Solr, and I could definitely see Solr's MCP
> > using this.
> > 
> > Safety: This handler is only intended for use by a single person/LLM at a
> > time to protect the "production" load assumed to be happening concurrently
> > on the cluster.  Of course this aspect could be toggled off if desired
> > (e.g. a "dev" cluster).  Assume a timeAllowed/cpuAllowed.  Maybe even rate
> > limit to no more than one or two requests at a time (Solr has code for
> > this).
> > 
> > There will be a slippery slope & tension on what /discover should do versus
> > leave to existing/standard Solr methods.  In my view, /discover shouldn't
> > return entire documents -- just use /select if that is needed.
> > 
> > I think it'd be helpful for the response structure to mirror /select where
> > it fits.  Certainly for facets & stats, which is mostly what /discover
> > exposes.  Much of the logic of /discover is just applying safe defaults on
> > what facet/stats to return (possibly blocking "unsafe" requests), plus a
> > rather new field coverage component.  That's powered by
> > "facet.query=field:*" ultimately, but still.  In order to determine if it's
> > "safe" to get the facet values for a field, an approximate cardinality
> > check may be necessary first.
> > 
> > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 PARK) via
> > dev <[email protected]> wrote:
> > 
> >> Hi all,
> >> 
> >> I'd like your thoughts on adding a new data discovery handler to Solr. The
> >> design would revolve around two principles, simplicity and safety. It
> >> should be easy for anyone to use, and it should have protections built in
> >> so that exploring your data never puts the cluster at risk. In some cases,
> >> that may mean returning approximate results instead of exact ones.
> >> Questions, comments and concerns are all welcome.
> >> 
> >> Problem
> >> People often come to Solr with a question but no idea how the data is
> >> structured. Which fields exist, which ones are populated, and what do they
> >> contain? Developers, analysts and (increasingly) LLM agents all run into
> >> this, and Solr makes these questions hard to answer. Additionally, dynamic
> >> fields mean the schema doesn't show which fields actually exist, and
> >> documents in the same collection can often have different fields.
> >> Performing a very naive data discovery has the potential to execute very
> >> expensive Solr queries (e.g. faceting on a high cardinality field,
> >> returning a significant number of rows, etc…).
> >> 
> >> What exists today
> >> *Schema XML/Schema API shows the fields that are defined, but not the
> >> dynamic fields that actually exist in the index. It also has no statistics,
> >> so clients are unable to tell which fields have data.
> >> *Luke (/admin/luke) has per-field stats, but you can't narrow them down
> >> with filter queries. Newer versions of Solr can run it across shards, but
> >> the detailed term stats are still per-shard. Finally, it's also an advanced
> >> tool and hard for beginners to use.
> >> *Facets can answer the question, but only if you already know how to write
> >> a good Solr query. Every client has to build a facet for each field and
> >> batch them, which isn't obvious to someone new to Solr. The concern of
> >> expensive queries is directly related to utilizing facets.
> >> 
> >> Proposal
> >> A new DiscoveryHandler that takes filter queries (fq) and supports:
> >> *Field coverage: how many matching documents have each field
> >> *Field statistics/facets: distinct count, top values, and min/max for
> >> numbers and dates. For low cardinality fields, the handler would return the
> >> top facets. Where an exact answer would be too expensive, these may be
> >> provided as approximate values.
> >> *Pivot field: optionally compute stats and coverage by the values of a
> >> field, such as the document type. Each value-group then gets its own
> >> numbers.
> >> Future Plans
> >> Once the basic stats are available, there's a lot more the handler could
> >> return:
> >> *Field relationships: field coverage inter-relationships, such as fields
> >> that only appear when another field is present or has a certain value.
> >> *Visualization: the Admin UI could eventually be updated to display this
> >> information. How exactly is out of scope for this proposal.
> >> The aim is for discovery to eventually grow from "what fields are here"
> >> into a real picture of how a user’s data is structured.
> >> 
> >> Thank you,
> >> Srikanth
> >> 
> >> 
> >> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
> 
> 

This lines up with the idea that in the future, we actually expose (in a 
safe-ish manner) all the capabilities of our search environment to the agents 
so they build a "perfect" one off query.   Isntead of a narrow "we accept just 
a query and maybe some filters" and then expand everything out, we flip that 
scirpt and say "here are all the tools, build your perfect query" and we then 
provide all the validation/guard rails/workload management to support that 
approach!

I've looked at https://llmstxt.org/ and the idea of having a 
http://localhost:8983/llms.txt that told you about the cluster, what 
collections are available based on who you are and your permissions.  And then 
imagine we had https://localhost:8983/solr/techproducts/llms.txt that was 
specific to that collection?

I think there is some overlap here with what solr-mcp might provide in being a 
"safer" way of exposing primitives to Agents....

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to