[
https://issues.apache.org/jira/browse/SOLR-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142090#comment-18142090
]
David Smiley commented on SOLR-18592:
-------------------------------------
I don't think we need multiple solutions (each has
maintenance/documentation/...)
My motivation for the particular approach in this JIRA stems from my
observation on how {{useDocValuesAsStored}} works. I'll copy-paste
[comment|https://issues.apache.org/jira/browse/SOLR-9467?focusedId=15888964&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15888964]
my comment from a related ticket:
{quote}If useDocValuesAsStored was generalized to a hypothetical
excludeFromWildcardRetrieval (damn it's hard to come up with good names for
this!) then it could not only work as it does now but also for stored fields.
{quote}
Basically, we have something _already_ that's *almost* behaving as needed to be
useful for this use-case (casually exploring data out of Solr but not returning
the large fields). useDocValuesAsStored is really implemented as a wildcard
exclusion mechanism; it doesn't really intrinsically have to do with docValues
in terms of how it's implemented. Yes it only applies to docValues fields but
imagine removing/simplifying that criteria and giving this a more general name
like matchesWildcard.
That said, there is some nuance to useDocValuesAsStored when a field is both
stored and has docValues, since in such a case, whether or not the docValues
data can be a substitute for the stored data is configurable with this
mechanism. Stored multi-value retains an order but not docValues. I wish that
was handled differently.
> Schema fl= matchesWildcard setting
> ----------------------------------
>
> Key: SOLR-18592
> URL: https://issues.apache.org/jira/browse/SOLR-18592
> Project: Solr
> Issue Type: Improvement
> Reporter: David Smiley
> Priority: Major
>
> h3. Motivation
> Some fields are known to often be large (full text bodies, raw payloads,
> etc.). When exploring an index, whether a human in a diagnostic situation or
> an LLM-driven exploratory tool, {{fl=*}} (and the default field list) returns
> those large values, and the only way to suppress one is to list every other
> field. That is a hassle, and wasteful when the response is large or fed into
> a limited context window.
> Today a stored field always matches {{*}} and glob patterns in {{fl}}. The
> only lever is {{useDocValuesAsStored=false}}, which applies only to
> docValues-only (non-stored) fields.
> h3. Proposal
> A new boolean schema property on fields and field types, tentatively
> {{matchesWildcard}}, default {{true}}. When {{false}}, the field is not
> returned when matching {{*}} (including the default field list) or a glob
> pattern in {{fl}}. It is returned only when named explicitly (including as
> the source of an alias or transformer). This is what
> {{useDocValuesAsStored=false}} already means for docValues-only fields, which
> per the ref guide can still be requested by name but do not match glob
> patterns.
> {code:xml}
> <field name="body" type="text_general" stored="true" matchesWildcard="false"/>
> {code}
> {noformat}
> fl=* -> all fields except body
> fl=*,body -> all fields including body
> {noformat}
> h3. Relationship to useDocValuesAsStored
> Keep {{useDocValuesAsStored}}; the new property is separate for backwards
> compatibility. Effective rule: if {{matchesWildcard}} is set, it wins.
> Otherwise a stored field matches (as today) and a docValues-only field takes
> the value of {{useDocValuesAsStored}}. For docValues-only fields the two are
> therefore equivalent, and a schema may use either.
> The old setting must *not* be read as the new one for stored fields. Many
> stored fields have {{useDocValuesAsStored=false}}, explicitly or by default
> (e.g. SortableTextField, numeric range fields), and for stored fields it does
> not affect whether they match {{*}}.
> h3. Open questions
> * Name: {{matchesWildcard}} vs. e.g. {{returnedByWildcard}}.
> * Should an explicit glob such as {{fl=big_*}} match a field with the setting
> off? Proposed: no, as with {{useDocValuesAsStored=false}} today.
> * Other consumers of the field list (highlighting {{hl.fl=*}}, real-time get,
> CSV output, distributed shard requests) should be checked for consistency.
> h3. Related
> * SOLR-3191: field exclusion from fl
> * SOLR-9467: Document Transformer to Remove Fields (query time; complementary
> to a schema setting)
> * SOLR-11364: Fields with useDocValuesAsStored=false never be returned in
> case of pattern matching
> * https://github.com/apache/solr/pull/5188 centralizes the decision of which
> index fields {{fl}} needs in {{ReturnFields}}, which this would build on.
> 🤖 This issue description was AI-generated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
