[
https://issues.apache.org/jira/browse/SOLR-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141726#comment-18141726
]
David Smiley commented on SOLR-18592:
-------------------------------------
In retrospect, this is *very* duplicative with SOLR-3191; maybe I shouldn't
have filed it. Boy is the discussion there long. Nick, can you please
summarize the approaches in any patches/PRs there with the proposal here? Not
in great detail.
FWIW I completely agree with Yonik's statement that ReturnFields should have
been a single class/implementation and not pluggable (it isn't pluggable but
there was discussion about that). I would not want to go in a direction that
results in another subclass!
> Schema fl= matchesWildcard setting
> ----------------------------------
>
> Key: SOLR-18592
> URL: https://issues.apache.org/jira/browse/SOLR-18592
> Project: Solr
> Issue Type: Improvement
> Reporter: David Smiley
> Priority: Major
>
> h3. Motivation
> Some fields are known to often be large (full text bodies, raw payloads,
> etc.). When exploring an index, whether a human in a diagnostic situation or
> an LLM-driven exploratory tool, {{fl=*}} (and the default field list) returns
> those large values, and the only way to suppress one is to list every other
> field. That is a hassle, and wasteful when the response is large or fed into
> a limited context window.
> Today a stored field always matches {{*}} and glob patterns in {{fl}}. The
> only lever is {{useDocValuesAsStored=false}}, which applies only to
> docValues-only (non-stored) fields.
> h3. Proposal
> A new boolean schema property on fields and field types, tentatively
> {{matchesWildcard}}, default {{true}}. When {{false}}, the field is not
> returned when matching {{*}} (including the default field list) or a glob
> pattern in {{fl}}. It is returned only when named explicitly (including as
> the source of an alias or transformer). This is what
> {{useDocValuesAsStored=false}} already means for docValues-only fields, which
> per the ref guide can still be requested by name but do not match glob
> patterns.
> {code:xml}
> <field name="body" type="text_general" stored="true" matchesWildcard="false"/>
> {code}
> {noformat}
> fl=* -> all fields except body
> fl=*,body -> all fields including body
> {noformat}
> h3. Relationship to useDocValuesAsStored
> Keep {{useDocValuesAsStored}}; the new property is separate for backwards
> compatibility. Effective rule: if {{matchesWildcard}} is set, it wins.
> Otherwise a stored field matches (as today) and a docValues-only field takes
> the value of {{useDocValuesAsStored}}. For docValues-only fields the two are
> therefore equivalent, and a schema may use either.
> The old setting must *not* be read as the new one for stored fields. Many
> stored fields have {{useDocValuesAsStored=false}}, explicitly or by default
> (e.g. SortableTextField, numeric range fields), and for stored fields it does
> not affect whether they match {{*}}.
> h3. Open questions
> * Name: {{matchesWildcard}} vs. e.g. {{returnedByWildcard}}.
> * Should an explicit glob such as {{fl=big_*}} match a field with the setting
> off? Proposed: no, as with {{useDocValuesAsStored=false}} today.
> * Other consumers of the field list (highlighting {{hl.fl=*}}, real-time get,
> CSV output, distributed shard requests) should be checked for consistency.
> h3. Related
> * SOLR-3191: field exclusion from fl
> * SOLR-9467: Document Transformer to Remove Fields (query time; complementary
> to a schema setting)
> * SOLR-11364: Fields with useDocValuesAsStored=false never be returned in
> case of pattern matching
> * https://github.com/apache/solr/pull/5188 centralizes the decision of which
> index fields {{fl}} needs in {{ReturnFields}}, which this would build on.
> 🤖 This issue description was AI-generated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
