[
https://issues.apache.org/jira/browse/SOLR-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141737#comment-18141737
]
Nick Shanin commented on SOLR-18592:
------------------------------------
[~dsmiley] I checked the SOLR-3191 patches and [PR
271|https://github.com/apache/lucene-solr/pull/271]. They propose request-time
exclusions such as {{fl=*,-body}} and exclusion globs. The PR changes the
existing {{SolrReturnFields}} implementation.
SOLR-18592 instead proposes a schema default: a field marked
{{matchesWildcard=false}} is omitted from the default field list and globs, but
can still be requested by name. It would use the existing return-fields
implementation.
I see three directions:
# *Request-time exclusion only:* pursue SOLR-3191 and close this ticket as a
duplicate. This lets each client choose exclusions, but clients using the
default field list would need to change their requests.
# *Schema default first:* pursue SOLR-18592 as the narrower change. Decide
whether SOLR-3191 has enough support to continue as a separately scoped
request-time feature; if so, identify its next step and owner, or otherwise
close it.
# *Design both together:* agree first on the schema default, request-time
exclusion syntax, and their precedence, then split the implementation into
reviewable changes.
I recommend the second: it addresses large fields for clients that use the
default field list, without adding exclusion syntax to {{fl}}. The expected
changes are in schema properties, return-field selection and fetching, the
JavaBin writer, tests, and the reference guide.
h4. Background
* SOLR-3191: six patch revisions and PR 271 explored request-time exclusions,
including globs, aliases and transformers.
* SOLR-9467: two patches proposed a document transformer to remove a field from
the response. You later suggested a schema-level wildcard exclusion for stored
fields, close to the proposal here.
* SOLR-11364 / PR 5188 fixed explicit doc-values fields and transformer inputs
alongside globs. It is related infrastructure.
Does this comparison give enough context to choose a direction?
🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
> Schema fl= matchesWildcard setting
> ----------------------------------
>
> Key: SOLR-18592
> URL: https://issues.apache.org/jira/browse/SOLR-18592
> Project: Solr
> Issue Type: Improvement
> Reporter: David Smiley
> Priority: Major
>
> h3. Motivation
> Some fields are known to often be large (full text bodies, raw payloads,
> etc.). When exploring an index, whether a human in a diagnostic situation or
> an LLM-driven exploratory tool, {{fl=*}} (and the default field list) returns
> those large values, and the only way to suppress one is to list every other
> field. That is a hassle, and wasteful when the response is large or fed into
> a limited context window.
> Today a stored field always matches {{*}} and glob patterns in {{fl}}. The
> only lever is {{useDocValuesAsStored=false}}, which applies only to
> docValues-only (non-stored) fields.
> h3. Proposal
> A new boolean schema property on fields and field types, tentatively
> {{matchesWildcard}}, default {{true}}. When {{false}}, the field is not
> returned when matching {{*}} (including the default field list) or a glob
> pattern in {{fl}}. It is returned only when named explicitly (including as
> the source of an alias or transformer). This is what
> {{useDocValuesAsStored=false}} already means for docValues-only fields, which
> per the ref guide can still be requested by name but do not match glob
> patterns.
> {code:xml}
> <field name="body" type="text_general" stored="true" matchesWildcard="false"/>
> {code}
> {noformat}
> fl=* -> all fields except body
> fl=*,body -> all fields including body
> {noformat}
> h3. Relationship to useDocValuesAsStored
> Keep {{useDocValuesAsStored}}; the new property is separate for backwards
> compatibility. Effective rule: if {{matchesWildcard}} is set, it wins.
> Otherwise a stored field matches (as today) and a docValues-only field takes
> the value of {{useDocValuesAsStored}}. For docValues-only fields the two are
> therefore equivalent, and a schema may use either.
> The old setting must *not* be read as the new one for stored fields. Many
> stored fields have {{useDocValuesAsStored=false}}, explicitly or by default
> (e.g. SortableTextField, numeric range fields), and for stored fields it does
> not affect whether they match {{*}}.
> h3. Open questions
> * Name: {{matchesWildcard}} vs. e.g. {{returnedByWildcard}}.
> * Should an explicit glob such as {{fl=big_*}} match a field with the setting
> off? Proposed: no, as with {{useDocValuesAsStored=false}} today.
> * Other consumers of the field list (highlighting {{hl.fl=*}}, real-time get,
> CSV output, distributed shard requests) should be checked for consistency.
> h3. Related
> * SOLR-3191: field exclusion from fl
> * SOLR-9467: Document Transformer to Remove Fields (query time; complementary
> to a schema setting)
> * SOLR-11364: Fields with useDocValuesAsStored=false never be returned in
> case of pattern matching
> * https://github.com/apache/solr/pull/5188 centralizes the decision of which
> index fields {{fl}} needs in {{ReturnFields}}, which this would build on.
> 🤖 This issue description was AI-generated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
