David Smiley created SOLR-18592:
-----------------------------------
Summary: Schema fl= matchesWildcard setting
Key: SOLR-18592
URL: https://issues.apache.org/jira/browse/SOLR-18592
Project: Solr
Issue Type: Improvement
Reporter: David Smiley
h3. Motivation
Some fields are known to often be large (full text bodies, raw payloads, etc.).
When exploring an index, whether a human in a diagnostic situation or an
LLM-driven exploratory tool, {{fl=*}} (and the default field list) returns
those large values, and the only way to suppress one is to list every other
field. That is a hassle, and wasteful when the response is large or fed into a
limited context window.
Today a stored field always matches {{*}} and glob patterns in {{fl}}. The only
lever is {{useDocValuesAsStored=false}}, which applies only to docValues-only
(non-stored) fields.
h3. Proposal
A new boolean schema property on fields and field types, tentatively
{{matchesWildcard}}, default {{true}}. When {{false}}, the field is not
returned when matching {{*}} (including the default field list) or a glob
pattern in {{fl}}. It is returned only when named explicitly (including as the
source of an alias or transformer). This is what {{useDocValuesAsStored=false}}
already means for docValues-only fields, which per the ref guide can still be
requested by name but do not match glob patterns.
{code:xml}
<field name="body" type="text_general" stored="true" matchesWildcard="false"/>
{code}
{noformat}
fl=* -> all fields except body
fl=*,body -> all fields including body
{noformat}
h3. Relationship to useDocValuesAsStored
Keep {{useDocValuesAsStored}}; the new property is separate for backwards
compatibility. Effective rule: if {{matchesWildcard}} is set, it wins.
Otherwise a stored field matches (as today) and a docValues-only field takes
the value of {{useDocValuesAsStored}}. For docValues-only fields the two are
therefore equivalent, and a schema may use either.
The old setting must *not* be read as the new one for stored fields. Many
stored fields have {{useDocValuesAsStored=false}}, explicitly or by default
(e.g. SortableTextField, numeric range fields), and for stored fields it does
not affect whether they match {{*}}.
h3. Open questions
* Name: {{matchesWildcard}} vs. e.g. {{returnedByWildcard}}.
* Should an explicit glob such as {{fl=big_*}} match a field with the setting
off? Proposed: no, as with {{useDocValuesAsStored=false}} today.
* Other consumers of the field list (highlighting {{hl.fl=*}}, real-time get,
CSV output, distributed shard requests) should be checked for consistency.
h3. Related
* SOLR-3191: field exclusion from fl
* SOLR-9467: Document Transformer to Remove Fields (query time; complementary
to a schema setting)
* SOLR-11364: Fields with useDocValuesAsStored=false never be returned in case
of pattern matching
* https://github.com/apache/solr/pull/5188 centralizes the decision of which
index fields {{fl}} needs in {{ReturnFields}}, which this would build on.
🤖 This issue description was AI-generated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
