[
https://issues.apache.org/jira/browse/SOLR-18674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146900#comment-18146900
]
Andrzej Bialecki commented on SOLR-18674:
-----------------------------------------
[~isaric] thanks for the comments, I'm happy to split the work :)
* good points on the current parsing.
* I prefer theĀ {{<metadata>}} child element, too. XML attributes are more
constraining than elements, besides a long list of long attributes quickly
becomes unreadable for humans.
* not sure what other use cases Elastic field meta is for, other than units
and tags. The motivation for this ticket is to provide rich semantic context to
users (humans or agents) that is otherwise impossible to get from the data, or
easy to mistake for something that it's not. Unit, metric_type and other labels
would fit in the "generic key-value map" part of this proposal. However, I'd
like to propose using some already existing standard for common metadata, such
as description, instead of inventing completely new ones. This probably doesn't
change the implementation, unless we want to go with a base vocabulary (plus
custom extensions), or require some labels, e.g. "dc:description" if <metadata>
is present.
> Extend Solr schema with field-level metadata
> --------------------------------------------
>
> Key: SOLR-18674
> URL: https://issues.apache.org/jira/browse/SOLR-18674
> Project: Solr
> Issue Type: Improvement
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
>
> h2. Problem
> Schema fields carry only indexing and storage properties. There is no
> supported way to record what a field is {*}for{*}, such as a human
> description, its purpose, owner, or semantic role. Teams keep this in wikis
> or in {{_comment}} XML comments, which are lost on Schema API round-trips.
> Admin UI users, tooling, and LLM/agent clients that inspect a schema get no
> semantic context.
> Especially for LLM/agent clients this additional information can reduce the
> effort (and error rate) of data discovery.
> h2. Current behavior
> * {{SchemaField}} keeps unrecognized attributes in its {{args}} map
> ({{{}SchemaField.java:49{}}}).
> * {{getNamedPropertyValues(false)}} coerces every value to a boolean
> ({{{}SchemaField.java:472-478{}}}). An arbitrary string attribute such as
> {{description="..."}} is returned by the Schema API as {{{}false{}}}.
> * Nothing is documented or validated, and {{{}copyField{}}}, dynamic fields
> and field types have no equivalent.
> h2. Proposal
> Add an optional, non-functional {{metadata}} concept to fields, dynamic
> fields, and field types:
> * *First-class {{{}description{}}}:* a free-text string attribute. Also
> support {{{}purpose{}}}, a short free-text string.
> * *Generic {{metadata}} map:* string keys to string values, for user-defined
> entries such as {{{}owner{}}}, {{{}pii=true{}}}, or {{{}source-system{}}}.
> Keys are namespaced by convention, e.g. {{{}team.owner{}}}.
> * *Schema API:* {{{}add-field{}}}, {{replace-field}} and the other field
> commands accept these attributes. {{GET /schema/fields}} returns them as
> strings. A new {{includeMetadata}} parameter, default true, controls this.
> * *Managed schema XML:* serialized as attributes, or as a child
> {{<metadata>}} element for the map. Classic {{schema.xml}} parsing accepts
> both forms.
> * *Semantics:* metadata never affects indexing, query behavior, or the
> schema's compatibility checks. Changing only metadata should not require a
> reindex, and replace-field should say so in its response.
> * *Constraints:* maximum key and value lengths, plus a total size cap, to
> keep ZooKeeper nodes small.
> h2. Design notes
> # Fix the boolean coercion first. Store metadata in a dedicated field on
> {{SchemaField}} rather than reusing {{{}args{}}}.
> # Reserve the {{metadata}} and {{description}} names so they are not treated
> as field properties.
> # SolrJ gets typed accessors in {{SchemaRepresentation}} and the schema
> request classes. Admin UI shows the description as a tooltip and column.
> # Ref guide: document the attributes in the field and Schema API pages. Add
> a changelog entry once a JIRA or PR exists.
> h2. Compatibility and risks
> * *Backward compatible:* old schemas load unchanged. Older nodes that see
> the new attributes already ignore them, but the boolean-coercion bug may
> break their Schema API output. That matters for rolling upgrades and needs a
> test.
> * *Size growth in ZooKeeper:* mitigated by the length caps above.
> * *Overlap:* other attributes might be better handled separately, such as
> {{useDocValuesAsStored}} and similar functional flags. Keep this proposal
> strictly descriptive.
> h2. Open questions
> * Should metadata also be allowed on {{copyField}} and on the schema as a
> whole?
> * Fixed set of keys, such as {{purpose}} as an enum, or a free-form map only?
> * Common set of recommended keys that follow some existing recommendation?
> Examples:
> ** [https://mlcommons.org/working-groups/data/croissant/]
> **
> [https://blog.open-metadata.org/announcing-openmetadata-2-0-the-open-context-layer-for-ai-agents-83b8ce8b9dde]
> ** [https://docs.databricks.com/aws/en/uc-semantics/agent-metadata]
> * Should {{description}} support localization, or just one language?
> h2. Plan
> # Fix coercion, add the {{SchemaField}} metadata storage and XML round-trip,
> with tests.
> # Schema API read and write support, plus SolrJ.
> # Dynamic fields, field types, and Admin UI.
> # Docs and changelog.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]