JingsongLi commented on code in PR #8239:
URL: https://github.com/apache/paimon/pull/8239#discussion_r3417434863
##########
paimon-vector/paimon-vector-index/src/main/java/org/apache/paimon/vector/index/VectorGlobalIndexerFactory.java:
##########
@@ -37,14 +37,20 @@ public abstract class VectorGlobalIndexerFactory implements
GlobalIndexerFactory
public GlobalIndexer create(DataField field, Options options) {
String identifier = identifier();
return new VectorGlobalIndexer(
- field.type(), nativeOptions(field.type(), options,
identifier), identifier);
+ field.type(),
+ nativeOptions(field.type(), options, identifier, field.name()),
+ identifier);
}
static Map<String, String> nativeOptions(
- DataType fieldType, Options tableOptions, String identifier) {
+ DataType fieldType, Options tableOptions, String identifier,
String fieldName) {
Map<String, String> nativeOptions = new LinkedHashMap<>();
String optionPrefix = identifier + ".";
- for (Map.Entry<String, String> entry :
tableOptions.toMap().entrySet()) {
+ String fieldPrefix = "fields." + fieldName + ".";
Review Comment:
Could we keep this override scoped by the current index type? With the
unprefixed `fields.<field>` key, a field-level option for one vector index type
overrides every vector index built on the same column. For example,
`ivf-flat.nlist=128, ivf-pq.nlist=512, fields.vec.nlist=256` makes
`nativeOptions(..., ivf-pq, "vec")` return `nlist=256`, so the ivf-pq build
silently loses its own setting. Global index metadata already distinguishes
both `indexFieldId` and `indexType`, so the same field can carry multiple index
types. I think we should either keep the key under
`<index-type>.fields.<field-name>.<option>` or otherwise apply the field
override only when it is scoped to the current `identifier`, and add this as a
regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
