J-HowHuang commented on code in PR #19046:
URL: https://github.com/apache/pinot/pull/19046#discussion_r3669366725
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/VectorIndexUtils.java:
##########
@@ -43,9 +46,52 @@
public class VectorIndexUtils {
+ private static final String METADATA_KEY_DIMENSION = "dimension";
+ private static final String METADATA_KEY_DISTANCE_FUNCTION =
"distanceFunction";
+
private VectorIndexUtils() {
}
+ /**
+ * Writes a lightweight metadata file next to the HNSW index directory,
recording the vector dimension and
+ * similarity function. This file is used by
+ * {@link
org.apache.pinot.segment.local.segment.index.loader.invertedindex.VectorIndexHandler}
+ * to detect config changes and trigger an index rebuild when necessary.
+ *
+ * <p>Legacy segments built before this method existed have no metadata
file; the handler skips config-change
+ * detection for those segments (treating the absence of the file as
"unknown, assume unchanged").
+ */
+ public static void writeVectorIndexMetadata(File segmentIndexDir, String
column, int dimension,
Review Comment:
Same here. This helper class isn't for HNSW implementation only, so this
method shouldn't only serve HNSW
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/VectorIndexHandler.java:
##########
@@ -545,6 +551,58 @@ && hasCombinedFile(indexDir, column, desiredBackend)) {
}
}
+ /**
+ * Checks whether the HNSW vector index config has changed by comparing the
metadata file written at index-creation
Review Comment:
This only applies to HNSW vector index, however we share the same
`VectorIndexHandler` for all vector index implementations.
Please consider making this extendable to all other implementations, it
shouldn't only serve one specific HNSW implementation.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/VectorIndexUtils.java:
##########
@@ -43,9 +46,52 @@
public class VectorIndexUtils {
+ private static final String METADATA_KEY_DIMENSION = "dimension";
+ private static final String METADATA_KEY_DISTANCE_FUNCTION =
"distanceFunction";
+
private VectorIndexUtils() {
}
+ /**
+ * Writes a lightweight metadata file next to the HNSW index directory,
recording the vector dimension and
+ * similarity function. This file is used by
+ * {@link
org.apache.pinot.segment.local.segment.index.loader.invertedindex.VectorIndexHandler}
+ * to detect config changes and trigger an index rebuild when necessary.
+ *
+ * <p>Legacy segments built before this method existed have no metadata
file; the handler skips config-change
+ * detection for those segments (treating the absence of the file as
"unknown, assume unchanged").
+ */
+ public static void writeVectorIndexMetadata(File segmentIndexDir, String
column, int dimension,
+ VectorSimilarityFunction similarityFunction)
+ throws IOException {
+ File metadataFile = new File(segmentIndexDir, column +
Indexes.VECTOR_HNSW_INDEX_METADATA_FILE_EXTENSION);
+ Properties props = new Properties();
+ props.setProperty(METADATA_KEY_DIMENSION, String.valueOf(dimension));
+ props.setProperty(METADATA_KEY_DISTANCE_FUNCTION,
similarityFunction.name());
+ try (FileOutputStream out = new FileOutputStream(metadataFile)) {
+ props.store(out, null);
+ }
+ }
+
+ /**
+ * Reads the HNSW vector index metadata file for the given column. Returns
{@code null} if the file does not
+ * exist (legacy segment) or cannot be parsed.
+ */
+ @Nullable
+ public static Properties readVectorIndexMetadata(File segmentIndexDir,
String column) {
Review Comment:
Same here. This helper class isn't for HNSW implementation only, so this
method shouldn't only serve HNSW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
