Akanksha-kedia opened a new pull request, #19046: URL: https://github.com/apache/pinot/pull/19046
## Description Extends `VectorIndexHandler` to detect and rebuild a HNSW vector index when the vector dimension or distance function changes in the table config — mirroring the same pattern used for bloom filter (#18898) and JSON index (#18920) rebuilds. ### Problem Previously, if a table's vector index config changed (e.g. dimension or distance function), the existing index would be silently reused without rebuilding: - A changed **dimension** produces a structurally incompatible index that will fail or return garbage results. - A changed **distance function** silently returns wrong query results (wrong similarity metric applied). Neither failure was detectable without the stored build-time config. ### Solution Write a lightweight metadata file (`.vector.hnsw.metadata`) alongside the HNSW Lucene index directory at `seal()` time, recording: - `vectorDimension` (int) - `vectorSimilarityFunction` (Lucene `VectorSimilarityFunction` name) `VectorIndexHandler.needUpdateIndices()` reads this file and triggers a rebuild when either field differs from the current table config. **Backward compatibility:** Legacy segments without a metadata file skip the check and receive the file on their next rebuild — no forced migration. ### Files changed - `HnswVectorIndexCreator` – writes `.vector.hnsw.metadata` at seal time - `VectorIndexHandler` – reads metadata file and triggers rebuild on mismatch - `VectorIndexUtils` – helper for reading/writing the metadata file - `V1Constants` – adds `VECTOR_HNSW_METADATA_FILE_EXTENSION` constant - `VectorIndexHandlerTest` – unit tests for all rebuild/skip branches ## Test plan - [ ] New `VectorIndexHandlerTest` covers: dimension change → rebuild, distance-function change → rebuild, no change → skip, missing metadata → skip (backward compat), metadata present + matching → skip - [ ] Existing vector index integration tests pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
