msokolov commented on code in PR #15732:
URL: https://github.com/apache/lucene/pull/15732#discussion_r2911971863
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryFlatVectorsScorer.java:
##########
@@ -54,7 +55,11 @@ public Lucene102BinaryFlatVectorsScorer(FlatVectorsScorer
nonQuantizedDelegate)
public RandomVectorScorerSupplier getRandomVectorScorerSupplier(
VectorSimilarityFunction similarityFunction, KnnVectorValues
vectorValues)
throws IOException {
- throw new UnsupportedOperationException("Old codecs may only be used for
reading");
+ if (vectorValues instanceof BinarizedByteVectorValues) {
Review Comment:
this is fine, but I wonder if we could let the delegate throw the exception
in order to avoid a class cast? Or, if the delegate is null, could we instead
check for that?
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryQuantizedVectorsReader.java:
##########
@@ -448,4 +565,115 @@ BinarizedByteVectorValues getQuantizedVectorValues()
throws IOException {
return quantizedVectorValues;
}
}
+
+ // When accessing vectorValue method, targerOrd here means a row ordinal.
Review Comment:
targerOrd -> targetOrd
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryQuantizedVectorsReader.java:
##########
@@ -342,6 +358,107 @@ private FieldEntry readField(IndexInput input, FieldInfo
info) throws IOExceptio
return FieldEntry.create(input, vectorEncoding,
info.getVectorSimilarityFunction());
}
+ @Override
+ public BaseQuantizedByteVectorValues getQuantizedVectorValues(String
fieldName) {
+ return null;
+ }
+
+ @Override
+ public ScalarQuantizer getQuantizationState(String fieldName) {
+ return null;
+ }
+
+ @Override
+ public CloseableRandomVectorScorerSupplier
getRandomVectorScorerSupplierForMerge(
Review Comment:
This is a backwards codec -- why do we need to add writer methods? We
wouldn't usually be supporting writing old formats, only reading them
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene102/Lucene102BinaryQuantizedVectorsReader.java:
##########
@@ -448,4 +565,115 @@ BinarizedByteVectorValues getQuantizedVectorValues()
throws IOException {
return quantizedVectorValues;
}
}
+
+ // When accessing vectorValue method, targerOrd here means a row ordinal.
+ static class OffHeapBinarizedQueryVectorValues {
+ private final IndexInput slice;
+ private final int dimension;
+ private final int size;
+ protected final byte[] binaryValue;
+ protected final ByteBuffer byteBuffer;
+ private final int byteSize;
+ protected final float[] correctiveValues;
+ private int lastOrd = -1;
+ private int quantizedComponentSum;
+
+ OffHeapBinarizedQueryVectorValues(IndexInput data, int dimension, int
size) {
+ this.slice = data;
+ this.dimension = dimension;
+ this.size = size;
+ // 4x the quantized binary dimensions
+ int binaryDimensions = (discretize(dimension, 64) / 8) * QUERY_BITS;
Review Comment:
do these magic numbers have some meaning?
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java:
##########
@@ -348,6 +350,18 @@ public ScalarQuantizer getQuantizationState(String field) {
return fieldEntry.scalarQuantizer;
}
+ @Override
+ public CloseableRandomVectorScorerSupplier
getRandomVectorScorerSupplierForMerge(
Review Comment:
same Q as above - is this needed? can it throw unsupported exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
