kaivalnp commented on code in PR #15979: URL: https://github.com/apache/lucene/pull/15979#discussion_r3691313762
########## lucene/core/src/java/org/apache/lucene/codecs/dedup/DedupFlatVectorsReader.java: ########## @@ -0,0 +1,421 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.lucene.codecs.dedup; + +import java.io.IOException; +import java.util.Map; +import java.util.Objects; +import java.util.stream.Stream; +import org.apache.lucene.codecs.CodecUtil; +import org.apache.lucene.codecs.hnsw.FlatVectorsReader; +import org.apache.lucene.codecs.hnsw.FlatVectorsScorer; +import org.apache.lucene.codecs.lucene95.OrdToDocDISIReaderConfiguration; +import org.apache.lucene.index.ByteVectorValues; +import org.apache.lucene.index.CorruptIndexException; +import org.apache.lucene.index.FieldInfo; +import org.apache.lucene.index.FieldInfos; +import org.apache.lucene.index.FloatVectorValues; +import org.apache.lucene.index.IndexFileNames; +import org.apache.lucene.index.KnnVectorValues; +import org.apache.lucene.index.SegmentReadState; +import org.apache.lucene.index.VectorEncoding; +import org.apache.lucene.index.VectorSimilarityFunction; +import org.apache.lucene.internal.hppc.IntObjectHashMap; +import org.apache.lucene.store.ChecksumIndexInput; +import org.apache.lucene.store.DataAccessHint; +import org.apache.lucene.store.FileDataHint; +import org.apache.lucene.store.FileTypeHint; +import org.apache.lucene.store.IOContext; +import org.apache.lucene.store.IOContext.FileOpenHint; +import org.apache.lucene.store.IndexInput; +import org.apache.lucene.store.RandomAccessInput; +import org.apache.lucene.util.IOUtils; +import org.apache.lucene.util.LongValues; +import org.apache.lucene.util.RamUsageEstimator; +import org.apache.lucene.util.hnsw.RandomVectorScorer; +import org.apache.lucene.util.packed.DirectReader; + +/** + * Reads a {@link DedupFlatVectorsFormat} segment. + * + * <p>Layout: {@code .dvc} holds pool vector bytes (contiguous per pool) followed by per-field DISI + * + packed {@code docOrd → vecOrd} maps. {@code .dvm} holds pool/field metadata. + * + * @lucene.experimental + */ +public final class DedupFlatVectorsReader extends FlatVectorsReader { + + private static final long SHALLOW_SIZE = + RamUsageEstimator.shallowSizeOfInstance(DedupFlatVectorsReader.class); + + private final FlatVectorsScorer vectorScorer; + private final FlatVectorsScorer translatingScorer; + private final FieldInfos fieldInfos; + private final IndexInput vectorData; + private final IOContext dataContext; + + private final IntObjectHashMap<FieldEntry> fields = new IntObjectHashMap<>(); + + public DedupFlatVectorsReader(SegmentReadState state, FlatVectorsScorer scorer) + throws IOException { + this(state, scorer, DataAccessHint.RANDOM); Review Comment: Ah right, monotonic docid traversal happens in the exact search path -- although I don't think we explicitly advise the kernel with a sequential hint on the vectors (because the same file may be concurrently used for random-access HNSW traversal from other searches). The benefit in the original format is that vectors matching the filter + present close to each other on disk _may_ be retrieved from disk in the same load? (but this would be highly dependent on the dimension of vectors, filter, and index sort) I guess this implicit benefit can be ported to the de-duplicating format if we lay out vectors on disk in the same index sort order? Although this has its own caveats around choosing which doc to "primarily" attribute the vector to, for sorting purposes. Perhaps radical change: if we only de-dupe a vector _within_ a document, it simplifies some things for us.. - Since each vector is associated to exactly one document, the raw file can be deterministically sorted by index sort. - Merging of already de-duped segments becomes much cheaper, because there is no read-back of vectors from disk on hash collisions _across_ documents (for the full equality check). - The added cost is the per-document tracking at flush time + lower overall de-duplication factor. I think this can be a reasonable option, but needs some work to implement. Should we add a TODO to evaluate de-duping only within a document? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
