ethbak commented on code in PR #16938:
URL: https://github.com/apache/lucene/pull/16938#discussion_r4485960125
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsFormat.java:
##########
@@ -24,19 +24,19 @@
import org.apache.lucene.index.SegmentWriteState;
/**
- * Flat vector format that stores each distinct vector once.
+ * Flat vector format that stores each distinct vector of a document once.
*
* <p>Vectors that share the same dimension and encoding form a <i>group</i>.
Within a group, an
- * identical vector is stored a single time regardless of how many documents
(across all fields that
- * map to that group) reference it; each field then keeps a {@code
fieldOrdToGroupOrd} map from its
- * document ordinal to the group ordinal of the shared vector. This is well
suited to indexes with
- * repeated vectors, e.g. several fields derived from the same embedding, or
heavily duplicated
- * content.
+ * identical vector is stored a single time per document regardless of how
many of its fields (among
+ * all fields that map to that group) reference it; each field then keeps a
{@code
+ * fieldOrdToGroupOrd} map from its document ordinal to the group ordinal of
the shared vector. This
+ * is well suited to indexes with several fields derived from the same
embedding. Identical vectors
+ * of different documents are stored once each, which keeps the group's
vectors in doc order.
*
* <h2>.vdd (vector de-dup data) file</h2>
*
* <ul>
- * <li>For each group, its distinct vectors, aligned to 4 bytes (BYTE) or 64
bytes (FLOAT32 and
+ * <li>For each group, its distinct vectors in doc order, aligned to 4 bytes
(BYTE) or 64 bytes
Review Comment:
Looks like "(FLOAT 32 and" got cut off here
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsReader.java:
##########
@@ -337,21 +366,38 @@ public Float16VectorValues getFloat16VectorValues(String
field) throws IOExcepti
}
@Override
- public FlatVectorsReader getMergeInstance() {
- // TODO: Can we improve performance using sequential IO + avoiding
read-backs? One way is to
- // de-duplicate only within a document, allowing for cleaner sort of
vectors during flush +
- // avoid read-backs for full equality checks during merge.
- return this;
+ public FlatVectorsReader getMergeInstance() throws IOException {
+ // vectors are stored in doc order and merged front to back, so read them
sequentially
+ IndexInput data = original.mergeVectorData();
+ return data == vectorData ? this : new DedupFlatVectorsReader(this,
data.clone());
}
- @Override
- public void finishMerge() {
- // TODO: Converse of getMergeInstance()
+ /** Returns the input for merges, opening the data file again if needed. */
+ private synchronized IndexInput mergeVectorData() throws IOException {
+ assert original == this;
+ if (closed) {
Review Comment:
Should this use `ensureOpen()` for consistency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
