Pranshu-S commented on issue #16029:
URL: https://github.com/apache/lucene/issues/16029#issuecomment-6918788909

   Hey @mccullocht @kaivalnp, I was trying to see what would be the cleanest 
way to unblock data-blind quantization, as it was reverted earlier due to how 
it behaves when merges read segments through reader wrappers, which results in 
behaviour discrepancies. This also sort of raises a wider question on how the 
updated codec will behave with wrappers and merges, as well as backward 
compatibility. 
   
   I had my two cents on it and wanted a wider opinion before I commit to doing 
the change.
   
   **The problem**
   
   In the original PR, `Lucene104ScalarQuantizedVectorsWriter` decided how to 
merge each input segment by checking whether it still has its original floats. 
However, that check only works when the input unwraps to a 
`Lucene104ScalarQuantizedVectorsReader`. Otherwise it treats the input as 
another format and checks `getFloatVectorValues().size() > 0`. A data-blind 
segment answers that with floats reconstructed from its quantized bytes, so the 
writer concludes the original floats exist.
   
   Merges wrap inputs in several cases:
   
   - `SlowCodecReaderWrapper`, from merge policies that wrap inputs
   - `SortingCodecReader` over `SlowCompositeCodecReaderWrapper`, for 
reordering merges
   - `DocRangeKnnVectorsReader`, for partitioned merges
   
   When that happens: a segment without floats can be merged into a `CENTERED` 
or `DATA_BLIND_WITH_FLOATS` output, which then stores reconstructions as if 
they were the original vectors
   
   If we look closely, the problem is not the code but the fact that Lucene 
works with the assumption that every format's merge fallback is lossless: when 
a merge doesn't recognize an input, it just reads the values back through the 
standard API and writes them again. So failing to recognize an input only makes 
the merge slower in the fallback cases.
   
   `DATA_BLIND_WITHOUT_FLOATS` is the first mode where this is NOT true by 
design, since we don't keep the original vectors at all.
   
   One way would be to update the wrappers to expose what they wrap. But 
wrappers are also something users can write themselves, and existing ones 
wouldn't forward a new method, so the problem would quietly remain for them.
   
   **Proposal**
   
   If we honestly think about it, since the quantized vector is standalone, 
without any backing float vector, it can be considered a derived encoding of 
its own. Doing it this way also makes sense because every codec and wrapper 
then has to handle it explicitly, instead of being handed floats that look like 
the originals. Since data-blind quantizes against a zero centroid, the 
quantized record doesn't depend on the segment, so it can be the field's stored 
value.
   
    A new `VectorEncoding` (e.g. `QUANTIZED`) would store the records as the 
field's real values, and floats would only be input at index time. `CENTERED` 
and `DATA_BLIND_WITH_FLOATS` would stay codec modes of float fields, since they 
keep the original floats and don't have this problem.
   
   Pros:
   
   - Merges are lossless by design. Wrappers and generic merges move the 
records the same way they already move `BYTE`, `FLOAT32` and `FLOAT16`, so 
reordering and partitioned merges just work.
   - The one-way-door rules (no re-encoding, no re-centering, the same encoding 
across segments) become schema rules, checked at indexing time instead of 
failing during merge.
   - Reconstructions are never passed off as originals, so converting to other 
formats can't silently degrade the vectors.
   - The compiler enforces it: every `switch` over `VectorEncoding` has to 
handle the new value, so no code path can quietly treat these vectors as plain 
floats.
   
   Cons:
   
   - A big core change, similar in size to adding `FLOAT16`
   - Quantization becomes a schema choice instead of a codec setting, so moving 
an existing field means a new field or a reindex (which is also the right 
expectation in my opinion).
   
   Would love to hear what others think, especially whether a new encoding is 
acceptable here, or if we'd rather look at something lighter for now (e.g. a 
per-segment flag in `FieldInfo` attributes that merges check).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to