iprithv commented on PR #15982: URL: https://github.com/apache/lucene/pull/15982#issuecomment-4392297181
> Would it be better if instead of (normal + quantization overhead) we see it like keeping things specific to FieldWriter in its `#ramBytesUsed` otherwise all on `Lucene[XYZ]ScalarQuantizedVectorsWriter#ramBytesUsed`? Very likely I might be confused with this accounting code part(lemme know if thats the case) but it would be good to keep accounting local basically. Right now we don't use FieldWriter for quantized fields as you mentioned but we do call quantizationOverheadBytesUsed in both places. Could we simplify keeping the accounting local? @shubhamvishu I think the current design is the right one, and there is no correct alternative for this specific structure. If we made FieldWriter.ramBytesUsed() return only quantizationOverheadBytesUsed() and called field.ramBytesUsed() at the writer level instead of quantizationOverheadBytesUsed(): 1. Byte vector fields still need separate handling, they have no FieldWriter at all, so rawVectorDelegate.ramBytesUsed() would still be required 2. Breaks the Accountable contract, FieldWriter.ramBytesUsed() would return an incomplete number when queried in isolation (e.g., in tests or via JMX introspection) 3. Gains nothing, we would still need two calls at the writer level, rawVectorDelegate for byte fields + field.ramBytesUsed() for float32 overhead I think the quantizationOverheadBytesUsed() method is the right abstraction precisely because the delegate already "owns" the flat data layer and the writer should only know about the extra quant state on top. Everything is accounted for exactly once. Thanks @shubhamvishu! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
