chrevanthreddy opened a new pull request, #19318: URL: https://github.com/apache/hudi/pull/19318
### Describe the issue this Pull Request addresses Resolves #19098 (umbrella: #19094, RFC-109) Third slice of RFC-109 (Hudi Native Vector Index): the RaBitQ encoder and metric-aware scorer contract with multibit-aware support. > **Stacked PR.** This branch is stacked on #19310 (options, #19096) and #19317 (schema/payload, #19097). Until those merge, the diff here shows their commits too; the only new commit in this PR is `feat(index): add RaBitQ encoder and multibit-aware scorer contract`. Review/merge order: #19310 → #19317 → this. ### Summary and Changelog - **`RaBitQEncoder`**: 1-bit and multibit (extended-plane) encoding contract, residual and non-residual encoding, L2/IP query-state preparation, and packed-binary/multibit dot-term scoring primitives. Encoding-version-aware so readers dispatch on the metadata written per generation. - **`VectorQuantizer`**: quantized-vector container (`QuantizedVector`) and quantizer contract shared by encoder and posting builders. - **`MetricQueryState`**: metric-aware query rotation/normalization state; enforces metric compatibility explicitly. - **`RaBitQPlaneKernel` / `VectorQueryPlanes`**: bit-plane scoring kernel (`Long.bitCount`-based inner products) and query-plane representation. - **`RaBitQByteLutScorer`**: byte-LUT scorer for multibit codes. - **`RaBitQNeutralFactors` / `RaBitQFactorConfig`**: neutral/rescale factor computation and configuration. - **Tests (59, all passing)**: `TestRaBitQEncoder` (20), `MetricEstimatorIdentityTest` (14), `TestVectorDistanceMetric` (9), `TestRaBitQNeutralFactors` (6), `TestMetricQueryStateRotation` (5), `TestRaBitQPlaneKernel` (3), `TestRaBitQResidualHypothesis` (1), `TestRaBitQResidualRecall` (1). ### Impact Additive: new engine-neutral encoder/scorer classes under `hudi-common`. No wiring into existing write/read paths in this PR (that happens in the bootstrap #19099 and read-path #19100 slices). No behavioral change to existing tables. ### Risk Level low Pure additive numeric/encoding logic with dedicated unit coverage; nothing references these classes from existing runtime paths yet. ### Documentation Update Encoder/scorer contracts and the multibit decomposition are documented as class/method Javadoc. User-facing RFC document tracked in #19309. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact described above - [x] Necessary doc changes done or have another open PR - [x] For large changes, please consider breaking it into sub-tasks under an umbrella issue (#19094) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
