>From Ali Alsuliman <[email protected]>:
Ali Alsuliman has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21686?usp=email )
Change subject: [ASTERIXDB-3817][STO] Share the scalar quantization formula
......................................................................
[ASTERIXDB-3817][STO] Share the scalar quantization formula
Two paths encode vectors for the same index. Bulk load and
static-structure build go through OptimizedScalarQuantizationCodec in
asterix-common; DML insert goes through VTreeDataTupleBuilder in
hyracks-storage-am-vtree, which cannot depend on asterix-common and so
re-implemented the arithmetic rather than calling it. Nothing compared
the two.
Drift between them would be close to invisible. Both paths keep
producing well-formed codes, the index stays structurally valid, and
every functional test still passes; what changes is that a DML-inserted
row's codes are no longer on the same scale as a bulk-loaded row's, so
approximate distances against them carry a bias that surfaces only as
recall falling on datasets that have been updated. No assertion about
which records came back can see that.
asterix-common already depends on hyracks-storage-am-vtree --
OptimizedScalarQuantizerFactory implements IVTreeQuantizerFactory -- so
the shared formula goes in VTreeScalarQuantization there and both
callers delegate down that existing edge. No new dependency, and the
codec keeps its validation, its Params/SimilarityFunction types and its
choice of storage width; only the per-dimension arithmetic moves.
They already agreed, and this is drift prevention rather than a fix:
with both files as they were, the new test passes. The reachable
configurations are SQ4 and SQ8 -- VectorQuantization is a closed enum of
the two -- and the two implementations were identical over them. Two
asymmetries did exist, both needing bits > 8:
- The DML path applied Math.toIntExact before the clamp. The codec
clamps in long deliberately, because for bits near 32 the pre-clamp
value can exceed Integer.MAX_VALUE, so the DML path would have thrown
ArithmeticException where the codec clamps. Gone with the shared
formula.
- The DML path wrote byte[] whatever the bit width, while the codec
selects byte[]/short[]/int[]. VTree leaf storage is byte[] only, so a
wider code has nowhere to go; VTreeDataTupleBuilder now refuses it at
construction instead of truncating silently. Unreachable today, so
this fails loudly if the enum grows without the storage format
following.
quantizeToShort and quantizeToInt are noted as having no reachable
caller and no reader on the VTree side, rather than removed.
QuantizationPathAgreementTest compares the two encoders byte for byte,
driving the DML side through the real tuple builder and reading it back
with VTreeDataTupleAccessor so the varlen framing of the embedding field
is covered too. Also asserted: agreement across every enum value, so
adding a scheme lands here; the quantile endpoints mapping to 0 and
levels - 1, since two paths agreeing proves nothing if both have the
same wrong alpha; the clamp holding outside the range; and decode
inverting encode to within half a step.
The fixture needed a second pass, which is worth recording. Round
quantile values make alpha round too -- 68 at SQ8, 4 at SQ4 -- and the
first draft's 0.0 and 1.0 put every product on an exact integer, so
floor and round agreed on all of them. With one path deliberately
changed to truncate, the test passed. It now uses values that round up
at both widths, and requireFixtureIsSensitiveToRounding fails the test
if none do, so the fixture demonstrates its own sensitivity rather than
resting on arithmetic done by hand. With that in place the truncating
path fails at element [4]: expected 111, got 110 at SQ8; expected 7, got
6 at SQ4.
Ext-ref: MB-73194
Co-Authored-By: Claude Opus 5 <[email protected]>
Change-Id: Ib7520274f950948b4fc8b9dcaeb7f5173ea57f4c
---
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/OptimizedScalarQuantizationCodec.java
A
asterixdb/asterix-common/src/test/java/org/apache/asterix/common/vector/QuantizationPathAgreementTest.java
M
hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeDataTupleBuilder.java
A
hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/utils/VTreeScalarQuantization.java
4 files changed, 339 insertions(+), 38 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/86/21686/1
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/OptimizedScalarQuantizationCodec.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/OptimizedScalarQuantizationCodec.java
index ae8d294..8a828ba 100644
---
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/OptimizedScalarQuantizationCodec.java
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/OptimizedScalarQuantizationCodec.java
@@ -21,6 +21,7 @@
import org.apache.asterix.common.exceptions.ErrorCode;
import org.apache.asterix.common.exceptions.RuntimeDataException;
import org.apache.hyracks.api.exceptions.HyracksDataException;
+import org.apache.hyracks.storage.am.vector.utils.VTreeScalarQuantization;
/**
* Optimized scalar quantization (OSQ) utilities for vector indexes.
@@ -187,25 +188,16 @@
}
/*
- * Per-dimension scalar encode/decode contract (used by quantizeToByte,
quantizeToShort, quantizeToInt).
- *
- * Parameter source: minQ, maxQ, alpha, and bits come from Params
(minQuantile, maxQuantile, alpha, bits),
- * populated at index creation by QuantizationConstantsAggregate:
- * levels = 2^bits
- * alpha = (levels - 1) / (maxQ - minQ)
- *
- * Encode (dimension i):
- * v = clamp(x[i], minQ, maxQ)
- * q = clamp(round((v - minQ) * alpha), 0, levels - 1)
- *
- * Decode (inverse, see dequantizeToDoubleArray):
- * x_hat[i] = q / alpha + minQ (minQ = Params.minQuantile)
- *
- * Endpoints: v = minQ -> q = 0; v = maxQ -> q = levels - 1 (after clamp
and round).
- * Rounding: Math.round selects the nearest integer code (standard
nearest-bin scalar quant).
+ * The per-dimension encode/decode formula itself lives in
VTreeScalarQuantization, in
+ * hyracks-storage-am-vtree, because the DML insert path
(VTreeDataTupleBuilder) has to apply exactly
+ * the same arithmetic and cannot depend on this module. Read the
contract, the parameter sources and
+ * the endpoint behaviour there; this class owns the validation, the
Params/SimilarityFunction types,
+ * and the choice of storage width.
*
* Storage by bits: bits <= 8 -> byte[] (SQ4 uses codes 0..15 in byte[]);
<= 16 -> short[]; <= 32 -> int[].
* Java byte/short are signed; codes above 127 or 32767 appear negative
unless read with & 0xFF / & 0xFFFF.
+ * Only SQ4 and SQ8 are reachable today -- VectorQuantization is a closed
enum of the two -- and VTree
+ * leaf storage is byte[] only, so the short[] and int[] widths have no
reader on the DML side.
*
* Debugging: log params.bits, levels, minQ, maxQ, alpha; check min/max q
across dims; compare
* dequantizeToDoubleArray(quantizeVector(x)) against x for round-trip
error on sample vectors.
@@ -227,11 +219,7 @@
private static byte[] quantizeToByte(double[] vector, float minQ, float
maxQ, float alpha, int levels) {
byte[] quantized = new byte[vector.length];
for (int i = 0; i < vector.length; i++) {
- // clamp to global quantile range, then map to integer code in [0,
levels - 1]
- double value = Math.max(minQ, Math.min(maxQ, vector[i]));
- long quantizedValue = Math.round((value - minQ) * alpha);
- quantizedValue = Math.max(0, Math.min(levels - 1, quantizedValue));
- quantized[i] = (byte) quantizedValue;
+ quantized[i] = (byte)
VTreeScalarQuantization.encodeDimension(vector[i], minQ, maxQ, alpha, levels);
}
return quantized;
}
@@ -249,11 +237,7 @@
private static short[] quantizeToShort(double[] vector, float minQ, float
maxQ, float alpha, int levels) {
short[] quantized = new short[vector.length];
for (int i = 0; i < vector.length; i++) {
- // clamp to global quantile range, then map to integer code in [0,
levels - 1]
- double value = Math.max(minQ, Math.min(maxQ, vector[i]));
- long quantizedValue = Math.round((value - minQ) * alpha);
- quantizedValue = Math.max(0, Math.min(levels - 1, quantizedValue));
- quantized[i] = (short) quantizedValue;
+ quantized[i] = (short)
VTreeScalarQuantization.encodeDimension(vector[i], minQ, maxQ, alpha, levels);
}
return quantized;
}
@@ -274,11 +258,7 @@
private static int[] quantizeToInt(double[] vector, float minQ, float
maxQ, float alpha, int levels) {
int[] quantized = new int[vector.length];
for (int i = 0; i < vector.length; i++) {
- // clamp to global quantile range, then map to integer code in [0,
levels - 1]
- double value = Math.max(minQ, Math.min(maxQ, vector[i]));
- long quantizedValue = Math.round((value - minQ) * alpha);
- quantizedValue = Math.max(0, Math.min(levels - 1, quantizedValue));
- quantized[i] = (int) quantizedValue;
+ quantized[i] = (int)
VTreeScalarQuantization.encodeDimension(vector[i], minQ, maxQ, alpha, levels);
}
return quantized;
}
@@ -331,7 +311,7 @@
throw new
RuntimeDataException(ErrorCode.VECTOR_DIMENSION_MISMATCH, dims, bytes.length);
}
for (int i = 0; i < dims; i++) {
- result[i] = ((double) (bytes[i] & 0xFF)) / params.alpha +
params.minQuantile;
+ result[i] = VTreeScalarQuantization.decodeDimension(bytes[i] &
0xFF, params.alpha, params.minQuantile);
}
} else if (bits <= 16) {
// short[] - treat as unsigned
@@ -344,7 +324,8 @@
throw new
RuntimeDataException(ErrorCode.VECTOR_DIMENSION_MISMATCH, dims, shorts.length);
}
for (int i = 0; i < dims; i++) {
- result[i] = ((double) (shorts[i] & 0xFFFF)) / params.alpha +
params.minQuantile;
+ result[i] =
+ VTreeScalarQuantization.decodeDimension(shorts[i] &
0xFFFF, params.alpha, params.minQuantile);
}
} else if (bits <= 32) {
// int[] - treat as unsigned
@@ -357,7 +338,8 @@
throw new
RuntimeDataException(ErrorCode.VECTOR_DIMENSION_MISMATCH, dims, ints.length);
}
for (int i = 0; i < dims; i++) {
- result[i] = ((double) (ints[i] & 0xFFFFFFFFL)) / params.alpha
+ params.minQuantile;
+ result[i] = VTreeScalarQuantization.decodeDimension(ints[i] &
0xFFFFFFFFL, params.alpha,
+ params.minQuantile);
}
} else {
throw new RuntimeDataException(ErrorCode.ILLEGAL_STATE,
diff --git
a/asterixdb/asterix-common/src/test/java/org/apache/asterix/common/vector/QuantizationPathAgreementTest.java
b/asterixdb/asterix-common/src/test/java/org/apache/asterix/common/vector/QuantizationPathAgreementTest.java
new file mode 100644
index 0000000..cbfc55f
--- /dev/null
+++
b/asterixdb/asterix-common/src/test/java/org/apache/asterix/common/vector/QuantizationPathAgreementTest.java
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.common.vector;
+
+import org.apache.hyracks.api.dataflow.value.ISerializerDeserializer;
+import org.apache.hyracks.api.exceptions.HyracksDataException;
+import org.apache.hyracks.dataflow.common.comm.io.ArrayTupleBuilder;
+import org.apache.hyracks.dataflow.common.comm.io.ArrayTupleReference;
+import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference;
+import
org.apache.hyracks.dataflow.common.data.marshalling.DoubleArraySerializerDeserializer;
+import
org.apache.hyracks.dataflow.common.data.marshalling.Integer64SerializerDeserializer;
+import org.apache.hyracks.dataflow.common.utils.TupleUtils;
+import org.apache.hyracks.storage.am.vector.api.VTreeQuantizationParams;
+import org.apache.hyracks.storage.am.vector.impls.VTreeDataTupleBuilder;
+import org.apache.hyracks.storage.am.vector.utils.VTreeDataTupleAccessor;
+import org.apache.hyracks.storage.am.vector.utils.VTreeScalarQuantization;
+import org.apache.hyracks.util.annotations.AiProvenance;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Two paths encode vectors for the same index: bulk load and static-structure
build go through {@link
+ * OptimizedScalarQuantizationCodec} here in {@code asterix-common}, and DML
insert goes through {@code
+ * VTreeDataTupleBuilder} in {@code hyracks-storage-am-vtree}, which cannot
depend on this module. They
+ * used to implement the arithmetic independently, and nothing compared them.
+ * <p>
+ * Drift between them is close to invisible. Both paths keep producing
well-formed codes, the index stays
+ * structurally valid, and every functional test still passes; what changes is
that a DML-inserted row's
+ * codes are no longer on the same scale as a bulk-loaded row's, so
approximate distances computed against
+ * them are wrong by a bias that only shows up as recall falling on datasets
that have been updated. No
+ * assertion about which records came back can see that.
+ * <p>
+ * These tests compare the two encoders on identical input, byte for byte. The
DML side is driven through
+ * the real tuple builder and read back with the production accessor, so the
comparison covers the varlen
+ * framing of the embedding field as well as the arithmetic.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.TEST_GENERATED)
+public class QuantizationPathAgreementTest {
+
+ private static final float MIN_Q = -1.25f;
+ private static final float MAX_Q = 2.5f;
+
+ /**
+ * Both endpoints, values outside the quantile range on each side (the
clamp is as much part of the
+ * formula as the scaling, and a divergence there would only affect
outliers), and interior values
+ * chosen so that some dimension's pre-round product has a fractional part
that rounds up at each
+ * supported bit width.
+ * <p>
+ * That last property is not decoration. {@code alpha} is {@code (levels -
1) / (maxQ - minQ)}, so round
+ * numbers for the quantiles make it a round number too -- here 68 at SQ8
and 4 at SQ4 -- and an
+ * obvious-looking fixture of {@code 0.0} and {@code 1.0} puts every
product on an exact integer. The
+ * two paths then agree even if one of them truncates instead of rounding,
and the comparison proves
+ * nothing. {@link #requireFixtureIsSensitiveToRounding} checks the
property rather than trusting the
+ * values below.
+ */
+ private static final double[] VECTOR =
+ { MIN_Q, MAX_Q, MIN_Q - 10.0, MAX_Q + 10.0, 0.375, -0.9, 1.703,
0.0088, 2.4999, -1.2499 };
+
+ /** SQ8: 256 levels, the default. */
+ @Test
+ public void bothPathsProduceTheSameCodesForSq8() throws
HyracksDataException {
+ assertPathsAgree(VectorQuantization.SQ8.bits(), VECTOR);
+ }
+
+ /** SQ4: 16 levels in a byte[], where a wrong level count is easy to miss.
*/
+ @Test
+ public void bothPathsProduceTheSameCodesForSq4() throws
HyracksDataException {
+ assertPathsAgree(VectorQuantization.SQ4.bits(), VECTOR);
+ }
+
+ /** Every quantization the product offers, so adding one to the enum lands
here. */
+ @Test
+ public void bothPathsAgreeForEveryQuantizationTheProductOffers() throws
HyracksDataException {
+ for (VectorQuantization quantization : VectorQuantization.values()) {
+ assertPathsAgree(quantization.bits(), VECTOR);
+ }
+ }
+
+ /**
+ * The endpoints pin the scale itself: {@code minQ} must land on code 0
and {@code maxQ} on the top
+ * code. A wrong {@code alpha} or an off-by-one in {@code levels} shifts
every other code with them,
+ * and the two paths agreeing would not catch it if both were wrong the
same way.
+ */
+ @Test
+ public void theQuantileEndpointsMapToTheEndsOfTheCodeRange() {
+ for (VectorQuantization quantization : VectorQuantization.values()) {
+ int levels = 1 << quantization.bits();
+ float alpha = (levels - 1) / (MAX_Q - MIN_Q);
+
+ Assert.assertEquals("minQ must encode to 0 for " + quantization, 0,
+ VTreeScalarQuantization.encodeDimension(MIN_Q, MIN_Q,
MAX_Q, alpha, levels));
+ Assert.assertEquals("maxQ must encode to the top code for " +
quantization, levels - 1,
+ VTreeScalarQuantization.encodeDimension(MAX_Q, MIN_Q,
MAX_Q, alpha, levels));
+ // And outside the range, the clamp holds them there rather than
wrapping.
+ Assert.assertEquals(0,
VTreeScalarQuantization.encodeDimension(MIN_Q - 1e6, MIN_Q, MAX_Q, alpha,
levels));
+ Assert.assertEquals(levels - 1,
+ VTreeScalarQuantization.encodeDimension(MAX_Q + 1e6,
MIN_Q, MAX_Q, alpha, levels));
+ }
+ }
+
+ /** Decode is the inverse to within one quantization step, which is what
bounds the recall loss. */
+ @Test
+ public void decodeInvertsEncodeToWithinOneStep() {
+ for (VectorQuantization quantization : VectorQuantization.values()) {
+ int levels = 1 << quantization.bits();
+ float alpha = (levels - 1) / (MAX_Q - MIN_Q);
+ double step = 1.0 / alpha;
+
+ for (double value : new double[] { MIN_Q, MAX_Q, 0.0, 1.0, 0.37,
-0.9 }) {
+ long code = VTreeScalarQuantization.encodeDimension(value,
MIN_Q, MAX_Q, alpha, levels);
+ double decoded = VTreeScalarQuantization.decodeDimension(code,
alpha, MIN_Q);
+ Assert.assertEquals("round-trip error above one step for " +
value + " at " + quantization, value,
+ decoded, step / 2 + 1e-6);
+ }
+ }
+ }
+
+ /**
+ * A quantized index whose codes are wider than a byte cannot round-trip
through the DML path, because
+ * leaf storage is {@code byte[]}. Unreachable while {@code
VectorQuantization} is SQ4/SQ8, so this
+ * asserts the guard rather than a behaviour: it is what turns a silent
truncation into a failure if
+ * the enum grows without the storage format following.
+ */
+ @Test
+ public void aCodeWiderThanAByteIsRefusedByTheDmlPath() {
+ VTreeQuantizationParams tooWide = new VTreeQuantizationParams(MIN_Q,
MAX_Q, 1.0f, 0.9f, 16, 1000);
+ Assert.assertThrows(IllegalArgumentException.class, () -> new
VTreeDataTupleBuilder(0, true, tooWide));
+ }
+
+ // ---- comparison
-----------------------------------------------------------------------------
+
+ /**
+ * Encode {@code vector} through both paths with the same parameters and
require identical bytes.
+ */
+ private void assertPathsAgree(int bits, double[] vector) throws
HyracksDataException {
+ int levels = 1 << bits;
+ float alpha = (levels - 1) / (MAX_Q - MIN_Q);
+
+ requireFixtureIsSensitiveToRounding(vector, bits, alpha);
+
+ byte[] fromBulkLoad = bulkLoadCodes(vector, bits, alpha);
+ byte[] fromDml = dmlCodes(vector, bits, alpha);
+
+ Assert.assertEquals("code count", fromBulkLoad.length, fromDml.length);
+ Assert.assertArrayEquals("the bulk-load and DML paths disagree at " +
bits
+ + " bits; a row inserted by DML would not be" + " comparable
with a bulk-loaded row", fromBulkLoad,
+ fromDml);
+ // Guard against both paths agreeing on nothing: at these parameters
the codes must actually differ
+ // across dimensions, or the comparison above would hold for any
constant encoder.
+ boolean allEqual = true;
+ for (byte code : fromDml) {
+ allEqual &= code == fromDml[0];
+ }
+ Assert.assertFalse("the fixture produced a constant code vector, so it
proves nothing", allEqual);
+ }
+
+ /**
+ * Require that at least one dimension's pre-round product would land
differently under a different
+ * rounding rule. Without that, the byte-for-byte comparison is blind to
the rounding half of the
+ * formula -- which is how this fixture was wrong on its first draft: it
passed while one path had been
+ * deliberately changed to truncate.
+ */
+ private void requireFixtureIsSensitiveToRounding(double[] vector, int
bits, float alpha) {
+ int sensitive = 0;
+ for (double value : vector) {
+ double product = (Math.max(MIN_Q, Math.min(MAX_Q, value)) - MIN_Q)
* alpha;
+ if (Math.round(product) != (long) Math.floor(product)) {
+ sensitive++;
+ }
+ }
+ Assert.assertTrue("the fixture is insensitive to the rounding rule at
" + bits + " bits: every"
+ + " dimension's pre-round product is an exact integer, so a
truncating encoder would agree"
+ + " with a rounding one", sensitive > 0);
+ }
+
+ /** The bulk-load / static-structure encoder. */
+ private byte[] bulkLoadCodes(double[] vector, int bits, float alpha)
throws HyracksDataException {
+ OptimizedScalarQuantizationCodec.Params params =
+ new OptimizedScalarQuantizationCodec.Params(bits,
vector.length, 1000, 0.9f, MIN_Q, MAX_Q, alpha);
+ OptimizedScalarQuantizationCodec.QuantizedVector quantized =
OptimizedScalarQuantizationCodec
+ .quantizeVector(vector, params,
OptimizedScalarQuantizationCodec.SimilarityFunction.EUCLIDEAN);
+ Assert.assertTrue("SQ4/SQ8 must encode to byte[]",
quantized.quantizedBytes instanceof byte[]);
+ return (byte[]) quantized.quantizedBytes;
+ }
+
+ /**
+ * The DML insert encoder, driven through the real tuple builder and read
back with the production
+ * accessor so the varlen framing of the embedding field is covered too.
+ */
+ private byte[] dmlCodes(double[] vector, int bits, float alpha) throws
HyracksDataException {
+ VTreeQuantizationParams params = new VTreeQuantizationParams(MIN_Q,
MAX_Q, alpha, 0.9f, bits, 1000);
+ VTreeDataTupleBuilder builder = new VTreeDataTupleBuilder(0, true,
params);
+
+ ITupleReference dataTuple = builder.buildDataTuple(vector, 0.5, 7,
inputTuple(vector, 42L));
+
+ return new
VTreeDataTupleAccessor(true).getQuantizedEmbedding(dataTuple);
+ }
+
+ /** {@code [vector, pk]} — the operator-side input layout with no include
fields. */
+ private static ITupleReference inputTuple(double[] vector, long
primaryKey) throws HyracksDataException {
+ ArrayTupleBuilder tupleBuilder = new ArrayTupleBuilder(2);
+ ArrayTupleReference tupleRef = new ArrayTupleReference();
+ ISerializerDeserializer[] serdes =
+ { DoubleArraySerializerDeserializer.INSTANCE,
Integer64SerializerDeserializer.INSTANCE };
+ TupleUtils.createTuple(tupleBuilder, tupleRef, serdes, new Object[] {
vector, primaryKey });
+ return tupleRef;
+ }
+}
diff --git
a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeDataTupleBuilder.java
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeDataTupleBuilder.java
index 72d9ea8..4a24edc 100644
---
a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeDataTupleBuilder.java
+++
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeDataTupleBuilder.java
@@ -29,6 +29,7 @@
import org.apache.hyracks.storage.am.vector.api.IVTreeDataTupleBuilder;
import org.apache.hyracks.storage.am.vector.api.VTreeQuantizationParams;
import org.apache.hyracks.storage.am.vector.utils.VTreeDataTupleAccessor;
+import org.apache.hyracks.storage.am.vector.utils.VTreeScalarQuantization;
import org.apache.hyracks.util.encoding.VarLenIntEncoderDecoder;
/**
@@ -58,6 +59,15 @@
public VTreeDataTupleBuilder(int numIncludeFields, boolean isQuantized,
VTreeQuantizationParams quantizationParams) {
+ // Leaf storage for the quantized embedding is byte[] (see
VTreeDataTupleAccessor#getQuantizedEmbedding),
+ // so a code wider than 8 bits has nowhere to go: it would be silently
truncated here while the
+ // bulk-load codec stored it as short[]/int[], and the two paths' rows
would stop being comparable.
+ // Unreachable today -- VectorQuantization offers SQ4 and SQ8 only --
so this fails loudly if that
+ // enum ever grows without the storage format following.
+ if (isQuantized && quantizationParams != null &&
quantizationParams.bits() > Byte.SIZE) {
+ throw new IllegalArgumentException("VTree leaf storage holds " +
Byte.SIZE
+ + "-bit quantization codes; this index was built with " +
quantizationParams.bits() + " bits");
+ }
this.numIncludeFields = numIncludeFields;
this.isQuantized = isQuantized;
this.quantizationParams = quantizationParams;
@@ -134,10 +144,8 @@
}
byte[] result = quantizeScratch;
for (int i = 0; i < vector.length; i++) {
- double value = Math.max(minQ, Math.min(maxQ, vector[i]));
- int quantizedValue = Math.toIntExact(Math.round((value - minQ)
* alpha));
- quantizedValue = Math.max(0, Math.min(levels - 1,
quantizedValue));
- result[i] = (byte) quantizedValue;
+ // Same formula as the bulk-load path's codec, by construction
-- see VTreeScalarQuantization.
+ result[i] = (byte)
VTreeScalarQuantization.encodeDimension(vector[i], minQ, maxQ, alpha, levels);
}
return result;
}
diff --git
a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/utils/VTreeScalarQuantization.java
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/utils/VTreeScalarQuantization.java
new file mode 100644
index 0000000..2ddb975
--- /dev/null
+++
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/utils/VTreeScalarQuantization.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hyracks.storage.am.vector.utils;
+
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/**
+ * The per-dimension scalar quantization formula, in one place.
+ * <p>
+ * Two paths encode vectors for the same index and must produce identical
codes for identical input:
+ * bulk load and static-structure build go through AsterixDB's {@code
OptimizedScalarQuantizationCodec}
+ * (module {@code asterix-common}), while DML insert goes through {@link
+ * org.apache.hyracks.storage.am.vector.impls.VTreeDataTupleBuilder} in this
module. {@code
+ * hyracks-storage-am-vtree} cannot depend on {@code asterix-common}, so the
DML path used to
+ * re-implement the arithmetic rather than call it, and the two copies could
drift silently: a
+ * DML-inserted row's codes would stop being comparable with a bulk-loaded
row's, and the only symptom
+ * would be recall quietly falling for updated datasets. {@code
asterix-common} already depends on this
+ * module, so the shared formula lives here and both callers delegate down to
it.
+ * <p>
+ * The parameters are global to the index, computed at creation by {@code
+ * QuantizationConstantsAggregate} and carried on the resource: {@code levels
= 2^bits} and {@code alpha
+ * = (levels - 1) / (maxQ - minQ)}.
+ * <p>
+ * <b>Encode</b> (dimension i): {@code v = clamp(x[i], minQ, maxQ)}, then
+ * {@code q = clamp(round((v - minQ) * alpha), 0, levels - 1)}.<br>
+ * <b>Decode</b> (the inverse): {@code x_hat[i] = q / alpha + minQ}.
+ * <p>
+ * Endpoints: {@code v == minQ} yields {@code q == 0}; {@code v == maxQ}
yields {@code q == levels - 1}.
+ * The intermediate expressions are evaluated in {@code double} and the code
in {@code long} because the
+ * clamp comes after the rounding: for {@code bits} near 32 the pre-clamp
value can exceed {@code
+ * Integer.MAX_VALUE}, so narrowing before the clamp would overflow. Callers
narrow the returned code to
+ * their own storage width.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.REFACTORED, notes = "Single implementation of a
formula previously duplicated across the asterix-common / hyracks module
boundary")
+public final class VTreeScalarQuantization {
+
+ private VTreeScalarQuantization() {
+ }
+
+ /**
+ * Encode one dimension to its integer code, clamped into {@code [0,
levels - 1]}.
+ *
+ * @param value the full-precision component
+ * @param minQ lower sample quantile over all dimensions
+ * @param maxQ upper sample quantile over all dimensions
+ * @param alpha {@code (levels - 1) / (maxQ - minQ)}
+ * @param levels {@code 1 << bits}
+ * @return the code, in {@code [0, levels - 1]}; narrow to the caller's
storage width
+ */
+ public static long encodeDimension(double value, float minQ, float maxQ,
float alpha, int levels) {
+ // Clamp to the global quantile range, then map onto the integer code
range.
+ double clamped = Math.max(minQ, Math.min(maxQ, value));
+ long code = Math.round((clamped - minQ) * alpha);
+ return Math.max(0, Math.min(levels - 1, code));
+ }
+
+ /**
+ * Decode one code back to an approximate component. Inverse of
+ * {@link #encodeDimension(double, float, float, float, int)}, up to the
quantization step.
+ * <p>
+ * Codes are unsigned: a caller reading a signed {@code byte} or {@code
short} must widen with
+ * {@code & 0xFF} / {@code & 0xFFFF} before passing it here.
+ *
+ * @param code the unsigned code
+ * @param alpha {@code (levels - 1) / (maxQ - minQ)}
+ * @param minQ lower sample quantile over all dimensions
+ */
+ public static double decodeDimension(long code, float alpha, float minQ) {
+ return (double) code / alpha + minQ;
+ }
+}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21686?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib7520274f950948b4fc8b9dcaeb7f5173ea57f4c
Gerrit-Change-Number: 21686
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <[email protected]>
