>From Ali Alsuliman <[email protected]>:
Ali Alsuliman has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21620?usp=email )
Change subject: [ASTERIXDB-3817][COMP][MTD] Enum for quantization
......................................................................
[ASTERIXDB-3817][COMP][MTD] Enum for quantization
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
The vector index `quantization` parameter was a String whose only real
consumer converted it to a bit width:
return QUANTIZATION_SQ4.equals(label) ? 4 : DEFAULT_QUANTIZATION_BITS_SQ8;
That is a per-constant attribute written as a two-branch lookup with a
silent fallback: "SQ16", "sq4", "" or a typo all yield 8. Nothing caught
it either, because once the label is guaranteed non-null there is no
validation left on the read path -- readFields accepted any string, so a
corrupt label would decode the stored embeddings at the wrong bit width
while the sibling `similarity` parameter, typed since the previous
change, rejects an unrecognized value outright.
Introduce VectorQuantization, a sibling of VectorSimilarityMetric in the
same package, carrying the bit width on the constant. VectorIndexParameters
holds it instead of a String, so the bit width comes straight off the
scheme and an unrecognized persisted label is reported rather than
defaulted. Folded into the enum and removed from VectorIndexParameters:
QUANTIZATION_SQ4, QUANTIZATION_SQ8, QUANTIZATION_LABELS,
DEFAULT_QUANTIZATION_BITS_SQ8, quantizationBits(String),
isAllowedQuantization(String) and quantizationList().
The label is unchanged on disk -- label() is name(), which is what was
persisted before -- so existing indexes read back identically. The enum
documents that this makes the constant names a durable contract: add a
constant rather than renaming one, and give label() its own string if a
name ever has to change.
isQuantized() is deliberately left alone. It stays the single predicate the
three tuple-layout decisions key off; when a non-quantized mode becomes
reachable from DDL it becomes `quantization != NONE` rather than the
constant it is today.
Tests:
- VectorIndexParametersTupleTranslatorTest asserts the scheme round-trips
and that its bit width survives, and continues to check that an absent
`quantization` reads back as the default.
- The vector/create-index-vtree-metadata fixture still expects
"quantization": "SQ4" in the persisted record, which is what confirms
the on-disk spelling did not move.
Ext-ref: MB-73194
Co-Authored-By: Claude Opus 5 <[email protected]>
Change-Id: Ie6987f447505231c0614de2efc9af0981b86cf89
---
A
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/VectorQuantization.java
M
asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
M
asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
M
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
5 files changed, 123 insertions(+), 57 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/20/21620/1
diff --git
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/VectorQuantization.java
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/VectorQuantization.java
new file mode 100644
index 0000000..4f6dea8
--- /dev/null
+++
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/vector/VectorQuantization.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.common.vector;
+
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.stream.Collectors;
+
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/**
+ * The quantization schemes a VTree index can encode its embeddings with,
together with the bit width each
+ * one quantizes to. Sibling of {@link VectorSimilarityMetric}: the single
source of truth for the values the
+ * index {@code quantization} option accepts.
+ * <p>
+ * The bit width is what actually reaches the storage layer — as {@code
OptimizedScalarQuantizationCodec.Params}
+ * and as a field on the local resource — so it belongs on the constant rather
than in a lookup that has to
+ * decide what an unrecognized label means.
+ * <p>
+ * {@link #label()} is the spelling written in DDL and <em>persisted</em> in
the {@code Metadata.Index} record.
+ * It is {@link #name()} today, which means renaming a constant strands
existing indexes: add a constant rather
+ * than renaming one, and if a name ever has to change, give {@code label()}
its own string instead.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.GENERATED)
+public enum VectorQuantization {
+ SQ4(4),
+ SQ8(8);
+
+ private final int bits;
+
+ VectorQuantization(int bits) {
+ this.bits = bits;
+ }
+
+ /** Bits per component this scheme quantizes an embedding to. */
+ public int bits() {
+ return bits;
+ }
+
+ /** The DDL spelling, and the form persisted in index metadata. */
+ public String label() {
+ return name();
+ }
+
+ /**
+ * Resolves a {@code quantization} label, trimmed and case-insensitively.
+ *
+ * @return the matching scheme, or {@code null} if the label is not
recognized.
+ */
+ public static VectorQuantization fromLabel(String label) {
+ if (label == null) {
+ return null;
+ }
+ String normalized = label.trim().toUpperCase(Locale.ROOT);
+ for (VectorQuantization quantization : values()) {
+ if (quantization.label().equals(normalized)) {
+ return quantization;
+ }
+ }
+ return null;
+ }
+
+ /** Comma-separated labels, for "allowed values are ..." diagnostics. */
+ public static String labelList() {
+ return
Arrays.stream(values()).map(VectorQuantization::label).collect(Collectors.joining(",
"));
+ }
+}
diff --git
a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
index cb2fcad..670e726 100644
---
a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
+++
b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
@@ -37,6 +37,7 @@
import org.apache.asterix.common.exceptions.AsterixException;
import org.apache.asterix.common.exceptions.CompilationException;
import org.apache.asterix.common.exceptions.ErrorCode;
+import org.apache.asterix.common.vector.VectorQuantization;
import org.apache.asterix.common.vector.VectorSimilarityMetric;
import org.apache.asterix.lang.common.expression.RecordConstructor;
import org.apache.asterix.object.base.AdmBigIntNode;
@@ -158,23 +159,22 @@
return metric;
}
- private static String validateQuantization(AdmObjectNode node) throws
CompilationException {
+ /**
+ * Validates {@code quantization} and returns the resolved scheme.
Optional: an absent value takes
+ * {@link VectorIndexParameters#DEFAULT_QUANTIZATION}, since every vector
index is quantized.
+ */
+ private static VectorQuantization validateQuantization(AdmObjectNode node)
throws CompilationException {
IAdmNode qNode = node.get(QUANTIZATION);
if (qNode == null) {
return DEFAULT_QUANTIZATION;
}
- if (qNode.getType() != ATypeTag.STRING) {
+ VectorQuantization quantization =
+ qNode.getType() == ATypeTag.STRING ?
VectorQuantization.fromLabel(((AdmStringNode) qNode).get()) : null;
+ if (quantization == null) {
throw new
CompilationException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED,
- "Invalid `quantization` parameter value. Allowed values: "
- + VectorIndexParameters.quantizationList());
+ "Invalid `quantization` parameter value. Allowed values: "
+ VectorQuantization.labelList());
}
- String normalized = ((AdmStringNode)
qNode).get().trim().toUpperCase(Locale.ROOT);
- if (!VectorIndexParameters.isAllowedQuantization(normalized)) {
- throw new
CompilationException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED,
- "Invalid `quantization` parameter value. Allowed values: "
- + VectorIndexParameters.quantizationList());
- }
- return normalized;
+ return quantization;
}
/**
diff --git
a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
index 9419742..67c9d4a 100644
---
a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
+++
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
@@ -549,9 +549,8 @@
"Run ANALYZE on the dataset before creating a vector
index.");
}
- // 2. Extract quantization parameters (default label SQ8 matches DDL;
bits 8 for SQ8)
- String qLabel = vectorParameters.getQuantization();
- int bits = VectorIndexParameters.quantizationBits(qLabel);
+ // 2. Extract quantization parameters (the scheme defaults to SQ8 at
DDL time, so bits is always set)
+ int bits = vectorParameters.getQuantization().bits();
// confidence_interval is not a declared WITH parameter (see
VectorIndexParameters), so it was never
// readable from the WITH clause; use the constant directly until it
becomes a real DDL knob.
float confidenceInterval = DEFAULT_CONFIDENCE_INTERVAL;
diff --git
a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
index 89bd755..2432ee1 100644
---
a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
+++
b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
@@ -37,6 +37,7 @@
import org.apache.asterix.common.config.DatasetConfig.IndexType;
import org.apache.asterix.common.metadata.DataverseName;
import org.apache.asterix.common.metadata.MetadataUtil;
+import org.apache.asterix.common.vector.VectorQuantization;
import org.apache.asterix.common.vector.VectorSimilarityMetric;
import org.apache.asterix.metadata.MetadataNode;
import org.apache.asterix.metadata.bootstrap.IndexEntity;
@@ -78,15 +79,16 @@
public void everyParameterRoundTrips() throws AlgebricksException,
IOException {
VectorIndexParameters written =
VectorIndexParameters.builder().setDimension(128).setSimilarity(VectorSimilarityMetric.EUCLIDEAN)
-
.setQuantization(VectorIndexParameters.QUANTIZATION_SQ4).setTrainListFraction(0.375)
-
.setEpsilon(0.625).setNumClusters(7).setCrossPollinationM(3).setRngFactor(1.5).build();
+
.setQuantization(VectorQuantization.SQ4).setTrainListFraction(0.375).setEpsilon(0.625)
+
.setNumClusters(7).setCrossPollinationM(3).setRngFactor(1.5).build();
VectorIndexParameters readBack = roundTrip(written);
Assert.assertEquals(written, readBack);
Assert.assertEquals(128, readBack.getDimension());
Assert.assertEquals(VectorSimilarityMetric.EUCLIDEAN,
readBack.getSimilarity());
- Assert.assertEquals(VectorIndexParameters.QUANTIZATION_SQ4,
readBack.getQuantization());
+ Assert.assertEquals(VectorQuantization.SQ4,
readBack.getQuantization());
+ Assert.assertEquals(4, readBack.getQuantization().bits());
Assert.assertTrue(readBack.isQuantized());
Assert.assertEquals(0.375, readBack.getTrainListFraction(), 0.0);
Assert.assertEquals(0.625, readBack.getEpsilon(), 0.0);
diff --git
a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
index 8140dc4..90dc817 100644
---
a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
+++
b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
@@ -26,6 +26,7 @@
import org.apache.asterix.builders.IARecordBuilder;
import org.apache.asterix.common.exceptions.AsterixException;
import org.apache.asterix.common.exceptions.ErrorCode;
+import org.apache.asterix.common.vector.VectorQuantization;
import org.apache.asterix.common.vector.VectorSimilarityMetric;
import org.apache.asterix.formats.nontagged.SerializerDeserializerProvider;
import org.apache.asterix.om.base.ADouble;
@@ -79,18 +80,7 @@
public static final String CROSS_POLLINATION_M = "cross_pollination_m";
public static final String RNG_FACTOR = "rng_factor";
- public static final String QUANTIZATION_SQ4 = "SQ4";
- public static final String QUANTIZATION_SQ8 = "SQ8";
- /**
- * Every accepted {@code quantization} label. Both the DDL validator and
the build path check against this
- * one declaration, so they cannot drift apart on which labels exist. A
{@code List} rather than a
- * {@code Set} because {@link #quantizationList()} renders it into a
diagnostic: {@code Set.of} iteration
- * order is randomized per JVM run, which would make that message's
wording vary between runs.
- */
- private static final List<String> QUANTIZATION_LABELS =
List.of(QUANTIZATION_SQ4, QUANTIZATION_SQ8);
- public static final String DEFAULT_QUANTIZATION = QUANTIZATION_SQ8;
- /** Bit width of {@link #DEFAULT_QUANTIZATION} ({@link
#QUANTIZATION_SQ8}); {@link #QUANTIZATION_SQ4} is 4. */
- public static final int DEFAULT_QUANTIZATION_BITS_SQ8 = 8;
+ public static final VectorQuantization DEFAULT_QUANTIZATION =
VectorQuantization.SQ8;
public static final double DEFAULT_TRAIN_LIST_FRACTION = 0.1;
public static final double DEFAULT_EPSILON = 0.25;
public static final int DEFAULT_CROSS_POLLINATION_M = 1;
@@ -107,7 +97,7 @@
private final int dimension;
private final VectorSimilarityMetric similarity;
- private final String quantization;
+ private final VectorQuantization quantization;
private final double trainListFraction;
private final double epsilon;
/** {@code null} when unset: the builder then derives it from the dataset
cardinality at build time. */
@@ -143,12 +133,12 @@
}
/**
- * The quantization label, never {@code null}: it is optional in the
{@code WITH} clause but defaults to
+ * The quantization scheme, never {@code null}: it is optional in the
{@code WITH} clause but defaults to
* {@link #DEFAULT_QUANTIZATION}, and an index record that predates the
parameter reads back as that
- * default too. {@code
SecondaryVectorOperationsHelper#buildCreationJobSpec} relies on this — it
derives
- * the quantization bit width from the label unconditionally, outside any
{@link #isQuantized()} guard.
+ * default too. {@code
SecondaryVectorOperationsHelper#buildCreationJobSpec} relies on this — it takes
the
+ * scheme's bit width unconditionally, outside any {@link #isQuantized()}
guard.
*/
- public String getQuantization() {
+ public VectorQuantization getQuantization() {
return quantization;
}
@@ -203,20 +193,6 @@
return String.join(", ", NAMES);
}
- public static boolean isAllowedQuantization(String label) {
- return QUANTIZATION_LABELS.contains(label);
- }
-
- /** Comma-separated {@code quantization} labels, for "allowed values are
..." diagnostics. */
- public static String quantizationList() {
- return String.join(", ", QUANTIZATION_LABELS);
- }
-
- /** Bit width the given (already normalized) quantization label encodes
at. */
- public static int quantizationBits(String label) {
- return QUANTIZATION_SQ4.equals(label) ? 4 :
DEFAULT_QUANTIZATION_BITS_SQ8;
- }
-
/**
* Serializes this configuration into {@code recordBuilder} as one open
field per present parameter, in
* {@link #NAMES} order. Only {@code num_clusters} is genuinely optional,
and it is written only when set,
@@ -226,7 +202,7 @@
FieldWriter writer = new FieldWriter(recordBuilder);
writer.writeInt(DIMENSION, dimension);
writer.writeString(SIMILARITY, similarity.canonical());
- writer.writeString(QUANTIZATION, quantization);
+ writer.writeString(QUANTIZATION, quantization.label());
writer.writeDouble(TRAIN_LIST_FRACTION, trainListFraction);
writer.writeDouble(EPSILON, epsilon);
if (numClusters != null) {
@@ -263,9 +239,16 @@
builder.setSimilarity(metric);
}
// An index record with no `quantization` field predates the
parameter; it reads back as the default,
- // which is the label its creation job would have quantized with
anyway.
- String quantization = reader.readString(QUANTIZATION);
- if (quantization != null) {
+ // which is the scheme its creation job would have quantized with
anyway. A field that is present but
+ // unrecognized is corrupt metadata, not a defaultable case: silently
falling back to SQ8 would decode
+ // the stored embeddings at the wrong bit width.
+ String quantizationStr = reader.readString(QUANTIZATION);
+ if (quantizationStr != null) {
+ VectorQuantization quantization =
VectorQuantization.fromLabel(quantizationStr);
+ if (quantization == null) {
+ throw new
AsterixException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED,
+ "Unrecognized `" + QUANTIZATION + "` value `" +
quantizationStr + "` in index metadata");
+ }
builder.setQuantization(quantization);
}
Double trainListFraction = reader.readDouble(TRAIN_LIST_FRACTION);
@@ -296,10 +279,9 @@
if (this == o) {
return true;
}
- if (!(o instanceof VectorIndexParameters)) {
+ if (!(o instanceof VectorIndexParameters other)) {
return false;
}
- VectorIndexParameters other = (VectorIndexParameters) o;
return dimension == other.dimension && Objects.equals(similarity,
other.similarity)
&& Objects.equals(quantization, other.quantization)
&& Double.compare(trainListFraction, other.trainListFraction)
== 0
@@ -318,7 +300,7 @@
StringBuilder sb = new StringBuilder("{ ");
sb.append(DIMENSION).append(": ").append(dimension);
sb.append(", ").append(SIMILARITY).append(":
").append(similarity.canonical());
- sb.append(", ").append(QUANTIZATION).append(": ").append(quantization);
+ sb.append(", ").append(QUANTIZATION).append(":
").append(quantization.label());
sb.append(", ").append(TRAIN_LIST_FRACTION).append(":
").append(trainListFraction);
sb.append(", ").append(EPSILON).append(": ").append(epsilon);
if (numClusters != null) {
@@ -338,7 +320,7 @@
private int dimension = -1;
private VectorSimilarityMetric similarity;
- private String quantization = DEFAULT_QUANTIZATION;
+ private VectorQuantization quantization = DEFAULT_QUANTIZATION;
private double trainListFraction = DEFAULT_TRAIN_LIST_FRACTION;
private double epsilon = DEFAULT_EPSILON;
private Integer numClusters;
@@ -358,7 +340,7 @@
return this;
}
- public Builder setQuantization(String quantization) {
+ public Builder setQuantization(VectorQuantization quantization) {
this.quantization = Objects.requireNonNull(quantization,
QUANTIZATION);
return this;
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21620?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ie6987f447505231c0614de2efc9af0981b86cf89
Gerrit-Change-Number: 21620
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <[email protected]>
