>From Hongyu Shi <[email protected]>: Hongyu Shi has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21650?usp=email )
Change subject: [ASTERIXDB-3760][RT] Report an unusable sample as a build failure ...................................................................... [ASTERIXDB-3760][RT] Report an unusable sample as a build failure - user model changes: yes - storage format changes: no - interface changes: no Details: When the sample yields no vector usable at the declared dimension, the quantization stage has no constants to emit and the build cannot proceed. Reporting that as ILLEGAL_STATE describes a broken invariant, when it is really a property of the user's data and is reachable with a collection whose indexed field is NULL in every record. A consumer that maps error codes to user-facing ones deliberately leaves the internal-invariant codes unmapped, so it surfaced as a bare "Internal error" with the explanation only in the server log. Report it as VECTOR_INDEX_BUILD_FAILED instead, which the sibling throws in this class and in VTreeStaticStructureBuilder and VTreeBulkLoader already use for the same family of condition, and drop the payload byte count from the text because the code already supplies "Failed to build the vector index". The two guards above it stay ILLEGAL_STATE because a second tuple or a multi-tuple frame really is a broken invariant. Ext-ref: MB-73278, MB-73424 Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: I6a3c11c1ac2d46c7a074a2ce01bcaeb570407ef1 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.01.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.02.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.03.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.04.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.99.ddl.sqlpp M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java 7 files changed, 162 insertions(+), 6 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/50/21650/1 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml index 376c7e5..1bd0783 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml @@ -80,6 +80,14 @@ </compilation-unit> </test-case> <test-case FilePath="vector"> + <compilation-unit name="create-index-vtree-null-embedding"> + <output-dir compare="Text">create-index-vtree-null-embedding</output-dir> + <expected-error>Failed to build the vector index. The sampled records yielded no usable vector</expected-error> + <!-- Raised by a build job on an NC, so it carries no source location. --> + <source-location>false</source-location> + </compilation-unit> + </test-case> + <test-case FilePath="vector"> <compilation-unit name="create-index-vtree-bad-similarity"> <output-dir compare="Text">create-index-vtree-bad-similarity</output-dir> <expected-error>Allowed values: EUCLIDEAN, L2, EUCLIDEAN_SQUARED, L2_SQUARED, COSINE, COSINE SIMILARITY, DOT</expected-error> diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.01.ddl.sqlpp new file mode 100644 index 0000000..fd3dcbc --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.01.ddl.sqlpp @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +// Every record's embedding is NULL, so the sample yields no usable vector and the build cannot +// derive quantization constants. The failure must name that cause as a build failure the user can +// act on, not an internal-invariant error. The type is open so `embedding` can be NULL at all. +DROP DATAVERSE test IF EXISTS; +CREATE DATAVERSE test; +USE test; + +CREATE TYPE MovieType AS { + id: int +}; + +CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.02.update.sqlpp new file mode 100644 index 0000000..4b6eafa --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.02.update.sqlpp @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +INSERT INTO MovieSmall ([ + {"id": 1, "embedding": null}, + {"id": 2, "embedding": null}, + {"id": 3, "embedding": null}, + {"id": 4, "embedding": null}, + {"id": 5, "embedding": null}, + {"id": 6, "embedding": null}, + {"id": 7, "embedding": null}, + {"id": 8, "embedding": null}, + {"id": 9, "embedding": null}, + {"id": 10, "embedding": null}, + {"id": 11, "embedding": null}, + {"id": 12, "embedding": null}, + {"id": 13, "embedding": null}, + {"id": 14, "embedding": null}, + {"id": 15, "embedding": null}, + {"id": 16, "embedding": null}, + {"id": 17, "embedding": null}, + {"id": 18, "embedding": null}, + {"id": 19, "embedding": null}, + {"id": 20, "embedding": null}, + {"id": 21, "embedding": null}, + {"id": 22, "embedding": null}, + {"id": 23, "embedding": null}, + {"id": 24, "embedding": null}, + {"id": 25, "embedding": null}, + {"id": 26, "embedding": null}, + {"id": 27, "embedding": null}, + {"id": 28, "embedding": null}, + {"id": 29, "embedding": null}, + {"id": 30, "embedding": null} +]); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.03.ddl.sqlpp new file mode 100644 index 0000000..b840167 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.03.ddl.sqlpp @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000}; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.04.ddl.sqlpp new file mode 100644 index 0000000..65ee6d8 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.04.ddl.sqlpp @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +CREATE INDEX idx_emb + ON MovieSmall(embedding VECTOR) + TYPE VTREE + WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 2, "train_list_fraction": 1.0, "seed": 42 } + EXCLUDE UNKNOWN KEY; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.99.ddl.sqlpp new file mode 100644 index 0000000..d240ef2 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-null-embedding/create-index-vtree-null-embedding.99.ddl.sqlpp @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +DROP DATAVERSE test IF EXISTS; diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java index fbecb7c..bcabada 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java @@ -29,6 +29,7 @@ import org.apache.hyracks.api.dataflow.value.RecordDescriptor; import org.apache.hyracks.api.exceptions.ErrorCode; import org.apache.hyracks.api.exceptions.HyracksDataException; +import org.apache.hyracks.util.annotations.AiProvenance; import org.apache.hyracks.api.job.IOperatorDescriptorRegistry; import org.apache.hyracks.data.std.primitive.ByteArrayPointable; import org.apache.hyracks.dataflow.common.comm.io.FrameTupleAccessor; @@ -123,6 +124,7 @@ // confidenceInterval (float), sampleCount (int). private static final int PARAMS_PAYLOAD_BYTES = 6 * Integer.BYTES; + @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED) private VTreeQuantizationParams extractQuantizationParams(FrameTupleReference tuple) throws HyracksDataException { byte[] data = tuple.getFieldData(0); @@ -136,12 +138,13 @@ // The global aggregate emits either the full parameter block or nothing, and nothing means the // sample yielded no usable vector. if (contentLength < PARAMS_PAYLOAD_BYTES) { - throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE, - "no usable quantization parameters for this vector index (" + contentLength + " of " - + PARAMS_PAYLOAD_BYTES + " bytes). The sampled records yielded no usable vector for " - + "the indexed field: it may be missing, null or not a list in every sampled record, " - + "the dataset may be empty, or no sampled vector may match the dimension the index " - + "declares."); + // A user data condition, not a broken invariant: report it as a build failure the user can + // act on rather than ILLEGAL_STATE, which cbas deliberately leaves unmapped and so surfaces + // as a bare "Internal error". + throw HyracksDataException.create(ErrorCode.VECTOR_INDEX_BUILD_FAILED, + "The sampled records yielded no usable vector for the indexed field: it may be missing, " + + "null or not a list in every sampled record, the dataset may be empty, or no " + + "sampled vector may match the dimension the index declares."); } // Big-endian; DataInputStream is the exact inverse of the DataOutput/ByteBuffer writer and -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21650?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I6a3c11c1ac2d46c7a074a2ce01bcaeb570407ef1 Gerrit-Change-Number: 21650 Gerrit-PatchSet: 1 Gerrit-Owner: Hongyu Shi <[email protected]>
