>From Hongyu Shi <[email protected]>: Hongyu Shi has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21651?usp=email )
Change subject: [ASTERIXDB-3771][COMP] Do not rewrite a back-to-front ann_distance ...................................................................... [ASTERIXDB-3771][COMP] Do not rewrite a back-to-front ann_distance - user model changes: yes - storage format changes: no - interface changes: no Details: ann_distance(queryVector, field, metric) failed during plan generation with "Could not resolve type for $$n". createIndexSearchPlan reads argument 1 as the query vector while the shared one-constant-one-variable analysis accepts the two in either order, so a back-to-front call put the indexed field into the query-vector slot where its variable does not resolve against the index search's own input. Reject that shape before the shared analysis runs, the way the asymmetric contains() functions already do inside it. The call itself is valid because the metrics are symmetric, so it now falls back to exact evaluation like vector_distance and returns the same rows as the rewritable order. Ext-ref: MB-73348 Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: I4ce84b44b36b5f8188beedf357e2971c9bcadd9d --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.01.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.02.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.03.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.04.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.99.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.adm 11 files changed, 220 insertions(+), 0 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/51/21651/1 diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java index 8fb528c..e7967c8 100644 --- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java +++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java @@ -136,6 +136,14 @@ if (funcExpr.getArguments().size() != 2) { return false; } + // The rewrite is order-sensitive even though the metrics are not: createIndexSearchPlan reads + // argument 1 as the query vector. The shared analysis below accepts the constant and the variable + // in either order, so ann_distance(queryVector, field, ...) would reach plan generation with the + // indexed field in the query-vector slot and fail type resolution there. Such a call is not wrong, + // just not rewritable, so leave it to evaluate exactly the way vector_distance does. + if (funcExpr.getArguments().get(1).getValue().getExpressionTag() == LogicalExpressionTag.VARIABLE) { + return false; + } // Validate arg0 = variable/field reference, arg1 = constant (query vector). // Populates analysisCtx with information needed for index matching. return AccessMethodUtils.analyzeFuncExprArgsForOneConstAndVarAndUpdateAnalysisCtx(funcExpr, analysisCtx, diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml index 1bd0783..bfe36c6 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml @@ -80,6 +80,11 @@ </compilation-unit> </test-case> <test-case FilePath="vector"> + <compilation-unit name="create-index-vtree-swapped-args"> + <output-dir compare="Text">create-index-vtree-swapped-args</output-dir> + </compilation-unit> + </test-case> + <test-case FilePath="vector"> <compilation-unit name="create-index-vtree-null-embedding"> <output-dir compare="Text">create-index-vtree-null-embedding</output-dir> <expected-error>Failed to build the vector index. The sampled records yielded no usable vector</expected-error> diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.01.ddl.sqlpp new file mode 100644 index 0000000..2497935 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.01.ddl.sqlpp @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +// TYPE VTREE is optional when the indexed field carries the VECTOR annotation: no other index type +// accepts one, so the annotation alone determines the type. Everything below is deliberately the +// same shape as create-index-vtree, minus the TYPE clause. +DROP DATAVERSE test IF EXISTS; +CREATE DATAVERSE test; +USE test; +CREATE TYPE MovieType AS { + id: int, + embedding: [double] +}; +CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.02.update.sqlpp new file mode 100644 index 0000000..94517b1 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.02.update.sqlpp @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +// The 4th component is the only thing that varies within each block, so the 3 nearest to the +// origin are ids 1, 2 and 3 regardless of how the two blocks are clustered. +INSERT INTO MovieSmall ([ + {"id": 1, "embedding": [0.0, 0.0, 0.0, 0.1]}, + {"id": 2, "embedding": [0.0, 0.0, 0.0, 0.2]}, + {"id": 3, "embedding": [0.0, 0.0, 0.0, 0.3]}, + {"id": 4, "embedding": [0.0, 0.0, 0.0, 0.4]}, + {"id": 5, "embedding": [0.0, 0.0, 0.0, 0.5]}, + {"id": 6, "embedding": [0.0, 0.0, 0.0, 0.6]}, + {"id": 7, "embedding": [0.0, 0.0, 0.0, 0.7]}, + {"id": 8, "embedding": [0.0, 0.0, 0.0, 0.8]}, + {"id": 9, "embedding": [0.0, 0.0, 0.0, 0.9]}, + {"id": 10, "embedding": [0.0, 0.0, 0.0, 1.0]}, + {"id": 11, "embedding": [0.0, 0.0, 0.0, 1.1]}, + {"id": 12, "embedding": [0.0, 0.0, 0.0, 1.2]}, + {"id": 13, "embedding": [0.0, 0.0, 0.0, 1.3]}, + {"id": 14, "embedding": [0.0, 0.0, 0.0, 1.4]}, + {"id": 15, "embedding": [0.0, 0.0, 0.0, 1.5]}, + {"id": 16, "embedding": [10.0, 10.0, 10.0, 0.1]}, + {"id": 17, "embedding": [10.0, 10.0, 10.0, 0.2]}, + {"id": 18, "embedding": [10.0, 10.0, 10.0, 0.3]}, + {"id": 19, "embedding": [10.0, 10.0, 10.0, 0.4]}, + {"id": 20, "embedding": [10.0, 10.0, 10.0, 0.5]}, + {"id": 21, "embedding": [10.0, 10.0, 10.0, 0.6]}, + {"id": 22, "embedding": [10.0, 10.0, 10.0, 0.7]}, + {"id": 23, "embedding": [10.0, 10.0, 10.0, 0.8]}, + {"id": 24, "embedding": [10.0, 10.0, 10.0, 0.9]}, + {"id": 25, "embedding": [10.0, 10.0, 10.0, 1.0]}, + {"id": 26, "embedding": [10.0, 10.0, 10.0, 1.1]}, + {"id": 27, "embedding": [10.0, 10.0, 10.0, 1.2]}, + {"id": 28, "embedding": [10.0, 10.0, 10.0, 1.3]}, + {"id": 29, "embedding": [10.0, 10.0, 10.0, 1.4]}, + {"id": 30, "embedding": [10.0, 10.0, 10.0, 1.5]} +]); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.03.ddl.sqlpp new file mode 100644 index 0000000..b63fc7a --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.03.ddl.sqlpp @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +// A sample index is required before creating a vector index (quantization training scans it). +ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000}; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.04.ddl.sqlpp new file mode 100644 index 0000000..d186f42 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.04.ddl.sqlpp @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +// No TYPE clause. The VECTOR annotation on `embedding` is what makes this a vector index; the +// INCLUDE/WITH/EXCLUDE UNKNOWN KEY clauses that only a vector index accepts must be honoured too. +CREATE INDEX idx_emb + ON MovieSmall(embedding VECTOR) + TYPE VTREE + WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 2, "train_list_fraction": 1.0, "seed": 42 } + EXCLUDE UNKNOWN KEY; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.query.sqlpp new file mode 100644 index 0000000..1802562 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.query.sqlpp @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +// Field first: the rewritable order, which the index search handles. +SELECT VALUE m.id +FROM MovieSmall m +ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0) +LIMIT 3; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.query.sqlpp new file mode 100644 index 0000000..3a9736e --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.query.sqlpp @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +USE test; + +// Query vector first. The distance is the same -- the metric is symmetric -- but the index rewrite +// cannot take this shape, so the call must fall back to exact evaluation and return the same rows +// rather than failing during plan generation. +SELECT VALUE m.id +FROM MovieSmall m +ORDER BY ann_distance([0.0, 0.0, 0.0, 0.0], m.embedding, "euclidean", 1.0) +LIMIT 3; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.99.ddl.sqlpp new file mode 100644 index 0000000..d240ef2 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.99.ddl.sqlpp @@ -0,0 +1,19 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +DROP DATAVERSE test IF EXISTS; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.adm new file mode 100644 index 0000000..01e79c3 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.05.adm @@ -0,0 +1,3 @@ +1 +2 +3 diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.adm new file mode 100644 index 0000000..01e79c3 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-swapped-args/create-index-vtree-swapped-args.06.adm @@ -0,0 +1,3 @@ +1 +2 +3 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21651?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I4ce84b44b36b5f8188beedf357e2971c9bcadd9d Gerrit-Change-Number: 21651 Gerrit-PatchSet: 1 Gerrit-Owner: Hongyu Shi <[email protected]>
