>From Hongyu Shi <[email protected]>: Hongyu Shi has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21287?usp=email )
Change subject: [ASTERIXDB-3771] ANN query optimizer rule for top-k ANN vector search ...................................................................... [ASTERIXDB-3771] ANN query optimizer rule for top-k ANN vector search - user model changes: yes - storage format changes: no - interface changes: yes Add optimizer rules and physical operators for approximate nearest neighbor (ANN) top-k queries using VTree vector indexes. Components: - IntroduceTopKAccessMethodRule: matches LIMIT k -> ORDER BY ann_distance(...) pattern and transforms to vector index access - VectorIndexAccessMethod: implements IAccessMethod for vector index selection - VectorSearchPOperator: physical operator for vector search execution - PushFilterIntoVectorSearchRule: pushes INCLUDE field filters into vector search - ANNDistanceDescriptor: runtime evaluator for ann_distance function - VectorDistanceFunctionFactory: factory for distance metric implementations - Metadata/DDL integration for vector index query path Change-Id: Iaa502b57915c75921cd5aa5c62fc0e24261bb0f6 --- A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorIndexFilterSchema.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorIndexFilterTypeEnvironment.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/CheckFullParallelSortRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMethodUtils.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorJobGenParams.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/QueryIndexRewriter.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/io/PersistedResourceRegistry.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/JobResultCallback.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/base/Statement.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateVectorIndexStatement.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/CommonFunctionMapUtil.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/FunctionUtil.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj M asterixdb/asterix-metadata/pom.xml M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/BTreeResourceFactoryProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataManagerUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/KeyFieldTypeUtil.java R asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/common/AOrderedListVectorBinaryAccessor.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/ANNDistanceDescriptor.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/functions/FunctionCollection.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VCTreeBulkLoaderAndGroupingOperatorDescriptor.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VCTreeStaticStructureCreatorOperatorDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/utils/VectorDistanceFunctionFactory.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/OperatorAnnotations.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/PhysicalOperatorTag.java D hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java 50 files changed, 4,713 insertions(+), 234 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/87/21287/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21287?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Iaa502b57915c75921cd5aa5c62fc0e24261bb0f6 Gerrit-Change-Number: 21287 Gerrit-PatchSet: 1 Gerrit-Owner: Hongyu Shi <[email protected]>
