>From Hongyu Shi <[email protected]>: Hongyu Shi has uploaded a new patch set (#2). ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21384?usp=email )
Change subject: [ASTERIXDB-3783] CLUSTER BY compile-time support and EXPLAIN ...................................................................... [ASTERIXDB-3783] CLUSTER BY compile-time support and EXPLAIN Add compile-time support for the SQL++ CLUSTER BY clause, a fuzzy, similarity-based analog of GROUP BY for vector embeddings. A CLUSTER BY query now parses, validates its WITH options, and compiles into an optimizable logical plan and a complete, buildable Hyracks job, so its optimized physical plan can be printed via EXPLAIN or a query request. The clustering runtime itself is not yet implemented: the placeholder ClusterAssign runtime fails loudly if the job is actually executed. CLUSTER BY desugars into a new blocking ClusterAssign operator that labels each input tuple with cluster_id/centroid/cluster_radius, feeding a stock GROUP BY on cluster_id (plus LISTIFY for CLUSTER AS). Reusing the existing grouping machinery means no optimizer rule needs special-casing. - Grammar: CLUSTER BY / CLUSTER AS / WITH, with CLUSTER as a soft keyword and structural mutual exclusion with GROUP BY. - AST: ClusterbyClause node wired into SelectBlock and the SQL++ visitors. - Rewrite: validates WITH options and derives the CLUSTER AS member list. - Translation: desugars to ClusterAssign -> GROUP BY (+ LISTIFY) and the cluster descriptor record. - Logical/physical: ClusterAssignOperator and tag implemented across all operator visitors; ClusterAssignPOperator contributes a buildable placeholder runtime so the job compiles for EXPLAIN. - Tests: optimizer plan tests (including an APE 33 join example), WITH-option validation negative tests, and a parser round-trip test. Change-Id: I250cf1abe7cfe8dccac75b154856f6d2d771cf28 --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SweepIllegalNonfunctionalFunctions.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/cbo/EstimatedCostComputationVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/visitor/PushdownOperatorVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineAllNtsInSubplanVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/SubplanSpecialFlatteningCheckVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/SqlppExpressionToPlanTranslator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/PlanStagesGenerator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/resource/RequiredCapacityVisitor.java A asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/cluster-by-ape-design1.sqlpp A asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/cluster-by-descriptor-only.sqlpp A asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/cluster-by-listify-members.sqlpp A asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/cluster-by-with-members.sqlpp A asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/cluster-by-ape-design1.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/cluster-by-descriptor-only.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/cluster-by-listify-members.plan A asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/cluster-by-with-members.plan A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/ClusterByQueries.xml A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/negative/missing-numclusters/test.1.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/negative/nonpositive-numclusters/test.1.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/negative/unknown-distance/test.1.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/negative/unknown-option/test.1.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cluster-by/negative/unsupported-algorithm/test.1.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/base/Clause.java A asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/clause/ClusterbyClause.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/clause/SelectBlock.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/SqlppQueryRewriter.java A asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/SqlppClusterByVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/SqlppGatherFunctionCallsVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/SqlppInlineUdfsVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/CheckDatasetOnlyResolutionVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/CheckSql92AggregateVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/CheckSubqueryVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/DeepCopyVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/FreeVariableVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/SqlppAstPrintVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/SqlppCloneAndSubstituteVariablesVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/SqlppFormatPrintVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/base/AbstractSqlppAstVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/base/AbstractSqlppContainsExpressionVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/base/AbstractSqlppExpressionScopingVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/base/AbstractSqlppSimpleExpressionVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/visitor/base/ISqlppVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj A asterixdb/asterix-lang-sqlpp/src/test/java/org/apache/asterix/lang/sqlpp/parser/ClusterByParserTest.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/LogicalOperatorTag.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/PhysicalOperatorTag.java A hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/ClusterAssignOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/CardinalityInferenceVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/FDsAndEquivClassesVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismVariableMappingVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalOperatorDeepCopyWithNewVariablesVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalPropertiesVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/PrimaryKeyVariablesVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/ProducedVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SchemaVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java A hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/ClusterAssignPOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/prettyprint/LogicalOperatorPrettyPrintVisitorJson.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/ILogicalOperatorVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/visitors/LogicalExpressionReferenceTransformVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/utils/LogicalOperatorDotVisitor.java M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetAlgebricksPhysicalOperatorsRule.java M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/SetMemoryRequirementsRule.java M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/subplan/ReplaceNtsWithSubplanInputOperatorVisitor.java A hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/cluster/ClusterAssignRuntimeFactory.java 71 files changed, 1,787 insertions(+), 2 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/84/21384/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21384?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newpatchset Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I250cf1abe7cfe8dccac75b154856f6d2d771cf28 Gerrit-Change-Number: 21384 Gerrit-PatchSet: 2 Gerrit-Owner: Hongyu Shi <[email protected]>
