>From Shahrzad Shirazi <[email protected]>: Shahrzad Shirazi has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21984?usp=email )
Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: label rows in Lloyd loop ...................................................................... [NO ISSUE][COMP][RT] CLUSTER BY: label rows in Lloyd loop - user model changes: no - storage format changes: no - interface changes: no Details: CLUSTER BY labelled its rows with a join: the final centroids were sorted, listified and broadcast, then nested-loop joined onto a materialized copy of every input row and placed with nearest-centroid. That copy was the largest temporary file of the query, and the join is what required it. The loop that stores the rows now keeps the columns the clusters are built from beside each vector, position-aligned, and the Lloyd loop labels every row itself after its last iteration: the position of the nearest centroid in the final set ordered by value, ties to the first, as before. The labelling join, the centroid listify, nearest-centroid and the materialized replicate output are removed. Results are unchanged; the input is written to disk once. - KMeansStageOperator: row variables (typed on the operator) and an optional assignment-centroid variable; LLOYD_LOOP emits them with the cluster id. - Payload column beside the vector run file, handed from the oversampling loop to the Lloyd loop with the vectors; the Lloyd loop's drained input carries only the vector. - KMEANS_STAGE and CLUSTER_BY are filter-pushdown scopes. - Variable substitution updates the stage's type environment. - A row with no finite distance to any centroid warns. - nearest-centroid is removed; nothing uses it. - Expected plans for cluster-by regenerated. Ext-ref: MB-74142 Generated-by: Claude Opus 5.5 <[email protected]> Change-Id: Iddc985b91e7f3cfb043a29f6ad48e6e4f39b2fe2 --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/AbstractKMeansLoopPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/AbstractKMeansStagePOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansOversampleLoopPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/AsterixInlineVariablesRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/RewriteClusterByToKMeansRule.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/PushdownContext.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/SqlppExpressionToPlanTranslator.java M asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/kmeans-syntax-validate.sqlpp M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-decor-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-init-random-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-join-members-projected.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-limit-input-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-syntax-full-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-syntax-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-filtered-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-members-twice-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-members-validate.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-positional-validate.plan M asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan M asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan M asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan M asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan M asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan M asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java D asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/NearestCentroidDescriptor.java D asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/NearestCentroidScalarEvaluator.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/functions/FunctionCollection.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansCostControllerOperatorDescriptor.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLloydControllerOperatorDescriptor.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLoopIO.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansVectorCodec.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/LoopControlState.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/KMeansStageOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/CardinalityInferenceVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalOperatorDeepCopyWithNewVariablesVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/ProducedVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SchemaVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java 57 files changed, 3,100 insertions(+), 4,138 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/84/21984/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21984?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Iddc985b91e7f3cfb043a29f6ad48e6e4f39b2fe2 Gerrit-Change-Number: 21984 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi <[email protected]>
