>From Shahrzad Shirazi <[email protected]>:
Attention is currently required from: Hongyu Shi, Shahrzad Shirazi.
Hello Anon. E. Moose #1000171, Hongyu Shi, Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21984?usp=email
to look at the new patch set (#3).
The following approvals got outdated and were removed:
Contrib-2 by Anon. E. Moose #1000171
Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: label rows in Lloyd loop
......................................................................
[NO ISSUE][COMP][RT] CLUSTER BY: label rows in Lloyd loop
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
CLUSTER BY labelled its rows with a join: the final centroids
were sorted, listified and broadcast, then nested-loop joined
onto a materialized copy of every input row and placed with
nearest-centroid. That copy was the largest temporary file of
the query, and the join is what required it.
The loop that stores the rows now keeps the columns the
clusters are built from beside each vector, position-aligned,
and the Lloyd loop labels every row itself after its last
iteration: the position of the nearest centroid in the final
set ordered by value, ties to the first, as before. The
labelling join, the centroid listify, nearest-centroid and the
materialized replicate output are removed. Results are
unchanged; the input is written to disk once.
- KMeansStageOperator: row variables (typed on the operator)
and an optional assignment-centroid variable; LLOYD_LOOP
emits them with the cluster id.
- Payload column beside the vector run file, handed from the
oversampling loop to the Lloyd loop with the vectors; the
Lloyd loop's drained input carries only the vector.
- KMEANS_STAGE and CLUSTER_BY are filter-pushdown scopes.
- Variable substitution updates the stage's type environment.
- A row with no finite distance to any centroid warns.
- nearest-centroid is removed; nothing uses it.
- Expected plans for cluster-by regenerated.
Ext-ref: MB-74142
Generated-by: Claude Opus 5.5 <[email protected]>
Change-Id: Iddc985b91e7f3cfb043a29f6ad48e6e4f39b2fe2
---
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/AbstractKMeansLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/AbstractKMeansStagePOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansOversampleLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/AsterixInlineVariablesRule.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/RewriteClusterByToKMeansRule.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/pushdown/PushdownContext.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/SqlppExpressionToPlanTranslator.java
M
asterixdb/asterix-app/src/test/resources/optimizerts/queries/cluster-by/kmeans-syntax-validate.sqlpp
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-decor-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-init-random-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-join-members-projected.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-limit-input-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-syntax-full-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-syntax-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-filtered-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-members-twice-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-members-validate.plan
M
asterixdb/asterix-app/src/test/resources/optimizerts/results/cluster-by/kmeans-unnest-positional-validate.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_cbo/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_full_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.10.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.11.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.12.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.8.plan
M
asterixdb/asterix-app/src/test/resources/runtimets/results_less_parallelism/cluster-by/kmeans-num-iterations/kmeans-num-iterations.9.plan
M
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java
D
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/NearestCentroidDescriptor.java
D
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/NearestCentroidScalarEvaluator.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/functions/FunctionCollection.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansCostControllerOperatorDescriptor.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLloydControllerOperatorDescriptor.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLoopIO.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansVectorCodec.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/LoopControlState.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/KMeansStageOperator.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/CardinalityInferenceVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/IsomorphismOperatorVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalOperatorDeepCopyWithNewVariablesVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/ProducedVariableVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SchemaVariableVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/UsedVariableVisitor.java
57 files changed, 3,029 insertions(+), 4,140 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/84/21984/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21984?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Iddc985b91e7f3cfb043a29f6ad48e6e4f39b2fe2
Gerrit-Change-Number: 21984
Gerrit-PatchSet: 3
Gerrit-Owner: Shahrzad Shirazi <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Hongyu Shi <[email protected]>
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Attention: Shahrzad Shirazi <[email protected]>
Gerrit-Attention: Hongyu Shi <[email protected]>