[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Hongyu Shi : Attention is currently required from: Shahrzad Shirazi. Hongyu Shi has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Code-Review+2 (1 comment) Patchset: PS1: I went through this carefully and it looks correct to me and indeed actually an potential resource optimization in my backlog (never got the chance to do but it's good to be caught by perf tests ...). note: Under kmeans_parallel, both loops used to write the same per-partition vector file. Now the oversampling loop keeps its file and the Lloyd loop reads it, then deletes it after its last iteration. That saves a second decode pass over the input and a second input-sized write per partition, and halves the peak disk space for vectors. The per-iteration reads are unchanged and results stay identical (should be) since the Lloyd loop reads exactly the vectors it would have stored. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Comment-Date: Sat, 03 Oct 2026 06:45:46 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Shahrzad Shirazi : Attention is currently required from: Shahrzad Shirazi. Shahrzad Shirazi has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Removed Integration-Tests-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: deleteVote Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Shahrzad Shirazi :
Shahrzad Shirazi has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email )
Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
..
[NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Under kmeans_parallel init, the oversampling loop and the Lloyd
loop each wrote every input vector to a run file of their own,
so the input reached disk twice before the labelling copy.
Both loops run on the same locations, and each partition's file
set covers the input once, which is all the Lloyd update needs.
The Lloyd loop now reads the oversampling loop's vector file
instead of writing its own:
- KMeansStageOperator names the shared store; the expansion
rule sets it on both loops, so they agree by construction.
- The oversampling loop keeps its file on success; the Lloyd
loop drains its own vector input unread, reads that file and
deletes it after its last iteration.
- The random init is unchanged: there is no oversampling loop,
and the Lloyd loop keeps its own store.
Results and plans are unchanged; one input-sized temporary
file per partition is gone.
Ext-ref: MB-74142
Generated-by: Claude Opus 5.5
Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985
Tested-by: Jenkins
Integration-Tests: Hongyu Shi
Reviewed-by: Hongyu Shi
---
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansOversampleLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/RewriteClusterByToKMeansRule.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansCostControllerOperatorDescriptor.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLloydControllerOperatorDescriptor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/KMeansStageOperator.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalOperatorDeepCopyWithNewVariablesVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
9 files changed, 81 insertions(+), 21 deletions(-)
Approvals:
Jenkins: Verified
Hongyu Shi: Looks good to me, approved; Verified
Objections:
Anon. E. Moose #1000171: Violations found
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
index 76d521f..bf24563 100644
---
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
+++
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
@@ -56,12 +56,15 @@
ILogicalOperator src0, ILogicalOperator src1) throws
AlgebricksException {
JobSpecification spec = builder.getJobSpec();
// Unique + stable per loop instance, and distinct from any
oversampling loop's key in the same job.
-String loopKey = "kmeansLloydLoop#" + kop.getCandidateVariable();
+String loopKey = "kmeansLloydLoop#" +
kop.getCandidateVariable().getId();
+// The oversampling loop's vector store under k-means||; null under
the random init.
+String sharedVectorsKey = kop.getVectorStoreVariable() == null ? null
+:
KMeansOversampleLoopPOperator.loopKeyOf(kop.getVectorStoreVariable());
int participants = clusterLocations.length;
KMeansLloydControllerOperatorDescriptor op1 = new
KMeansLloydControllerOperatorDescriptor(spec, centroidRecDesc,
-KMeansLoopIO.PARTIAL_RD, loopKey, vectorColumn,
centroidColumn, kop.getLoopRounds(), kop.getTopCount(),
-framesLimit(), kop.getDimension(), metricOf(kop));
+KMeansLoopIO.PARTIAL_RD, loopKey, sharedVectorsKey,
vectorColumn, centroidColumn, kop.getLoopRounds(),
+kop.getTopCount(), framesLimit(), kop.getDimension(),
metricOf(kop));
contributeOpDesc(builder, op, op1);
builder.contributeGraphEdge(src0, 0, op, 0);
builder.contributeGraphEdge(src1, 0, op,
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Hongyu Shi : Attention is currently required from: Shahrzad Shirazi. Hongyu Shi has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Integration-Tests+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Comment-Date: Sat, 03 Oct 2026 06:47:58 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Hongyu Shi : Attention is currently required from: Shahrzad Shirazi. Hongyu Shi has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: (1 comment) Patchset: PS1: this is supposed to be caused by other changes. but not sure what made -2 from cb-jenkins. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Comment-Date: Sat, 03 Oct 2026 06:47:29 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Hongyu Shi : Attention is currently required from: Shahrzad Shirazi. Hongyu Shi has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Integration-Tests+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Comment-Date: Sat, 03 Oct 2026 06:48:11 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Hongyu Shi : Attention is currently required from: Shahrzad Shirazi. Hongyu Shi has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: -Integration-Tests -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Comment-Date: Sat, 03 Oct 2026 06:48:07 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Jenkins : Attention is currently required from: Hongyu Shi, Shahrzad Shirazi. Jenkins has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/8086/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Reviewer: Jenkins Gerrit-Attention: Shahrzad Shirazi Gerrit-Attention: Hongyu Shi Gerrit-Comment-Date: Sat, 03 Oct 2026 03:15:18 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
Attention is currently required from: Hongyu Shi, Shahrzad Shirazi. Anon. E. Moose #1000171 has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/mBcLBT3LCb17VnNt9 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi Gerrit-Attention: Shahrzad Shirazi Gerrit-Attention: Hongyu Shi Gerrit-Comment-Date: Sat, 03 Oct 2026 01:36:12 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
Anon. E. Moose #1000171 has posted comments on this change by Shahrzad Shirazi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email ) Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store .. Patch Set 1: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/Zd8yQjQy73FbiMsW7 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14 Gerrit-Change-Number: 21985 Gerrit-PatchSet: 1 Gerrit-Owner: Shahrzad Shirazi Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Comment-Date: Fri, 02 Oct 2026 23:53:40 + Gerrit-HasComments: No Gerrit-Has-Labels: No
[M] Change in asterixdb[master]: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
>From Shahrzad Shirazi :
Shahrzad Shirazi has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21985?usp=email )
Change subject: [NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
..
[NO ISSUE][COMP][RT] CLUSTER BY: share the k-means vector store
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Under kmeans_parallel init, the oversampling loop and the Lloyd
loop each wrote every input vector to a run file of their own,
so the input reached disk twice before the labelling copy.
Both loops run on the same locations, and each partition's file
set covers the input once, which is all the Lloyd update needs.
The Lloyd loop now reads the oversampling loop's vector file
instead of writing its own:
- KMeansStageOperator names the shared store; the expansion
rule sets it on both loops, so they agree by construction.
- The oversampling loop keeps its file on success; the Lloyd
loop drains its own vector input unread, reads that file and
deletes it after its last iteration.
- The random init is unchanged: there is no oversampling loop,
and the Lloyd loop keeps its own store.
Results and plans are unchanged; one input-sized temporary
file per partition is gone.
Ext-ref: MB-74142
Generated-by: Claude Opus 5.5
Change-Id: I2d34e7e35ae889edbce294e96889d187186c9d14
---
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansOversampleLoopPOperator.java
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/RewriteClusterByToKMeansRule.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansCostControllerOperatorDescriptor.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/kmeans/KMeansLloydControllerOperatorDescriptor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/KMeansStageOperator.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/LogicalOperatorDeepCopyWithNewVariablesVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/OperatorDeepCopyVisitor.java
M
hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/logical/visitors/SubstituteVariableVisitor.java
9 files changed, 81 insertions(+), 21 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/85/21985/1
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
index 76d521f..bf24563 100644
---
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
+++
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansLloydLoopPOperator.java
@@ -56,12 +56,15 @@
ILogicalOperator src0, ILogicalOperator src1) throws
AlgebricksException {
JobSpecification spec = builder.getJobSpec();
// Unique + stable per loop instance, and distinct from any
oversampling loop's key in the same job.
-String loopKey = "kmeansLloydLoop#" + kop.getCandidateVariable();
+String loopKey = "kmeansLloydLoop#" +
kop.getCandidateVariable().getId();
+// The oversampling loop's vector store under k-means||; null under
the random init.
+String sharedVectorsKey = kop.getVectorStoreVariable() == null ? null
+:
KMeansOversampleLoopPOperator.loopKeyOf(kop.getVectorStoreVariable());
int participants = clusterLocations.length;
KMeansLloydControllerOperatorDescriptor op1 = new
KMeansLloydControllerOperatorDescriptor(spec, centroidRecDesc,
-KMeansLoopIO.PARTIAL_RD, loopKey, vectorColumn,
centroidColumn, kop.getLoopRounds(), kop.getTopCount(),
-framesLimit(), kop.getDimension(), metricOf(kop));
+KMeansLoopIO.PARTIAL_RD, loopKey, sharedVectorsKey,
vectorColumn, centroidColumn, kop.getLoopRounds(),
+kop.getTopCount(), framesLimit(), kop.getDimension(),
metricOf(kop));
contributeOpDesc(builder, op, op1);
builder.contributeGraphEdge(src0, 0, op, 0);
builder.contributeGraphEdge(src1, 0, op, 1);
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/KMeansOversampleLoopPOperator.java
