>From Hongyu Shi <[email protected]>:
Attention is currently required from: Hongyu Shi.
Hello Ali Alsuliman, Anon. E. Moose #1000171, Ian Maxon, Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21504?usp=email
to look at the new patch set (#3).
The following approvals got outdated and were removed:
Code-Review+1 by Hongyu Shi, Contrib+1 by Anon. E. Moose #1000171,
Integration-Tests+1 by Jenkins, Verified+1 by Jenkins
Change subject: [NO ISSUE][RT] VTREE: remove repeated work in vector index build
......................................................................
[NO ISSUE][RT] VTREE: remove repeated work in vector index build
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
Centroid training for a vector index decoded every training vector
from ADM on every pass, measured each point against every center drawn
so far in each k-means|| round, kept a second copy of every
oversampled candidate, and made two passes per Lloyd iteration. Its
heap held two copies of the candidate set, about 10 x num_clusters
vectors per partition, so a large num_clusters risked exhausting the
heap, and a build allocated gigabytes of short-lived objects.
- Decode the training sample once into raw doubles.
- Keep each point's D(x) across k-means|| rounds and lower it only
against the previous round's draws; the sample pass reads it and
decodes only the points it draws.
- Assign and accumulate in one pass per Lloyd iteration.
- Merge duplicate candidates by measuring only those whose projections
on one fixed direction lie within the merge distance, instead of a
pairwise distance scan.
- Drop state nothing reads: the per-row level-0 assignments and the
parent-child maps.
The tree built is unchanged. Training holds one copy of the
candidates instead of two and takes about half the time.
Generated-by: Claude Opus 5.5 <[email protected]>
Change-Id: I7b3e5c1a9d24f068e3b7a105c48d2f9e63104b7a
---
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/KMeansUtils.java
2 files changed, 269 insertions(+), 184 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/04/21504/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21504?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I7b3e5c1a9d24f068e3b7a105c48d2f9e63104b7a
Gerrit-Change-Number: 21504
Gerrit-PatchSet: 3
Gerrit-Owner: Hongyu Shi <[email protected]>
Gerrit-Reviewer: Ali Alsuliman <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Hongyu Shi <[email protected]>
Gerrit-Reviewer: Ian Maxon <[email protected]>
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Attention: Hongyu Shi <[email protected]>
