>From Hongyu Shi <[email protected]>:
Attention is currently required from: Hongyu Shi.
Hello Anon. E. Moose #1000171, Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21652?usp=email
to look at the new patch set (#4).
The following approvals got outdated and were removed:
Integration-Tests+1 by Jenkins, Verified-1 by Jenkins
Change subject: [ASTERIXDB-3817][COMP][MTD] Infer vector index knobs from
cardinality
......................................................................
[ASTERIXDB-3817][COMP][MTD] Infer vector index knobs from cardinality
- user model changes: yes
- storage format changes: no
- interface changes: yes
Details:
A vector index needs two size knobs the user is not expected to supply:
the leaf cluster count and the training list the k-means sees. Neither
was derived from the data. num_clusters defaulted to sqrt(D), which on a
small collection leaves too few rows per cluster to train on, and it
reached the index record only when the user named it, so what a build
actually used was unrecoverable afterwards. The two knobs were not
correlated with each other either.
VTreeParamsInference derives both from the per-partition cardinality D,
which is the unit a build works in because each partition trains its own
tree. num_clusters becomes sqrt(D) and falls back to ceil(D/100) on a
small collection, while the advised train list holds 50 records per
cluster. A user-supplied value is checked against the same D, so what
the build cannot work with is an error and what is merely unusual warns.
Both report through the existing vector index creation error code.
Inference runs on the DDL path and the result is persisted, so the index
record states what the build used. That also moves the existing
missing-ANALYZE and empty-collection checks from job build to compile
time.
Ext-ref: MB-73665, MB-73194
Co-Authored-By: Claude Opus 5 <[email protected]>
Change-Id: I5d90fdd079f0931edaeb46a26ac2c717e07a043c
---
M
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.01.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.02.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.03.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.04.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.99.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.01.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.02.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.03.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.04.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.99.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.01.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.02.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.03.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.04.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.99.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.01.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.02.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.03.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.04.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.99.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.adm
A
asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.adm
M
asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
A
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeParamsInference.java
M
asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
A
asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/utils/VTreeParamsInferenceTest.java
M
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
32 files changed, 1,209 insertions(+), 51 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/52/21652/4
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21652?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5d90fdd079f0931edaeb46a26ac2c717e07a043c
Gerrit-Change-Number: 21652
Gerrit-PatchSet: 4
Gerrit-Owner: Hongyu Shi <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Attention: Hongyu Shi <[email protected]>
