This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new f8acb110ff [#13376] feat(clickhouse): preserve legacy Annoy and
USearch index metadata (#13432)
f8acb110ff is described below
commit f8acb110ff3f6b2df6c55fde087fdb72e9d39470
Author: Thunderkeg <[email protected]>
AuthorDate: Sun Oct 11 20:36:16 2026 +0800
[#13376] feat(clickhouse): preserve legacy Annoy and USearch index metadata
(#13432)
### What changes were proposed in this pull request?
Preserve legacy ClickHouse Annoy and USearch index metadata when the
catalog loads tables. Parse the documented optional parameters, retain
the complete reported type expression, and expose recognized values
through Gravitino's index properties.
Annoy parameters are exposed as `annoy_distance_function` and
`annoy_trees`; USearch parameters are exposed as
`usearch_distance_function` and `usearch_scalar_kind`. The original
expression remains available in `clickhouse_type_full`. Empty parameter
lists and bare legacy type names are preserved without a parsing
warning; unrecognized expressions remain available in
`clickhouse_type_full` and produce a warning.
Treat both legacy types as metadata-only. Gravitino rejects CREATE
TABLE, ALTER TABLE ADD INDEX, and table recreation requests that use
them, and does not translate them to `vector_similarity`.
### Why are the changes needed?
ClickHouse 24.x can report legacy Annoy and USearch index definitions
through `system.data_skipping_indices`. The catalog needs to preserve
those definitions and their documented parameters when loading tables,
without implying that the legacy indexes support writes or vector
search.
Fix: #13376
### Does this PR introduce any user-facing change?
Yes. It adds the `DATA_SKIPPING_ANNOY` and `DATA_SKIPPING_USEARCH` index
types and the read-only properties `annoy_distance_function`,
`annoy_trees`, `usearch_distance_function`, `usearch_scalar_kind`, and
`clickhouse_type_full`. Clients must understand the new enum values to
expose these indexes.
### How was this patch tested?
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
'org.apache.gravitino.catalog.clickhouse.operations.TestClickHouseTableOperationsUnit'
-PskipITs` — 89 tests passed, including documented parameter forms,
malformed parameters, warning behavior, and raw-expression preservation.
- `TestJsonUtils` — 13 tests passed.
- Focused Python serde tests — 57 tests passed.
- `CatalogClickHouseIT.testLoadLegacyUsearchIndexMetadata` and
`CatalogClickHouseIT.testLoadLegacyAnnoyIndexMetadata` — 2 tests passed
using ClickHouse 23.8.16.16 and 24.8.14.39 fixtures.
- `GRAVITINO_CLICKHOUSE_TEST_IMAGE=clickhouse:25.8.14.17 ./gradlew
:catalogs-contrib:catalog-jdbc-clickhouse:test --tests
'org.apache.gravitino.catalog.clickhouse.integration.test.CatalogClickHouseClusterIT'
-PskipDockerTests=false` — 21 tests passed.
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:build -x test -x
rat` and `./gradlew compileJava -x test` passed; Spotless, the OpenAPI
documentation build, Black, and `git diff --check` passed.
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.../org/apache/gravitino/rel/indexes/Index.java | 6 +
.../catalog/clickhouse/ClickHouseConstants.java | 21 +
.../operations/ClickHouseTableOperations.java | 225 ++++++++++-
.../integration/test/CatalogClickHouseIT.java | 226 +++++++++++
.../TestClickHouseTableOperationsUnit.java | 433 ++++++++++++++++++++-
.../gravitino/api/rel/indexes/index.py | 8 +
.../dto/rel/indexes/json_serdes/index_serdes.py | 9 +-
.../tests/unittests/dto/rel/test_index_serdes.py | 60 +++
.../org/apache/gravitino/json/TestJsonUtils.java | 14 +
docs/jdbc-clickhouse-catalog.md | 10 +-
docs/open-api/indexes.yaml | 2 +
11 files changed, 996 insertions(+), 18 deletions(-)
diff --git a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
index e139ac9c69..dbfdbe0195 100644
--- a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
+++ b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
@@ -149,5 +149,11 @@ public interface Index {
* Vector similarity data skipping index for approximate nearest-neighbor
search in ClickHouse.
*/
DATA_SKIPPING_VECTOR_SIMILARITY,
+
+ /** Legacy Annoy data-skipping index metadata in ClickHouse. */
+ DATA_SKIPPING_ANNOY,
+
+ /** Legacy USearch data-skipping index metadata in ClickHouse. */
+ DATA_SKIPPING_USEARCH,
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
index 6f40eedf6f..6263ac27c8 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
@@ -106,6 +106,27 @@ public class ClickHouseConstants {
public static final String HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION =
"hnsw_candidate_list_size_for_construction";
+ /** The name of the legacy Annoy data-skipping index type in ClickHouse. */
+ public static final String DATA_SKIPPING_ANNOY = "annoy";
+
+ /** The name of the legacy USearch data-skipping index type in ClickHouse.
*/
+ public static final String DATA_SKIPPING_USEARCH = "usearch";
+
+ /** Property key for the number of trees in a legacy Annoy index. */
+ public static final String ANNOY_TREES = "annoy_trees";
+
+ /** Property key for the distance function in a legacy Annoy index. */
+ public static final String ANNOY_DISTANCE_FUNCTION =
"annoy_distance_function";
+
+ /** Property key for the distance function in a legacy USearch index. */
+ public static final String USEARCH_DISTANCE_FUNCTION =
"usearch_distance_function";
+
+ /** Property key for the scalar kind in a legacy USearch index. */
+ public static final String USEARCH_SCALAR_KIND = "usearch_scalar_kind";
+
+ /** Property key for ClickHouse's complete index type expression. */
+ public static final String CLICKHOUSE_TYPE_FULL = "clickhouse_type_full";
+
/** Property key for bloom filter size in ngrambf_v1 and tokenbf_v1 index
properties. */
public static final String BLOOM_FILTER_SIZE = "bloom_filter_size";
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index bb07965db0..5570da8825 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -18,12 +18,17 @@
*/
package org.apache.gravitino.catalog.clickhouse.operations;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.ANNOY_DISTANCE_FUNCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.ANNOY_TREES;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.BLOOM_FILTER_SIZE;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.CLICKHOUSE_TYPE_FULL;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_ANNOY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_BLOOM_FILTER;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_MINMAX_VALUE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_NGRAMBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_SET;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_TOKENBFV1;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_USEARCH;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_VECTOR_SIMILARITY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.GRANULARITY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.HASH_FUNCTIONS;
@@ -32,6 +37,8 @@ import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexC
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.NGRAM_SIZE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.RANDOM_SEED;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.SET_MAX_VALUES;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_DISTANCE_FUNCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_SCALAR_KIND;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_DIMENSIONS;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_DISTANCE_FUNCTION;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_QUANTIZATION;
@@ -356,6 +363,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
Transform[] partitioning,
Distribution distribution,
Index[] indexes) {
+ rejectLegacyAnnUsearchIndexes(tableName, indexes);
throw new UnsupportedOperationException(
"generateCreateTableSql with out sortOrders in clickhouse is not
supported");
}
@@ -371,6 +379,8 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
Index[] indexes,
SortOrder[] sortOrders) {
+ rejectLegacyAnnUsearchIndexes(tableName, indexes);
+
Preconditions.checkArgument(
Distributions.isNone(distribution), "ClickHouse does not support
distribution");
@@ -1326,6 +1336,9 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
private String addIndexDefinition(JdbcTable table, TableChange.AddIndex
addIndex) {
Preconditions.checkArgument(
StringUtils.isNotBlank(addIndex.getName()), "Index name is required");
+ if (isLegacyAnnUsearchIndex(addIndex.getType())) {
+ throw legacyAnnUsearchWriteException(table.name(), addIndex.getName(),
addIndex.getType());
+ }
Preconditions.checkArgument(
ArrayUtils.isNotEmpty(addIndex.getFieldNames()), "Index field names
are required");
@@ -2049,21 +2062,49 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
continue;
}
+ if (isLegacyAnnUsearchIndex(indexType) &&
StringUtils.isBlank(parameterSource)) {
+ parameterSource = type;
+ parameterSourceName = "legacy type";
+ }
+
Map<String, String> parameterProperties = Collections.emptyMap();
String[][] fields;
try {
fields = parseIndexFields(expression);
} catch (IllegalArgumentException ignored) {
- LOG.warn(
- "Skip unsupported data skipping index {} for {}.{} with type
{} because its "
- + "expression cannot be represented as index field names",
- name,
- databaseName,
- tableName,
- type);
+ if (isLegacyAnnUsearchIndex(indexType)) {
+ LOG.warn(
+ "Cannot expose legacy ClickHouse index {} of type {} on
{}.{} because its "
+ + "expression '{}' cannot be represented as Gravitino
index field names; "
+ + "the index is omitted from the loaded metadata",
+ name,
+ type,
+ databaseName,
+ tableName,
+ expression);
+ } else {
+ LOG.warn(
+ "Skip unsupported data skipping index {} for {}.{} with type
{} because its "
+ + "expression cannot be represented as index field
names",
+ name,
+ databaseName,
+ tableName,
+ type);
+ }
continue;
}
if (ArrayUtils.isEmpty(fields)) {
+ if (isLegacyAnnUsearchIndex(indexType)) {
+ LOG.warn(
+ "Cannot expose legacy ClickHouse index {} of type {} on
{}.{} because its "
+ + "expression '{}' contains no Gravitino index fields;
the index is "
+ + "omitted from the loaded metadata",
+ name,
+ type,
+ databaseName,
+ tableName,
+ expression);
+ }
continue;
}
@@ -2081,6 +2122,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
if (indexType == Index.IndexType.DATA_SKIPPING_SET
|| isParameterizedBloomFilterIndex(indexType)
+ || isLegacyAnnUsearchIndex(indexType)
|| indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
{
try {
parameterProperties =
@@ -2117,7 +2159,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
? DEFAULT_VECTOR_SIMILARITY_GRANULARITY
: DEFAULT_INDEX_GRANULARITY;
- if (granularity != defaultGranularity) {
+ if (granularity != defaultGranularity ||
isLegacyAnnUsearchIndex(indexType)) {
properties.put(GRANULARITY, String.valueOf(granularity));
}
if (!includesTypeFull
@@ -2252,6 +2294,9 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_TOKENBFV1:
return parseBloomFilterPropertiesForQuery(
indexType, parameterSource, indexName, allowBareLegacyType);
+ case DATA_SKIPPING_ANNOY:
+ case DATA_SKIPPING_USEARCH:
+ return parseLegacyAnnUsearchProperties(indexType, parameterSource,
indexName);
case DATA_SKIPPING_VECTOR_SIMILARITY:
if (allowBareLegacyType && !StringUtils.contains(parameterSource,
"(")) {
return Collections.emptyMap();
@@ -2539,11 +2584,157 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
|| indexType == Index.IndexType.DATA_SKIPPING_TOKENBFV1;
}
+ private static Map<String, String> parseLegacyAnnUsearchProperties(
+ Index.IndexType indexType, String typeFull, String indexName) {
+ String expectedType = legacyAnnUsearchTypeName(indexType);
+ String rawTypeFull = StringUtils.defaultString(typeFull);
+ String normalizedTypeFull = rawTypeFull.trim();
+ Map<String, String> properties = new HashMap<>();
+ if (StringUtils.isNotEmpty(rawTypeFull)) {
+ properties.put(CLICKHOUSE_TYPE_FULL, rawTypeFull);
+ }
+
+ int paramsStart = normalizedTypeFull.indexOf('(');
+ if (paramsStart < 0 && StringUtils.equalsIgnoreCase(expectedType,
normalizedTypeFull)) {
+ return Map.copyOf(properties);
+ }
+ int paramsEnd = normalizedTypeFull.lastIndexOf(')');
+ if (paramsStart <= 0
+ || paramsEnd != normalizedTypeFull.length() - 1
+ || !StringUtils.equalsIgnoreCase(
+ expectedType, normalizedTypeFull.substring(0,
paramsStart).trim())) {
+ LOG.warn(
+ "Could not parse legacy ClickHouse type expression '{}' for index
'{}' of type {}; "
+ + "preserving it in '{}'",
+ rawTypeFull,
+ indexName,
+ expectedType,
+ CLICKHOUSE_TYPE_FULL);
+ return Map.copyOf(properties);
+ }
+
+ // ClickHouse documents both parameter lists as optional, so an empty list
is valid metadata.
+ String parameterText = normalizedTypeFull.substring(paramsStart + 1,
paramsEnd);
+ if (StringUtils.isBlank(parameterText)) {
+ return Map.copyOf(properties);
+ }
+ // These parameters are fixed identifiers or an integer, so commas delimit
the argument list.
+ String[] parameters = parameterText.split(",", -1);
+
+ if (indexType == Index.IndexType.DATA_SKIPPING_ANNOY) {
+ if (parameters.length == 1) {
+ String parameter = unquoteMetadataParameter(parameters[0].trim());
+ if (isUnsignedInteger(parameter)) {
+ // Preserve the one-argument form that predates the optional
distance parameter.
+ properties.put(ANNOY_TREES, parameter);
+ } else if (isLegacyAnnUsearchDistanceFunction(parameter)) {
+ properties.put(ANNOY_DISTANCE_FUNCTION, parameter);
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ } else if (parameters.length == 2) {
+ String distanceFunction =
unquoteMetadataParameter(parameters[0].trim());
+ String trees = unquoteMetadataParameter(parameters[1].trim());
+ if (isLegacyAnnUsearchDistanceFunction(distanceFunction) &&
isUnsignedInteger(trees)) {
+ properties.put(ANNOY_DISTANCE_FUNCTION, distanceFunction);
+ properties.put(ANNOY_TREES, trees);
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ } else if (parameters.length == 1) {
+ String distanceFunction = unquoteMetadataParameter(parameters[0].trim());
+ if (isLegacyAnnUsearchDistanceFunction(distanceFunction)) {
+ properties.put(USEARCH_DISTANCE_FUNCTION, distanceFunction);
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ } else if (parameters.length == 2) {
+ String distanceFunction = unquoteMetadataParameter(parameters[0].trim());
+ String scalarKind = unquoteMetadataParameter(parameters[1].trim());
+ if (isLegacyAnnUsearchDistanceFunction(distanceFunction)
+ && isLegacyUsearchScalarKind(scalarKind)) {
+ properties.put(USEARCH_DISTANCE_FUNCTION, distanceFunction);
+ properties.put(USEARCH_SCALAR_KIND, scalarKind);
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ } else {
+ warnUnparsedLegacyAnnUsearchParameters(rawTypeFull, expectedType,
indexName);
+ }
+ return Map.copyOf(properties);
+ }
+
+ private static boolean isUnsignedInteger(String value) {
+ return value.matches("[0-9]+");
+ }
+
+ private static boolean isLegacyAnnUsearchDistanceFunction(String value) {
+ return StringUtils.equalsIgnoreCase("L2Distance", value)
+ || StringUtils.equalsIgnoreCase("cosineDistance", value);
+ }
+
+ private static boolean isLegacyUsearchScalarKind(String value) {
+ return StringUtils.equalsAnyIgnoreCase(value, "f64", "f32", "f16", "i8");
+ }
+
+ private static void warnUnparsedLegacyAnnUsearchParameters(
+ String typeFull, String expectedType, String indexName) {
+ LOG.warn(
+ "Could not parse parameters in legacy ClickHouse type expression '{}'
for index '{}' "
+ + "of type {}; preserving the complete expression",
+ typeFull,
+ indexName,
+ expectedType);
+ }
+
+ private static String unquoteMetadataParameter(String parameter) {
+ if (parameter.length() >= 2
+ && ((parameter.startsWith("'") && parameter.endsWith("'"))
+ || (parameter.startsWith("\"") && parameter.endsWith("\"")))) {
+ return parameter.substring(1, parameter.length() - 1);
+ }
+ return parameter;
+ }
+
+ private static boolean isLegacyAnnUsearchIndex(Index.IndexType indexType) {
+ return indexType == Index.IndexType.DATA_SKIPPING_ANNOY
+ || indexType == Index.IndexType.DATA_SKIPPING_USEARCH;
+ }
+
+ private static String legacyAnnUsearchTypeName(Index.IndexType indexType) {
+ return indexType == Index.IndexType.DATA_SKIPPING_ANNOY
+ ? DATA_SKIPPING_ANNOY
+ : DATA_SKIPPING_USEARCH;
+ }
+
+ private static void rejectLegacyAnnUsearchIndexes(String tableName, Index[]
indexes) {
+ if (ArrayUtils.isEmpty(indexes)) {
+ return;
+ }
+ for (Index index : indexes) {
+ if (isLegacyAnnUsearchIndex(index.type())) {
+ throw legacyAnnUsearchWriteException(tableName, index.name(),
index.type());
+ }
+ }
+ }
+
+ private static IllegalArgumentException legacyAnnUsearchWriteException(
+ String tableName, String indexName, Index.IndexType indexType) {
+ return new IllegalArgumentException(
+ ("ClickHouse legacy index type %s is metadata-only and cannot be
written for table '%s', "
+ + "index '%s'")
+ .formatted(indexType, tableName, indexName));
+ }
+
/**
* Maps a ClickHouse data skipping index type string to the corresponding
Gravitino {@link
* Index.IndexType}. Returns {@code DATA_SKIPPING_MINMAX} for blank/null
input (ClickHouse
- * default). Also handles parameterized formats that some ClickHouse
versions may return from
- * {@code system.data_skipping_indices}.
+ * default). Also handles parameterized formats such as {@code set(N)},
{@code annoy(N)}, {@code
+ * usearch(distanceFunction)}, and {@code vector_similarity(...)} that some
ClickHouse versions
+ * may return from {@code system.data_skipping_indices}.
*
* @param rawType the index type string from ClickHouse metadata (e.g.
"minmax", "bloom_filter",
* "set", "set(0)", or "vector_similarity")
@@ -2567,6 +2758,10 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Index.IndexType.DATA_SKIPPING_NGRAMBFV1;
case DATA_SKIPPING_TOKENBFV1:
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
+ case DATA_SKIPPING_ANNOY:
+ return Index.IndexType.DATA_SKIPPING_ANNOY;
+ case DATA_SKIPPING_USEARCH:
+ return Index.IndexType.DATA_SKIPPING_USEARCH;
case DATA_SKIPPING_VECTOR_SIMILARITY:
return Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY;
default:
@@ -2581,6 +2776,12 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
if (rawType.startsWith(DATA_SKIPPING_TOKENBFV1 + "(")) {
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
}
+ if (isParameterizedLegacyAnnUsearchType(rawType, DATA_SKIPPING_ANNOY))
{
+ return Index.IndexType.DATA_SKIPPING_ANNOY;
+ }
+ if (isParameterizedLegacyAnnUsearchType(rawType,
DATA_SKIPPING_USEARCH)) {
+ return Index.IndexType.DATA_SKIPPING_USEARCH;
+ }
if (rawType.startsWith(DATA_SKIPPING_VECTOR_SIMILARITY + "(")) {
return Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY;
}
@@ -2588,6 +2789,10 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
}
}
+ private static boolean isParameterizedLegacyAnnUsearchType(String rawType,
String typeName) {
+ return rawType.startsWith(typeName + "(") && rawType.endsWith(")");
+ }
+
/**
* Validates that a property value is an integer with a given minimum bound,
returning it as a
* string for DDL interpolation. Unlike {@link #resolveIntProperty(Map,
String, int, int)}, this
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index f1705e2a3d..461308cf54 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -18,8 +18,10 @@
*/
package org.apache.gravitino.catalog.clickhouse.integration.test;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.CLICKHOUSE_TYPE_FULL;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.GRANULARITY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.SET_MAX_VALUES;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_DISTANCE_FUNCTION;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.ENGINE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.ENGINE.MERGETREE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.ENGINE.REPLACINGMERGETREE;
@@ -34,7 +36,11 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
import java.sql.SQLException;
+import java.sql.Statement;
import java.time.LocalDate;
import java.util.Arrays;
import java.util.Collections;
@@ -922,6 +928,226 @@ public class CatalogClickHouseIT extends BaseIT {
assertSetIndexMetadata(nativeLoaded, "idx_native_set", setProperties);
}
+ @Test
+ void testLoadLegacyUsearchIndexMetadata() throws Exception {
+ String legacyCatalogName =
GravitinoITUtils.genRandomName("clickhouse_legacy_catalog");
+ String legacySchemaName =
GravitinoITUtils.genRandomName("clickhouse_legacy_schema");
+ String legacyTableName =
GravitinoITUtils.genRandomName("clickhouse_legacy_usearch");
+ ClickHouseContainer legacyContainer =
+ ClickHouseContainer.builder()
+ .withImage("clickhouse/clickhouse-server:23.8.16.16")
+ .withHostName("gravitino-ci-clickhouse-legacy")
+ .withEnvVars(Map.of("CLICKHOUSE_PASSWORD",
ClickHouseContainer.PASSWORD))
+ .withExposePorts(Set.of(ClickHouseContainer.CLICKHOUSE_PORT))
+ .withNetwork(containerSuite.getNetwork())
+ .build();
+ Catalog legacyCatalog = null;
+ try {
+ legacyContainer.start();
+ legacyContainer.createDatabase(TEST_DB_NAME);
+
+ Map<String, String> catalogProperties = Maps.newHashMap();
+ String jdbcUrl = legacyContainer.getJdbcUrl(TEST_DB_NAME);
+ String baseJdbcUrl = jdbcUrl.substring(0, jdbcUrl.lastIndexOf("/"));
+ catalogProperties.put(JdbcConfig.JDBC_URL.getKey(), baseJdbcUrl);
+ catalogProperties.put(
+ JdbcConfig.JDBC_DRIVER.getKey(),
legacyContainer.getDriverClassName(TEST_DB_NAME));
+ catalogProperties.put(JdbcConfig.USERNAME.getKey(),
legacyContainer.getUsername());
+ catalogProperties.put(JdbcConfig.PASSWORD.getKey(),
legacyContainer.getPassword());
+ legacyCatalog =
+ metalake.createCatalog(
+ legacyCatalogName,
+ Catalog.Type.RELATIONAL,
+ provider,
+ "legacy ClickHouse index metadata fixture",
+ catalogProperties);
+ legacyCatalog = metalake.loadCatalog(legacyCatalogName);
+ legacyCatalog.asSchemas().createSchema(legacySchemaName, "legacy index
metadata", Map.of());
+
+ // This official 23.8 image exposes USearch but not Annoy; Annoy
metadata parsing is covered
+ // by the catalog unit tests.
+ String usearchTypeFull;
+ String legacyJdbcUrl = baseJdbcUrl +
"?custom_settings=allow_experimental_usearch_index%3D1";
+ try (Connection legacyConnection =
+ DriverManager.getConnection(
+ legacyJdbcUrl, legacyContainer.getUsername(),
legacyContainer.getPassword());
+ Statement legacyStatement = legacyConnection.createStatement()) {
+ legacyStatement.execute(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + "`id` UInt64, `usearch_embedding` Array(Float32), "
+ + "INDEX `idx_legacy_usearch` `usearch_embedding` TYPE
usearch('cosineDistance') GRANULARITY 1"
+ + ") ENGINE = MergeTree ORDER BY `id`",
+ legacySchemaName, legacyTableName));
+
+ try (ResultSet resultSet =
+ legacyStatement.executeQuery(
+ String.format(
+ "SELECT type_full FROM system.data_skipping_indices WHERE
database = '%s' "
+ + "AND table = '%s' AND name = 'idx_legacy_usearch'",
+ legacySchemaName, legacyTableName))) {
+ Assertions.assertTrue(resultSet.next(), "Expected to find the legacy
USearch index");
+ usearchTypeFull = resultSet.getString("type_full");
+ }
+ }
+ Assertions.assertEquals("usearch('cosineDistance')", usearchTypeFull);
+
+ Table loaded =
+ legacyCatalog
+ .asTableCatalog()
+ .loadTable(NameIdentifier.of(legacySchemaName, legacyTableName));
+ assertLegacyIndexMetadata(
+ loaded,
+ "idx_legacy_usearch",
+ Index.IndexType.DATA_SKIPPING_USEARCH,
+ "usearch_embedding",
+ Map.of(
+ USEARCH_DISTANCE_FUNCTION,
+ "cosineDistance",
+ CLICKHOUSE_TYPE_FULL,
+ usearchTypeFull,
+ GRANULARITY,
+ "1"));
+ } finally {
+ try {
+ if (legacyCatalog != null) {
+ try {
+ legacyCatalog.asSchemas().dropSchema(legacySchemaName, true);
+ } catch (Exception ignored) {
+ // The historical container is discarded below; cleanup failures
must not hide test
+ // errors.
+ }
+ metalake.disableCatalog(legacyCatalogName);
+ metalake.dropCatalog(legacyCatalogName, true);
+ }
+ } finally {
+ legacyContainer.close();
+ }
+ }
+ }
+
+ @Test
+ void testLoadLegacyAnnoyIndexMetadata() throws Exception {
+ String legacyCatalogName =
GravitinoITUtils.genRandomName("clickhouse_legacy_annoy_catalog");
+ String legacySchemaName =
GravitinoITUtils.genRandomName("clickhouse_legacy_annoy_schema");
+ String legacyTableName =
GravitinoITUtils.genRandomName("clickhouse_legacy_annoy_table");
+ ClickHouseContainer legacyContainer =
+ ClickHouseContainer.builder()
+ .withImage("clickhouse/clickhouse-server:24.8.14.39")
+ .withHostName("gravitino-ci-clickhouse-legacy-annoy")
+ .withEnvVars(Map.of("CLICKHOUSE_PASSWORD",
ClickHouseContainer.PASSWORD))
+ .withExposePorts(Set.of(ClickHouseContainer.CLICKHOUSE_PORT))
+ .withNetwork(containerSuite.getNetwork())
+ .build();
+ Catalog legacyCatalog = null;
+ try {
+ legacyContainer.start();
+ legacyContainer.createDatabase(TEST_DB_NAME);
+
+ Map<String, String> catalogProperties = Maps.newHashMap();
+ String jdbcUrl = legacyContainer.getJdbcUrl(TEST_DB_NAME);
+ String baseJdbcUrl = jdbcUrl.substring(0, jdbcUrl.lastIndexOf("/"));
+ catalogProperties.put(JdbcConfig.JDBC_URL.getKey(), baseJdbcUrl);
+ catalogProperties.put(
+ JdbcConfig.JDBC_DRIVER.getKey(),
legacyContainer.getDriverClassName(TEST_DB_NAME));
+ catalogProperties.put(JdbcConfig.USERNAME.getKey(),
legacyContainer.getUsername());
+ catalogProperties.put(JdbcConfig.PASSWORD.getKey(),
legacyContainer.getPassword());
+ legacyCatalog =
+ metalake.createCatalog(
+ legacyCatalogName,
+ Catalog.Type.RELATIONAL,
+ provider,
+ "legacy Annoy index metadata fixture",
+ catalogProperties);
+ legacyCatalog = metalake.loadCatalog(legacyCatalogName);
+ legacyCatalog.asSchemas().createSchema(legacySchemaName, "legacy Annoy
metadata", Map.of());
+
+ String nativeType;
+ String nativeTypeFull;
+ long nativeGranularity;
+ String legacyJdbcUrl =
+ baseJdbcUrl +
"?custom_settings=allow_experimental_vector_similarity_index%3D1";
+ try (Connection legacyConnection =
+ DriverManager.getConnection(
+ legacyJdbcUrl, legacyContainer.getUsername(),
legacyContainer.getPassword());
+ Statement legacyStatement = legacyConnection.createStatement()) {
+ // ClickHouse 24.8 retains the legacy Annoy metadata-load
compatibility path. Do not insert
+ // or search: legacy index operations are intentionally unsupported by
this contribution.
+ legacyStatement.execute(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + "`id` Int32, `embedding` Array(Float32), "
+ + "INDEX `idx_legacy_annoy` `embedding` TYPE annoy()"
+ + ") ENGINE = MergeTree ORDER BY `id`",
+ legacySchemaName, legacyTableName));
+
+ try (ResultSet resultSet =
+ legacyStatement.executeQuery(
+ String.format(
+ "SELECT type, type_full, granularity FROM
system.data_skipping_indices "
+ + "WHERE database = '%s' AND table = '%s' AND name =
'idx_legacy_annoy'",
+ legacySchemaName, legacyTableName))) {
+ Assertions.assertTrue(resultSet.next(), "Expected to find the legacy
Annoy index");
+ nativeType = resultSet.getString("type");
+ nativeTypeFull = resultSet.getString("type_full");
+ nativeGranularity = resultSet.getLong("granularity");
+ }
+ }
+
+ Assertions.assertEquals("annoy", nativeType);
+ Assertions.assertTrue(
+ StringUtils.startsWithIgnoreCase(nativeTypeFull, "annoy"),
+ "Expected ClickHouse to preserve the Annoy type expression, but got:
" + nativeTypeFull);
+ Assertions.assertTrue(nativeGranularity > 0);
+
+ Table loaded =
+ legacyCatalog
+ .asTableCatalog()
+ .loadTable(NameIdentifier.of(legacySchemaName, legacyTableName));
+ Index loadedAnnoy =
+ Arrays.stream(loaded.index())
+ .filter(index -> Objects.equals(index.name(),
"idx_legacy_annoy"))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("Missing legacy Annoy
index"));
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_ANNOY,
loadedAnnoy.type());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
loadedAnnoy.fieldNames());
+ Assertions.assertEquals(nativeTypeFull,
loadedAnnoy.properties().get(CLICKHOUSE_TYPE_FULL));
+ Assertions.assertEquals(
+ String.valueOf(nativeGranularity),
loadedAnnoy.properties().get(GRANULARITY));
+ } finally {
+ try {
+ if (legacyCatalog != null) {
+ try {
+ legacyCatalog.asSchemas().dropSchema(legacySchemaName, true);
+ } catch (Exception ignored) {
+ // The historical container is discarded below; cleanup failures
must not hide test
+ // errors.
+ }
+ metalake.disableCatalog(legacyCatalogName);
+ metalake.dropCatalog(legacyCatalogName, true);
+ }
+ } finally {
+ legacyContainer.close();
+ }
+ }
+ }
+
+ private void assertLegacyIndexMetadata(
+ Table table,
+ String indexName,
+ Index.IndexType indexType,
+ String fieldName,
+ Map<String, String> properties) {
+ Index index =
+ Arrays.stream(table.index())
+ .filter(candidate -> Objects.equals(indexName, candidate.name()))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("Missing index " +
indexName));
+ Assertions.assertEquals(indexType, index.type());
+ Assertions.assertArrayEquals(new String[][] {{fieldName}},
index.fieldNames());
+ Assertions.assertEquals(properties, index.properties());
+ }
+
private void assertSetIndexMetadata(
Table table, String indexName, Map<String, String> expectedProperties) {
Index index =
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
index 1a77ef0a1a..b16c407641 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
@@ -18,6 +18,12 @@
*/
package org.apache.gravitino.catalog.clickhouse.operations;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.ANNOY_DISTANCE_FUNCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.ANNOY_TREES;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.CLICKHOUSE_TYPE_FULL;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.GRANULARITY;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_DISTANCE_FUNCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_SCALAR_KIND;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.CLICKHOUSE_PROJECTIONS_KEY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseUtils.getSortOrders;
@@ -27,6 +33,7 @@ import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -39,6 +46,7 @@ import
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseExceptionConv
import
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseTypeConverter;
import org.apache.gravitino.catalog.jdbc.JdbcColumn;
import org.apache.gravitino.catalog.jdbc.JdbcTable;
+import org.apache.gravitino.catalog.jdbc.operation.JdbcTableOperations;
import org.apache.gravitino.exceptions.GravitinoRuntimeException;
import org.apache.gravitino.exceptions.NoSuchTableException;
import org.apache.gravitino.rel.TableChange;
@@ -51,6 +59,14 @@ import
org.apache.gravitino.rel.expressions.transforms.Transforms;
import org.apache.gravitino.rel.indexes.Index;
import org.apache.gravitino.rel.indexes.Indexes;
import org.apache.gravitino.rel.types.Types;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.AbstractConfiguration;
+import org.apache.logging.log4j.core.config.LoggerConfig;
+import org.apache.logging.log4j.core.layout.PatternLayout;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
@@ -58,6 +74,26 @@ import org.mockito.Mockito;
public class TestClickHouseTableOperationsUnit {
+ private static final class CaptureAppender extends AbstractAppender {
+ private final List<LogEvent> events = new ArrayList<>();
+
+ CaptureAppender(String name) {
+ super(name, null, PatternLayout.createDefaultLayout(), true, null);
+ }
+
+ @Override
+ public void append(LogEvent event) {
+ events.add(event.toImmutable());
+ }
+
+ List<LogEvent> getEvents() {
+ return events;
+ }
+ }
+
+ private static final String JDBC_TABLE_OPERATIONS_LOGGER_NAME =
+ JdbcTableOperations.class.getName();
+
private static final class ExposedClickHouseTableOperations extends
ClickHouseTableOperations {
private JdbcTable table;
@@ -110,6 +146,21 @@ public class TestClickHouseTableOperationsUnit {
getSortOrders("id"));
}
+ String callGenerateCreateTableSqlWithoutSort(JdbcColumn[] columns, Index[]
indexes) {
+ return generateCreateTableSql(
+ "test_table",
+ columns,
+ "",
+ Map.of(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ indexes);
+ }
+
+ Index.IndexType callGetClickHouseIndexType(String rawType) {
+ return getClickHouseIndexType(rawType);
+ }
+
void setTable(JdbcTable table) {
this.table = table;
}
@@ -147,6 +198,63 @@ public class TestClickHouseTableOperationsUnit {
Assertions.assertTrue(exception.getMessage().contains("ClickHouse does not
support varchar"));
}
+ @Test
+ void
testLegacyAnnUsearchTypesAreDistinctAndParameterizedFormsAreRecognized() {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_ANNOY,
ops.callGetClickHouseIndexType("annoy"));
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_ANNOY,
ops.callGetClickHouseIndexType("annoy(100)"));
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_USEARCH,
ops.callGetClickHouseIndexType("usearch"));
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_USEARCH,
+ ops.callGetClickHouseIndexType("usearch(cosineDistance)"));
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ ops.callGetClickHouseIndexType("vector_similarity"));
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
ops.callGetClickHouseIndexType("unknown_index"));
+ }
+
+ @Test
+ void testCreateAndAlterRejectLegacyAnnUsearchIndexesWithContext() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ for (Index.IndexType indexType :
+ List.of(Index.IndexType.DATA_SKIPPING_ANNOY,
Index.IndexType.DATA_SKIPPING_USEARCH)) {
+ Index index = Indexes.of(indexType, "idx_legacy", new String[][]
{{"id"}});
+ IllegalArgumentException createException =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> newOps().callGenerateCreateTableSql(columns, Map.of(), new
Index[] {index}));
+ assertLegacyWriteError(createException, indexType);
+
+ IllegalArgumentException createWithoutSortException =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> newOps().callGenerateCreateTableSqlWithoutSort(columns,
new Index[] {index}));
+ assertLegacyWriteError(createWithoutSortException, indexType);
+
+ IllegalArgumentException alterException =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ newAlterOps(Map.of())
+ .callGenerateAlterTableSql(
+ TableChange.addIndex(indexType, "idx_legacy", new
String[][] {{"id"}})));
+ assertLegacyWriteError(alterException, indexType);
+ }
+ }
+
@Test
void testToPartitionExpressionSupportsStartFunctions() {
Assertions.assertEquals(
@@ -228,6 +336,54 @@ public class TestClickHouseTableOperationsUnit {
return ops;
}
+ private static List<Index> getIndexesWithoutWarnings(
+ ExposedClickHouseTableOperations ops,
+ Connection connection,
+ String databaseName,
+ String tableName)
+ throws Exception {
+ return getIndexesExpectingWarnings(ops, connection, databaseName,
tableName, 0);
+ }
+
+ private static List<Index> getIndexesExpectingWarnings(
+ ExposedClickHouseTableOperations ops,
+ Connection connection,
+ String databaseName,
+ String tableName,
+ int expectedWarningCount)
+ throws Exception {
+ LoggerContext loggerContext =
+ (LoggerContext)
+
LogManager.getContext(ClickHouseTableOperations.class.getClassLoader(), false);
+ AbstractConfiguration configuration = (AbstractConfiguration)
loggerContext.getConfiguration();
+ CaptureAppender warningCapture = new
CaptureAppender("clickhouseMetadataWarningCapture");
+ warningCapture.start();
+ configuration.addAppender(warningCapture);
+ LoggerConfig loggerConfig =
+ new LoggerConfig(JDBC_TABLE_OPERATIONS_LOGGER_NAME, Level.WARN, false);
+ loggerConfig.addAppender(warningCapture, Level.WARN, null);
+ configuration.addLogger(JDBC_TABLE_OPERATIONS_LOGGER_NAME, loggerConfig);
+ loggerContext.updateLoggers();
+
+ try {
+ List<Index> indexes = ops.callGetIndexes(connection, databaseName,
tableName);
+ Assertions.assertEquals(
+ expectedWarningCount,
+ warningCapture.getEvents().size(),
+ "Unexpected warning count while loading legacy Annoy and USearch
metadata");
+ Assertions.assertTrue(
+ warningCapture.getEvents().stream()
+ .allMatch(event -> Level.WARN.equals(event.getLevel())),
+ "Legacy Annoy and USearch metadata warnings should use WARN level");
+ return indexes;
+ } finally {
+ configuration.removeLogger(JDBC_TABLE_OPERATIONS_LOGGER_NAME);
+ warningCapture.stop();
+ configuration.removeAppender(warningCapture.getName());
+ loggerContext.updateLoggers();
+ }
+ }
+
private ExposedClickHouseTableOperations newAlterOps(Map<String, String>
properties) {
ExposedClickHouseTableOperations ops = newOps();
JdbcColumn idColumn =
@@ -1435,10 +1591,6 @@ public class TestClickHouseTableOperationsUnit {
Assertions.assertEquals(
Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
ops.getClickHouseIndexType("vector_similarity('hnsw', 'L2Distance',
3)"));
- Assertions.assertThrows(
- IllegalArgumentException.class, () ->
ops.getClickHouseIndexType("annoy"));
- Assertions.assertThrows(
- IllegalArgumentException.class, () ->
ops.getClickHouseIndexType("usearch"));
}
@Test
@@ -1699,6 +1851,271 @@ public class TestClickHouseTableOperationsUnit {
indexes.get(0).properties());
}
+ @Test
+ void testGetIndexesPreservesLegacyAnnUsearchMetadata() throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, true, true, true,
true, true, false);
+ Mockito.when(secondaryRs.getString("name"))
+ .thenReturn(
+ "idx_annoy",
+ "idx_usearch",
+ "idx_annoy_without_parameters",
+ "idx_annoy_distance_only",
+ "idx_annoy_distance_and_trees",
+ "idx_usearch_without_parameters",
+ "idx_usearch_distance_and_scalar_kind");
+ Mockito.when(secondaryRs.getString("type"))
+ .thenReturn("annoy", "usearch", "annoy", "annoy", "annoy", "usearch",
"usearch");
+ // The native metadata DDL in ClickHouse issue #41729 uses annoy(100) with
GRANULARITY 1.
+ // This is a parser fixture only; it does not validate a successful
system-table read path.
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn(
+ "annoy(100)",
+ "usearch('cosineDistance')",
+ "annoy()",
+ "annoy('L2Distance')",
+ "annoy('cosineDistance', 100)",
+ "usearch()",
+ "usearch('L2Distance', 'f16')");
+ Mockito.when(secondaryRs.getString("expr"))
+ .thenReturn(
+ "embedding",
+ "tuple(embedding)",
+ "embedding",
+ "embedding",
+ "embedding",
+ "embedding",
+ "embedding");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(1L, 1L, 1L,
1L, 1L, 1L, 1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes = getIndexesWithoutWarnings(ops, connection, "db",
"tbl");
+
+ Assertions.assertEquals(7, indexes.size());
+ Index annoy = indexes.get(0);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_ANNOY, annoy.type());
+ Assertions.assertEquals("idx_annoy", annoy.name());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
annoy.fieldNames());
+ Assertions.assertEquals(
+ Map.of(ANNOY_TREES, "100", CLICKHOUSE_TYPE_FULL, "annoy(100)",
GRANULARITY, "1"),
+ annoy.properties());
+
+ Index usearch = indexes.get(1);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_USEARCH,
usearch.type());
+ Assertions.assertEquals("idx_usearch", usearch.name());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
usearch.fieldNames());
+ Assertions.assertEquals(
+ Map.of(
+ USEARCH_DISTANCE_FUNCTION,
+ "cosineDistance",
+ CLICKHOUSE_TYPE_FULL,
+ "usearch('cosineDistance')",
+ GRANULARITY,
+ "1"),
+ usearch.properties());
+
+ Index annoyWithoutParameters = indexes.get(2);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_ANNOY,
annoyWithoutParameters.type());
+ Assertions.assertEquals("idx_annoy_without_parameters",
annoyWithoutParameters.name());
+ Assertions.assertArrayEquals(
+ new String[][] {{"embedding"}}, annoyWithoutParameters.fieldNames());
+ Assertions.assertEquals(
+ Map.of(CLICKHOUSE_TYPE_FULL, "annoy()", GRANULARITY, "1"),
+ annoyWithoutParameters.properties());
+
+ Index annoyDistanceOnly = indexes.get(3);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_ANNOY,
annoyDistanceOnly.type());
+ Assertions.assertEquals(
+ Map.of(
+ ANNOY_DISTANCE_FUNCTION,
+ "L2Distance",
+ CLICKHOUSE_TYPE_FULL,
+ "annoy('L2Distance')",
+ GRANULARITY,
+ "1"),
+ annoyDistanceOnly.properties());
+
+ Index annoyDistanceAndTrees = indexes.get(4);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_ANNOY,
annoyDistanceAndTrees.type());
+ Assertions.assertEquals(
+ Map.of(
+ ANNOY_DISTANCE_FUNCTION,
+ "cosineDistance",
+ ANNOY_TREES,
+ "100",
+ CLICKHOUSE_TYPE_FULL,
+ "annoy('cosineDistance', 100)",
+ GRANULARITY,
+ "1"),
+ annoyDistanceAndTrees.properties());
+
+ Index usearchWithoutParameters = indexes.get(5);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_USEARCH,
usearchWithoutParameters.type());
+ Assertions.assertEquals(
+ Map.of(CLICKHOUSE_TYPE_FULL, "usearch()", GRANULARITY, "1"),
+ usearchWithoutParameters.properties());
+
+ Index usearchDistanceAndScalarKind = indexes.get(6);
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_USEARCH,
usearchDistanceAndScalarKind.type());
+ Assertions.assertEquals(
+ Map.of(
+ USEARCH_DISTANCE_FUNCTION,
+ "L2Distance",
+ USEARCH_SCALAR_KIND,
+ "f16",
+ CLICKHOUSE_TYPE_FULL,
+ "usearch('L2Distance', 'f16')",
+ GRANULARITY,
+ "1"),
+ usearchDistanceAndScalarKind.properties());
+ }
+
+ @Test
+ void testGetIndexesWarnsAndPreservesMalformedLegacyAnnUsearchParameters()
throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+
+ String[] typeFullExpressions = {
+ "annoy('bogus')",
+ "annoy('L2Distance', 'not-a-number')",
+ "annoy('L2Distance', 10, 20)",
+ "usearch('bogus')",
+ "usearch('L2Distance', 'invalid-kind')",
+ "usearch('L2Distance', 'f16', 'extra')"
+ };
+ String[] indexNames = {
+ "idx_annoy_invalid_distance",
+ "idx_annoy_invalid_trees",
+ "idx_annoy_extra_parameters",
+ "idx_usearch_invalid_distance",
+ "idx_usearch_invalid_scalar_kind",
+ "idx_usearch_extra_parameters"
+ };
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, true, true, true,
true, false);
+ Mockito.when(secondaryRs.getString("name"))
+ .thenReturn(
+ indexNames[0],
+ indexNames[1],
+ indexNames[2],
+ indexNames[3],
+ indexNames[4],
+ indexNames[5]);
+ Mockito.when(secondaryRs.getString("type"))
+ .thenReturn("annoy", "annoy", "annoy", "usearch", "usearch",
"usearch");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn(
+ typeFullExpressions[0],
+ typeFullExpressions[1],
+ typeFullExpressions[2],
+ typeFullExpressions[3],
+ typeFullExpressions[4],
+ typeFullExpressions[5]);
+ Mockito.when(secondaryRs.getString("expr"))
+ .thenReturn("embedding", "embedding", "embedding", "embedding",
"embedding", "embedding");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(1L, 1L, 1L,
1L, 1L, 1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes =
+ getIndexesExpectingWarnings(ops, connection, "db", "tbl",
typeFullExpressions.length);
+
+ Assertions.assertEquals(typeFullExpressions.length, indexes.size());
+ for (int i = 0; i < typeFullExpressions.length; i++) {
+ Assertions.assertEquals(indexNames[i], indexes.get(i).name());
+ Assertions.assertEquals(
+ Map.of(CLICKHOUSE_TYPE_FULL, typeFullExpressions[i], GRANULARITY,
"1"),
+ indexes.get(i).properties());
+ }
+ }
+
+ @Test
+ void testGetIndexesFallsBackAndPreservesLegacyAnnUsearchTypes() throws
Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement modernSecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ PreparedStatement legacySecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ ResultSet legacySecondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(modernSecondaryStmt.executeQuery())
+ .thenThrow(new SQLException("Unknown identifier 'type_full'"));
+
Mockito.when(legacySecondaryStmt.executeQuery()).thenReturn(legacySecondaryRs);
+ Mockito.when(legacySecondaryRs.next()).thenReturn(true, true, true, false);
+ Mockito.when(legacySecondaryRs.getString("name"))
+ .thenReturn("idx_legacy_annoy", "idx_legacy_usearch", "idx_unknown");
+ Mockito.when(legacySecondaryRs.getString("type"))
+ .thenReturn("annoy", "usearch", "vector_similarity");
+ Mockito.when(legacySecondaryRs.getString("expr"))
+ .thenReturn("embedding", "embedding", "embedding");
+ Mockito.when(legacySecondaryRs.getLong("granularity")).thenReturn(1L, 1L,
1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(modernSecondaryStmt)
+ .thenReturn(legacySecondaryStmt);
+
+ List<Index> indexes = getIndexesExpectingWarnings(ops, connection, "db",
"tbl", 2);
+
+ Assertions.assertEquals(2, indexes.size());
+ Assertions.assertEquals(
+ Map.of(CLICKHOUSE_TYPE_FULL, "annoy", GRANULARITY, "1"),
indexes.get(0).properties());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_USEARCH,
indexes.get(1).type());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
indexes.get(1).fieldNames());
+ Assertions.assertEquals(
+ Map.of(CLICKHOUSE_TYPE_FULL, "usearch", GRANULARITY, "1"),
indexes.get(1).properties());
+ }
+
+ @Test
+ void testGetIndexesReportsUnrepresentableLegacyExpressionByOmittingIt()
throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, false);
+ Mockito.when(secondaryRs.getString("name")).thenReturn("idx_annoy_expr");
+ Mockito.when(secondaryRs.getString("type")).thenReturn("annoy");
+ Mockito.when(secondaryRs.getString("type_full")).thenReturn("annoy(100)");
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("lower(embedding)");
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ Assertions.assertTrue(ops.callGetIndexes(connection, "db",
"tbl").isEmpty());
+ }
+
@Test
void testGetIndexesFallsBackAndReadsLegacySetParameters() throws Exception {
ExposedClickHouseTableOperations ops = newOps();
@@ -2411,4 +2828,12 @@ public class TestClickHouseTableOperationsUnit {
return Assertions.assertThrows(
IllegalArgumentException.class, () ->
getIndexesForSetMetadata(typeFull, expression));
}
+
+ private static void assertLegacyWriteError(
+ IllegalArgumentException exception, Index.IndexType indexType) {
+ Assertions.assertTrue(exception.getMessage().contains("test_table"));
+ Assertions.assertTrue(exception.getMessage().contains("idx_legacy"));
+ Assertions.assertTrue(exception.getMessage().contains(indexType.name()));
+ Assertions.assertTrue(exception.getMessage().contains("metadata-only"));
+ }
}
diff --git a/clients/client-python/gravitino/api/rel/indexes/index.py
b/clients/client-python/gravitino/api/rel/indexes/index.py
index d55929a040..0fe335acc2 100644
--- a/clients/client-python/gravitino/api/rel/indexes/index.py
+++ b/clients/client-python/gravitino/api/rel/indexes/index.py
@@ -32,6 +32,8 @@ class Index(ABC):
"""The enum IndexType defines the type of the index.
Catalogs may support generic key types and connector-specific index
types.
+ `DATA_SKIPPING_ANNOY` and `DATA_SKIPPING_USEARCH` are exposed for
reading legacy
+ ClickHouse index metadata only.
"""
PRIMARY_KEY = "PRIMARY_KEY"
@@ -50,6 +52,12 @@ class Index(ABC):
columns that are not designated as the primary key. The uniqueness
constraint imposed by
UNIQUE KEY helps in avoiding redundancy and ensuring data accuracy in
the database."""
+ DATA_SKIPPING_ANNOY = "DATA_SKIPPING_ANNOY"
+ """Legacy ClickHouse Annoy index metadata. This type is
metadata-only."""
+
+ DATA_SKIPPING_USEARCH = "DATA_SKIPPING_USEARCH"
+ """Legacy ClickHouse USearch index metadata. This type is
metadata-only."""
+
DATA_SKIPPING_VECTOR_SIMILARITY = "DATA_SKIPPING_VECTOR_SIMILARITY"
"""ClickHouse vector similarity data skipping index."""
diff --git
a/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
b/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
index eeca69bca2..608561c835 100644
---
a/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
+++
b/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
@@ -25,14 +25,17 @@ from gravitino.utils.serdes import SerdesUtilsBase
class IndexSerdes(SerdesUtilsBase, JsonSerializable[Index]):
+ INDEX_PROPERTIES = "properties"
+
@classmethod
def serialize(cls, value: Index) -> dict[str, Any]:
result: dict[str, Any] = {cls.INDEX_TYPE: value.type().name.upper()}
if value.name() is not None:
result[cls.INDEX_NAME] = value.name()
result[cls.INDEX_FIELD_NAMES] = value.field_names()
- if value.properties():
- result["properties"] = value.properties()
+ properties = value.properties()
+ if properties:
+ result[cls.INDEX_PROPERTIES] = properties
return result
@@ -56,5 +59,5 @@ class IndexSerdes(SerdesUtilsBase, JsonSerializable[Index]):
index_type,
data.get(cls.INDEX_NAME),
data[cls.INDEX_FIELD_NAMES],
- data.get("properties"),
+ data.get(cls.INDEX_PROPERTIES),
)
diff --git a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
index 18b960b87c..99247713ac 100644
--- a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
+++ b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
@@ -38,6 +38,21 @@ class MockDataClass(DataClassJsonMixin):
class TestIndexSerdes(unittest.TestCase):
+ def test_index_serdes_without_properties_override(self):
+ class IndexWithoutProperties(Index):
+ def type(self) -> Index.IndexType:
+ return Index.IndexType.PRIMARY_KEY
+
+ def name(self) -> str:
+ return "PRIMARY"
+
+ def field_names(self) -> list[list[str]]:
+ return [["id"]]
+
+ index = IndexWithoutProperties()
+ self.assertEqual({}, index.properties())
+ self.assertNotIn("properties", IndexSerdes.serialize(index))
+
def test_index_serdes_invalid_json(self):
invalid_json_string = [
'{"indexes": [{}]}',
@@ -117,6 +132,51 @@ class TestIndexSerdes(unittest.TestCase):
serialized_dict = json.loads(mock_data_class.to_json())
self.assertDictEqual(json_dict, serialized_dict)
+ def test_index_serdes_legacy_clickhouse_metadata(self):
+ annoy_properties = {
+ "annoy_trees": "100",
+ "clickhouse_type_full": "annoy(100)",
+ "granularity": "1",
+ }
+ usearch_properties = {
+ "clickhouse_type_full": "usearch('cosineDistance')",
+ "granularity": "1",
+ "usearch_distance_function": "cosineDistance",
+ }
+
+ annoy = IndexSerdes.deserialize(
+ {
+ "indexType": "data_skipping_annoy",
+ "name": "idx_annoy",
+ "fieldNames": [["embedding"]],
+ "properties": annoy_properties,
+ }
+ )
+ usearch = IndexSerdes.deserialize(
+ {
+ "indexType": "data_skipping_usearch",
+ "name": "idx_usearch",
+ "fieldNames": [["embedding"]],
+ "properties": usearch_properties,
+ }
+ )
+
+ self.assertEqual(Index.IndexType.DATA_SKIPPING_ANNOY, annoy.type())
+ self.assertEqual(annoy_properties, annoy.properties())
+ self.assertEqual(Index.IndexType.DATA_SKIPPING_USEARCH, usearch.type())
+ self.assertEqual(usearch_properties, usearch.properties())
+
+ serialized_json = MockDataClass([annoy, usearch]).to_json()
+ serialized = json.loads(serialized_json)
+ self.assertEqual(annoy_properties,
serialized["indexes"][0]["properties"])
+ self.assertEqual(usearch_properties,
serialized["indexes"][1]["properties"])
+
+ round_tripped = MockDataClass.from_json(serialized_json).indexes
+ self.assertEqual(Index.IndexType.DATA_SKIPPING_ANNOY,
round_tripped[0].type())
+ self.assertEqual(annoy_properties, round_tripped[0].properties())
+ self.assertEqual(Index.IndexType.DATA_SKIPPING_USEARCH,
round_tripped[1].type())
+ self.assertEqual(usearch_properties, round_tripped[1].properties())
+
def test_vector_similarity_index_properties_round_trip(self):
json_string = """
{
diff --git a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
index 4d836e4bf6..dd9dca0026 100644
--- a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
+++ b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
@@ -496,6 +496,20 @@ public class TestJsonUtils {
objectMapper.readValue(expected, IndexDTO.class),
objectMapper.readValue(jsonValue, IndexDTO.class));
+ Index legacyIndex =
+ IndexDTO.builder()
+ .withIndexType(Index.IndexType.DATA_SKIPPING_USEARCH)
+ .withName("idx_legacy_usearch")
+ .withFieldNames(new String[][] {{"embedding"}})
+ .withProperties(Map.of("clickhouse_type_full",
"usearch(cosineDistance)"))
+ .build();
+ jsonValue = JsonUtils.objectMapper().writeValueAsString(legacyIndex);
+ Index legacyRoundTrip = objectMapper.readValue(jsonValue, IndexDTO.class);
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_USEARCH,
legacyRoundTrip.type());
+ Assertions.assertEquals("idx_legacy_usearch", legacyRoundTrip.name());
+ Assertions.assertEquals(
+ Map.of("clickhouse_type_full", "usearch(cosineDistance)"),
legacyRoundTrip.properties());
+
Index idx2 =
IndexDTO.builder()
.withIndexType(Index.IndexType.PRIMARY_KEY)
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index ddab8ff151..370ba3758a 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -178,7 +178,7 @@ See [Manage Catalogs and
Schemas](./manage-catalogs-and-schemas.md#schema-operat
| Mapping | Gravitino table maps to a ClickHouse table
|
| Engines | **MergeTree family** (`MergeTree` default,
`ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`,
`CollapsingMergeTree`, `VersionedCollapsingMergeTree`, `GraphiteMergeTree`):
fully supported, data persists across restarts. **Log family** (`TinyLog`,
`StripeLog`, `Log`): supported, data and table definition persist across
restarts. **`Null`**: supported, table persists, data is always discarded by
design. **`Set`**: supported, table definition persists. [...]
| Ordering/Partition | MergeTree-family requires exactly one `ORDER BY`
column; `PARTITION BY` supports single-column identity and the function
expressions listed below. Other engines reject `ORDER BY`/`PARTITION BY`.
|
-| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, `DATA_SKIPPING_SET`,
`DATA_SKIPPING_NGRAMBFV1`, `DATA_SKIPPING_TOKENBFV1`, and
`DATA_SKIPPING_VECTOR_SIMILARITY` (configurable granularity via
`Index.properties()`).
[...]
+| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, `DATA_SKIPPING_SET`,
`DATA_SKIPPING_NGRAMBFV1`, `DATA_SKIPPING_TOKENBFV1`, and
`DATA_SKIPPING_VECTOR_SIMILARITY` (configurable granularity via
`Index.properties()`). Legacy `DATA_SKIPPING_ANNOY` and `DATA_SKIPPING_USEARCH`
definitions can be read as metadata only.
[...]
| Distribution | Gravitino enforces `Distributions.NONE`; no custom
distribution strategies.
|
| Column defaults | Supported.
|
| Unsupported | Engine and connector-owned property changes after
creation; mixing table setting changes with schema changes; auto-increment
columns.
|
@@ -303,16 +303,24 @@ The `engine_parameters` property applies to
`ReplacingMergeTree`, `SummingMergeT
When adding a data-skipping index to an existing table through Gravitino's
`ALTER TABLE ADD INDEX`, existing data parts are not synchronously backfilled,
and Gravitino does not issue a materialization command. To materialize the
index on existing data, run `ALTER TABLE <table> MATERIALIZE INDEX
<index_name>` directly in ClickHouse. This can take time, especially for vector
indexes; ClickHouse may also materialize indexes during later background
merges, depending on its settings.
+ - Legacy `DATA_SKIPPING_ANNOY` and `DATA_SKIPPING_USEARCH` metadata read
from ClickHouse versions that expose these definitions. Their reported type
expression is retained in `clickhouse_type_full`.
+ - Annoy parameters are exposed as `annoy_distance_function` (`L2Distance` or
`cosineDistance`) and `annoy_trees`. USearch parameters are exposed as
`usearch_distance_function` (`L2Distance` or `cosineDistance`) and
`usearch_scalar_kind` (`f64`, `f32`, `f16`, or `i8`). Only parameters present
in the reported expression are materialized; omitted parameters are not
inferred. Empty parameter lists and bare legacy type names are valid metadata
forms.
+ - These index types are metadata-only: Gravitino rejects CREATE TABLE
definitions, ALTER TABLE ADD INDEX requests, and table recreation that use
them. They do not provide insert or vector-search support. Gravitino does not
automatically migrate them to `vector_similarity`; any migration must be
explicit.
+
Custom `GRANULARITY` can be specified via the `Index.properties()` API (key
`granularity`, value must be a positive integer). For `DATA_SKIPPING_SET`, the
max unique values can be configured via `set_max_values` (non-negative
integer). If not specified, the defaults above apply.
On ClickHouse versions without `system.data_skipping_indices.type_full`,
Gravitino falls back to the legacy metadata query. If the legacy `type` value
does not include the bloom-filter parameters, the index type and fields are
preserved but the required parameter properties cannot be reconstructed;
provide the properties explicitly before recreating the table.
+ The legacy fallback query may return only the bare `annoy` or `usearch` type
name. Gravitino preserves that reported name without a parsing warning;
parameter properties remain absent because the server did not expose them.
+
For `DATA_SKIPPING_VECTOR_SIMILARITY`, Gravitino restores parameters from
the legacy `type` value when present. If the legacy value is bare and cannot
provide the required parameters, the index is skipped with a warning.
ClickHouse 26.4 and later can create native `vector_similarity` indexes with
`dotProduct`. Gravitino skips native vector indexes with unsupported parameter
values, including `dotProduct`, and logs a warning when loading the table.
Skipped indexes are absent from the returned table metadata and will not be
recreated from it. Creating or adding a `dotProduct` vector index through
Gravitino remains unsupported.
ClickHouse data-skipping indexes whose field expressions cannot be
represented as Gravitino field names, such as `lower(name)` or `name + 1`, are
skipped with a warning when the table is loaded and are not recreated. Direct
column references and tuples containing only column references remain supported.
+ ClickHouse legacy Annoy and USearch indexes with unrepresentable field
expressions are also omitted with an explicit warning. The Gravitino index API
does not represent arbitrary index expressions.
+
### Partitioning, Sorting, and Distribution
- `ORDER BY`: required for MergeTree-family engines and only columns identity
are supported;
diff --git a/docs/open-api/indexes.yaml b/docs/open-api/indexes.yaml
index cbcdbde96a..894761b776 100644
--- a/docs/open-api/indexes.yaml
+++ b/docs/open-api/indexes.yaml
@@ -40,6 +40,8 @@ components:
- "data_skipping_set"
- "data_skipping_ngrambfv1"
- "data_skipping_tokenbfv1"
+ - "data_skipping_annoy"
+ - "data_skipping_usearch"
- "data_skipping_vector_similarity"
name:
type: string
