This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new ebdaf30e02 [#11935] feat(clickhouse): add vector similarity index
support (#13430)
ebdaf30e02 is described below
commit ebdaf30e021ef351fc20a794775265ee6dded36f
Author: StormSpirit <[email protected]>
AuthorDate: Thu Oct 1 10:58:18 2026 +0800
[#11935] feat(clickhouse): add vector similarity index support (#13430)
### What changes were proposed in this pull request?
Add the public `DATA_SKIPPING_VECTOR_SIMILARITY` index type and
ClickHouse support for discovering, creating, altering, and reloading
`vector_similarity` indexes with validated HNSW properties and
normalized defaults. Add the Python enum and preserve index properties
through JSON round-trip so the Python client can deserialize
vector-index metadata for tables whose other index types it supports.
When loading a native vector index with an unsupported algorithm,
distance function, or quantization, skip that index with a warning while
preserving the rest of the table metadata; malformed supported metadata
still fails. Update the ClickHouse catalog guide to describe these
behaviors.
### Why are the changes needed?
Expose ClickHouse 25.8+ vector similarity indexes through Gravitino's
relational index API while preserving supported fields, parameters, and
effective granularity on reload. Without the Python enum, a response
containing the new index type prevents Python `load_table` from
returning the table. ClickHouse 26.4 added native dotProduct vector
indexes; since creation through Gravitino remains out of scope, their
metadata must not prevent the whole table from loading.
Fix: #11935
### Does this PR introduce _any_ user-facing change?
Adds `Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY` for ClickHouse
25.8 or later, with `L2Distance` and `cosineDistance`, HNSW dimensions,
optional quantization and construction parameters, and a default
granularity of 100000000. The Python client recognizes this type and
preserves non-empty index properties. Native dotProduct indexes on
ClickHouse 26.4 or later are omitted from loaded Gravitino metadata with
a warning, so recreating a table from that metadata will omit those
indexes; CREATE TABLE and ALTER TABLE ADD INDEX through Gravitino
continue to reject dotProduct.
### How was this patch tested?
- Before the follow-up, `./gradlew rat`, Spotless checks for `common`,
`integration-test-common`, and `catalog-jdbc-clickhouse`, `./gradlew
:common:test --tests 'org.apache.gravitino.json.TestJsonUtils'
-PskipITs` (13 tests), `./gradlew
:catalogs-contrib:catalog-jdbc-clickhouse:build -x test`, and `./gradlew
:docs:build` passed.
- Before the follow-up, `./gradlew
:catalogs-contrib:catalog-jdbc-clickhouse:test -PskipITs
-PskipDockerTests=true` passed 135 tests, and the same task with
`-PskipDockerTests=false` passed 179 tests; on ClickHouse 25.8.14.17 the
full `CatalogClickHouseIT` passed 63 tests and the full
`CatalogClickHouseClusterIT` passed 21 tests.
- On the synchronized follow-up code, the Python `TestIndexSerdes` and
`TestRelationalCatalog` suites passed 38 tests on Python 3.12.12; Black
and Ruff checks passed for the changed Python files.
- On the synchronized follow-up code, `./gradlew
:catalogs-contrib:catalog-jdbc-clickhouse:spotlessCheck
:catalogs-contrib:catalog-jdbc-clickhouse:test --tests
'org.apache.gravitino.catalog.clickhouse.operations.TestClickHouseTableOperationsUnit'
-PskipITs` passed 72 tests.
- Before synchronizing with newer main, the follow-up code with
`GRAVITINO_CLICKHOUSE_TEST_IMAGE=clickhouse:25.8.14.17` passed the
targeted `CatalogClickHouseIT.testVectorSimilarityIndex` once and the
full `CatalogClickHouseClusterIT` with 21 tests; neither run skipped
tests.
- Before synchronizing with newer main,
`CatalogClickHouseIT.testLoadTableWithUnsupportedNativeDotProductVectorIndex`
executed and passed once on ClickHouse 26.4.2.10 with HTTP compression
disabled in a temporary local test image; the log confirmed the warning
for the skipped dotProduct index. The repository's JDBC 0.7.1 driver
failed during container health checking with the default 26.4 HTTP
compression setting, before the test body. CI uses ClickHouse 25.8, so
the 26.4-only test is skipped there; this PR does not fix the
default-compression JDBC incompatibility. The main synchronization did
not change the vector-index production path, so these Docker suites were
not repeated on the synchronized code.
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.github/workflows/contrib-catalog-test.yml | 2 +
.../org/apache/gravitino/rel/indexes/Index.java | 9 +-
.../catalog/clickhouse/ClickHouseConstants.java | 22 +
.../operations/ClickHouseTableOperations.java | 414 ++++++++++++++++++-
.../test/CatalogClickHouseClusterIT.java | 85 ++++
.../integration/test/CatalogClickHouseIT.java | 164 +++++++-
.../TestClickHouseTableOperationsUnit.java | 453 ++++++++++++++++++++-
.../gravitino/api/rel/indexes/index.py | 11 +-
.../gravitino/dto/rel/indexes/index_dto.py | 5 +-
.../dto/rel/indexes/json_serdes/index_serdes.py | 7 +-
.../tests/unittests/dto/rel/test_index_dto.py | 26 ++
.../tests/unittests/dto/rel/test_index_serdes.py | 29 ++
.../tests/unittests/test_relational_catalog.py | 33 ++
.../org/apache/gravitino/json/TestJsonUtils.java | 16 +
docs/jdbc-clickhouse-catalog.md | 13 +-
docs/open-api/indexes.yaml | 1 +
...partitioning-distribution-sort-order-indexes.md | 1 +
.../integration/test/container/ContainerSuite.java | 35 ++
.../src/test/resources/clickhouse-test-users.xml | 25 ++
19 files changed, 1331 insertions(+), 20 deletions(-)
diff --git a/.github/workflows/contrib-catalog-test.yml
b/.github/workflows/contrib-catalog-test.yml
index 8adb813725..cef67c2097 100644
--- a/.github/workflows/contrib-catalog-test.yml
+++ b/.github/workflows/contrib-catalog-test.yml
@@ -100,6 +100,8 @@ jobs:
- name: Run contrib catalog tests
id: contribCatalogTest
+ env:
+ GRAVITINO_CLICKHOUSE_TEST_IMAGE: clickhouse:25.8.14.17
run: |
CHANGED_CONTRIB_MODULES="${{
needs.changes.outputs.changed_catalogs_contrib_modules }}"
diff --git a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
index ec3d6cbce7..e139ac9c69 100644
--- a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
+++ b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
@@ -54,8 +54,8 @@ public interface Index {
}
/**
- * The enum IndexType defines the type of the index. Currently, PRIMARY_KEY
and UNIQUE_KEY are
- * supported.
+ * The index type. Catalogs may support generic key types as well as
connector-specific index
+ * types.
*/
enum IndexType {
/**
@@ -144,5 +144,10 @@ public interface Index {
* v19.x.
*/
DATA_SKIPPING_TOKENBFV1,
+
+ /**
+ * Vector similarity data skipping index for approximate nearest-neighbor
search in ClickHouse.
+ */
+ DATA_SKIPPING_VECTOR_SIMILARITY,
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
index 94420487d7..6f40eedf6f 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
@@ -84,6 +84,28 @@ public class ClickHouseConstants {
/** The name of the data skipping index type for tokenbf_v1 in ClickHouse.
*/
public static final String DATA_SKIPPING_TOKENBFV1 = "tokenbf_v1";
+ /** The name of the vector similarity data skipping index type in
ClickHouse. */
+ public static final String DATA_SKIPPING_VECTOR_SIMILARITY =
"vector_similarity";
+
+ /** The property key for the algorithm used by vector similarity indexes.
*/
+ public static final String VECTOR_SIMILARITY_TYPE = "type";
+
+ /** The property key for the distance function used by vector similarity
indexes. */
+ public static final String VECTOR_SIMILARITY_DISTANCE_FUNCTION =
"distance_function";
+
+ /** The property key for the vector dimension used by vector similarity
indexes. */
+ public static final String VECTOR_SIMILARITY_DIMENSIONS = "dimensions";
+
+ /** The property key for HNSW vector quantization. */
+ public static final String VECTOR_SIMILARITY_QUANTIZATION = "quantization";
+
+ /** The property key for the HNSW maximum connections per layer. */
+ public static final String HNSW_MAX_CONNECTIONS_PER_LAYER =
"hnsw_max_connections_per_layer";
+
+ /** The property key for the HNSW candidate list size used during
construction. */
+ public static final String HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION =
+ "hnsw_candidate_list_size_for_construction";
+
/** Property key for bloom filter size in ngrambf_v1 and tokenbf_v1 index
properties. */
public static final String BLOOM_FILTER_SIZE = "bloom_filter_size";
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index 3af9eac869..dac8cd44ed 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -24,11 +24,18 @@ import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexC
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_NGRAMBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_SET;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_TOKENBFV1;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_VECTOR_SIMILARITY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.GRANULARITY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.HASH_FUNCTIONS;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.HNSW_MAX_CONNECTIONS_PER_LAYER;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.NGRAM_SIZE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.RANDOM_SEED;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.SET_MAX_VALUES;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_DIMENSIONS;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_DISTANCE_FUNCTION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_QUANTIZATION;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_TYPE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.CLICKHOUSE_ENGINE_KEY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.CLICKHOUSE_PROJECTIONS_KEY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.ENGINE_PROPERTY_ENTRY;
@@ -107,9 +114,28 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
"Invalid ClickHouse table SETTINGS metadata";
private static final int ERROR_CODE_UNKNOWN_TABLE = 60;
private static final Set<String> REQUIRED_PROJECTION_COLUMNS =
Set.of("name", "type", "query");
- /** Default GRANULARITY for data skipping indexes, matching ClickHouse's own
default. */
+ /** Default GRANULARITY for ordinary data skipping indexes, matching
ClickHouse's own default. */
private static final long DEFAULT_INDEX_GRANULARITY = 1;
+ private static final int DEFAULT_VECTOR_SIMILARITY_GRANULARITY = 100_000_000;
+ private static final int DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER = 32;
+ private static final int DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION =
128;
+ private static final String DEFAULT_VECTOR_SIMILARITY_TYPE = "hnsw";
+ private static final String DEFAULT_VECTOR_SIMILARITY_QUANTIZATION = "bf16";
+ private static final Set<String> VECTOR_SIMILARITY_DISTANCE_FUNCTIONS =
+ Set.of("L2Distance", "cosineDistance");
+ private static final Set<String> VECTOR_SIMILARITY_QUANTIZATIONS =
+ Set.of("f64", "f32", "f16", "bf16", "i8", "b1");
+ private static final Set<String> VECTOR_SIMILARITY_PROPERTIES =
+ Set.of(
+ VECTOR_SIMILARITY_TYPE,
+ VECTOR_SIMILARITY_DISTANCE_FUNCTION,
+ VECTOR_SIMILARITY_DIMENSIONS,
+ VECTOR_SIMILARITY_QUANTIZATION,
+ HNSW_MAX_CONNECTIONS_PER_LAYER,
+ HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION,
+ GRANULARITY);
+
private static final BigInteger MIN_SET_MAX_VALUES = BigInteger.ZERO;
private static final BigInteger MAX_SET_MAX_VALUES =
BigInteger.valueOf(Integer.MAX_VALUE);
private static final String SET_MAX_VALUES_RANGE =
@@ -695,6 +721,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_SET:
case DATA_SKIPPING_NGRAMBFV1:
case DATA_SKIPPING_TOKENBFV1:
+ case DATA_SKIPPING_VECTOR_SIMILARITY:
sqlBuilder
.append(" ")
.append(
@@ -1324,6 +1351,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_SET:
case DATA_SKIPPING_NGRAMBFV1:
case DATA_SKIPPING_TOKENBFV1:
+ case DATA_SKIPPING_VECTOR_SIMILARITY:
return "ADD "
+ buildDataSkippingIndexDdl(
addIndex.getName(), fieldStr, addIndex.getType(), properties);
@@ -2049,11 +2077,34 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
continue;
}
+ if (indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
+ && !includesTypeFull
+ && !StringUtils.contains(parameterSource, "(")) {
+ LOG.warn(
+ "Skip vector similarity index '{}' on {}.{} because the legacy
metadata query "
+ + "does not expose its required parameters",
+ name,
+ databaseName,
+ tableName);
+ continue;
+ }
+
if (indexType == Index.IndexType.DATA_SKIPPING_SET
- || isParameterizedBloomFilterIndex(indexType)) {
+ || isParameterizedBloomFilterIndex(indexType)
+ || indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
{
try {
parameterProperties =
parseIndexPropertiesForQuery(indexType, parameterSource,
name, !includesTypeFull);
+ } catch (UnsupportedVectorSimilarityIndexException e) {
+ LOG.warn(
+ "Skip unsupported vector similarity index '{}' on {}.{} with
{} '{}': {}",
+ name,
+ databaseName,
+ tableName,
+ parameterSourceName,
+ parameterSource,
+ e.getMessage());
+ continue;
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException(
"Failed to load data skipping index '%s' from %s.%s with %s
'%s': %s"
@@ -2072,7 +2123,11 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
// so that indexes created without explicit granularity have empty
properties
// and match the original creation state (avoids false index-change
diffs).
Map<String, String> properties = new HashMap<>();
- if (granularity != DEFAULT_INDEX_GRANULARITY) {
+ long defaultGranularity =
+ indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
+ ? DEFAULT_VECTOR_SIMILARITY_GRANULARITY
+ : DEFAULT_INDEX_GRANULARITY;
+ if (granularity != defaultGranularity) {
properties.put(GRANULARITY, String.valueOf(granularity));
}
if (!includesTypeFull
@@ -2207,6 +2262,11 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_TOKENBFV1:
return parseBloomFilterPropertiesForQuery(
indexType, parameterSource, indexName, allowBareLegacyType);
+ case DATA_SKIPPING_VECTOR_SIMILARITY:
+ if (allowBareLegacyType && !StringUtils.contains(parameterSource,
"(")) {
+ return Collections.emptyMap();
+ }
+ return parseVectorSimilarityProperties(parameterSource, indexName);
default:
return Collections.emptyMap();
}
@@ -2305,6 +2365,185 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return parseBloomFilterProperties(indexType, parameterSource, indexName);
}
+ /**
+ * Parses the positional parameters reported for a ClickHouse vector
similarity index.
+ *
+ * @param typeFull the complete ClickHouse index type expression
+ * @param indexName the index name for validation messages
+ * @return the canonical Gravitino properties for the index
+ * @throws IllegalArgumentException if the type expression or any parameter
is invalid
+ */
+ @VisibleForTesting
+ static Map<String, String> parseVectorSimilarityProperties(String typeFull,
String indexName) {
+ String normalizedTypeFull = StringUtils.trimToEmpty(typeFull);
+ int paramsStart = normalizedTypeFull.indexOf('(');
+ Preconditions.checkArgument(
+ paramsStart > 0 && normalizedTypeFull.endsWith(")"),
+ "Invalid type_full '%s' for vector_similarity index '%s'",
+ typeFull,
+ indexName);
+ Preconditions.checkArgument(
+ StringUtils.equalsIgnoreCase(
+ DATA_SKIPPING_VECTOR_SIMILARITY, normalizedTypeFull.substring(0,
paramsStart).trim()),
+ "type_full '%s' does not match vector_similarity index '%s'",
+ typeFull,
+ indexName);
+
+ String parameterText =
+ normalizedTypeFull.substring(paramsStart + 1,
normalizedTypeFull.length() - 1);
+ List<String> parameters = splitVectorSimilarityParameters(parameterText,
typeFull, indexName);
+ Preconditions.checkArgument(
+ parameters.size() >= 3 && parameters.size() <= 6,
+ "Expected 3 to 6 parameters for vector_similarity index '%s', but got
%s in '%s'",
+ indexName,
+ parameters.size(),
+ typeFull);
+
+ String type =
+ parseVectorSimilarityStringParameter(parameters.get(0),
VECTOR_SIMILARITY_TYPE, indexName);
+ if (!StringUtils.equalsIgnoreCase(DEFAULT_VECTOR_SIMILARITY_TYPE, type)) {
+ throw new UnsupportedVectorSimilarityIndexException(
+ "Unsupported %s '%s' for vector_similarity index '%s'"
+ .formatted(VECTOR_SIMILARITY_TYPE, type, indexName));
+ }
+ String distanceFunction =
+ parseVectorSimilarityStringParameter(
+ parameters.get(1), VECTOR_SIMILARITY_DISTANCE_FUNCTION, indexName);
+ if (!VECTOR_SIMILARITY_DISTANCE_FUNCTIONS.contains(distanceFunction)) {
+ throw new UnsupportedVectorSimilarityIndexException(
+ "Unsupported %s '%s' for vector_similarity index '%s'"
+ .formatted(VECTOR_SIMILARITY_DISTANCE_FUNCTION,
distanceFunction, indexName));
+ }
+ String dimensions =
+ requireIntWithMin(
+ parameters.get(2),
+ VECTOR_SIMILARITY_DIMENSIONS,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 1);
+
+ String quantization = DEFAULT_VECTOR_SIMILARITY_QUANTIZATION;
+ if (parameters.size() >= 4) {
+ quantization =
+ parseVectorSimilarityStringParameter(
+ parameters.get(3), VECTOR_SIMILARITY_QUANTIZATION, indexName);
+ if (!VECTOR_SIMILARITY_QUANTIZATIONS.contains(quantization)) {
+ throw new UnsupportedVectorSimilarityIndexException(
+ "Unsupported %s '%s' for vector_similarity index '%s'"
+ .formatted(VECTOR_SIMILARITY_QUANTIZATION, quantization,
indexName));
+ }
+ }
+
+ int maxConnections = DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER;
+ if (parameters.size() >= 5) {
+ maxConnections =
+ Integer.parseInt(
+ requireIntWithMin(
+ parameters.get(4),
+ HNSW_MAX_CONNECTIONS_PER_LAYER,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 0));
+ }
+
+ int candidateListSize = DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION;
+ if (parameters.size() >= 6) {
+ candidateListSize =
+ Integer.parseInt(
+ requireIntWithMin(
+ parameters.get(5),
+ HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 0));
+ }
+
+ Map<String, String> properties = new HashMap<>();
+ properties.put(VECTOR_SIMILARITY_TYPE, DEFAULT_VECTOR_SIMILARITY_TYPE);
+ properties.put(VECTOR_SIMILARITY_DISTANCE_FUNCTION, distanceFunction);
+ properties.put(VECTOR_SIMILARITY_DIMENSIONS, dimensions);
+ if (!DEFAULT_VECTOR_SIMILARITY_QUANTIZATION.equals(quantization)) {
+ properties.put(VECTOR_SIMILARITY_QUANTIZATION, quantization);
+ }
+ if (maxConnections != 0 && maxConnections !=
DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER) {
+ properties.put(HNSW_MAX_CONNECTIONS_PER_LAYER,
String.valueOf(maxConnections));
+ }
+ if (candidateListSize != 0
+ && candidateListSize !=
DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION) {
+ properties.put(HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION,
String.valueOf(candidateListSize));
+ }
+ return Map.copyOf(properties);
+ }
+
+ private static List<String> splitVectorSimilarityParameters(
+ String parameterText, String typeFull, String indexName) {
+ List<String> parameters = new ArrayList<>();
+ StringBuilder parameter = new StringBuilder();
+ boolean inSingleQuotedString = false;
+ for (int i = 0; i < parameterText.length(); i++) {
+ char current = parameterText.charAt(i);
+ if (current == '\'') {
+ if (inSingleQuotedString
+ && i + 1 < parameterText.length()
+ && parameterText.charAt(i + 1) == '\'') {
+ parameter.append("''");
+ i++;
+ } else {
+ inSingleQuotedString = !inSingleQuotedString;
+ parameter.append(current);
+ }
+ } else if (!inSingleQuotedString && current == ',') {
+ parameters.add(parameter.toString().trim());
+ parameter.setLength(0);
+ } else if (!inSingleQuotedString && (current == '(' || current == ')')) {
+ throw new IllegalArgumentException(
+ "Invalid nested parentheses in type_full '%s' for
vector_similarity index '%s'"
+ .formatted(typeFull, indexName));
+ } else {
+ parameter.append(current);
+ }
+ }
+ Preconditions.checkArgument(
+ !inSingleQuotedString,
+ "Invalid quoted parameter in type_full '%s' for vector_similarity
index '%s'",
+ typeFull,
+ indexName);
+ parameters.add(parameter.toString().trim());
+ Preconditions.checkArgument(
+ parameters.stream().noneMatch(StringUtils::isBlank),
+ "Empty parameter in type_full '%s' for vector_similarity index '%s'",
+ typeFull,
+ indexName);
+ return List.copyOf(parameters);
+ }
+
+ private static String parseVectorSimilarityStringParameter(
+ String rawValue, String propertyName, String indexName) {
+ String value = StringUtils.trimToEmpty(rawValue);
+ if (value.startsWith("'") || value.endsWith("'")) {
+ Preconditions.checkArgument(
+ value.length() >= 2 && value.startsWith("'") && value.endsWith("'"),
+ "Malformed %s parameter '%s' for vector_similarity index '%s'",
+ propertyName,
+ rawValue,
+ indexName);
+ value = value.substring(1, value.length() - 1).replace("''", "'");
+ } else {
+ Preconditions.checkArgument(
+ !StringUtils.contains(value, "'"),
+ "Malformed %s parameter '%s' for vector_similarity index '%s'",
+ propertyName,
+ rawValue,
+ indexName);
+ }
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(value),
+ "%s is required for vector_similarity index '%s'",
+ propertyName,
+ indexName);
+ return value;
+ }
+
private static boolean isParameterizedBloomFilterIndex(Index.IndexType
indexType) {
return indexType == Index.IndexType.DATA_SKIPPING_NGRAMBFV1
|| indexType == Index.IndexType.DATA_SKIPPING_TOKENBFV1;
@@ -2313,11 +2552,11 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
/**
* Maps a ClickHouse data skipping index type string to the corresponding
Gravitino {@link
* Index.IndexType}. Returns {@code DATA_SKIPPING_MINMAX} for blank/null
input (ClickHouse
- * default). Also handles the {@code set(N)} parameterized format that some
ClickHouse versions
- * may return from {@code system.data_skipping_indices}.
+ * default). Also handles parameterized formats that some ClickHouse
versions may return from
+ * {@code system.data_skipping_indices}.
*
* @param rawType the index type string from ClickHouse metadata (e.g.
"minmax", "bloom_filter",
- * "set", "set(0)")
+ * "set", "set(0)", or "vector_similarity")
* @return the corresponding Gravitino IndexType
* @throws IllegalArgumentException if the type is not supported
*/
@@ -2338,10 +2577,11 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Index.IndexType.DATA_SKIPPING_NGRAMBFV1;
case DATA_SKIPPING_TOKENBFV1:
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
+ case DATA_SKIPPING_VECTOR_SIMILARITY:
+ return Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY;
default:
- // ClickHouse may return type with parameters in some versions (e.g.
"set(0)",
- // "ngrambf_v1(3, 512, 3, 0)"). Match on prefix to handle both bare and
- // parameterized formats.
+ // ClickHouse may return type with parameters in some versions. Match
on prefix to handle
+ // both bare and parameterized formats.
if (rawType.startsWith(DATA_SKIPPING_SET + "(")) {
return Index.IndexType.DATA_SKIPPING_SET;
}
@@ -2351,6 +2591,9 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
if (rawType.startsWith(DATA_SKIPPING_TOKENBFV1 + "(")) {
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
}
+ if (rawType.startsWith(DATA_SKIPPING_VECTOR_SIMILARITY + "(")) {
+ return Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY;
+ }
throw new IllegalArgumentException("Unsupported data skipping index
type: " + rawType);
}
}
@@ -2449,11 +2692,20 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
String fieldStr,
Index.IndexType indexType,
Map<String, String> properties) {
+ Map<String, String> safeProperties = properties == null ?
Collections.emptyMap() : properties;
+ int defaultGranularity =
+ indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
+ ? DEFAULT_VECTOR_SIMILARITY_GRANULARITY
+ : (int) DEFAULT_INDEX_GRANULARITY;
+ int granularity =
+ indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
+ ? resolveVectorSimilarityGranularity(safeProperties, indexName)
+ : resolveGranularity(safeProperties, defaultGranularity);
return buildDataSkippingIndexDdl(
indexName,
fieldStr,
- resolveDataSkippingIndexTypeClause(indexType, properties, indexName),
- resolveGranularity(properties, 1));
+ resolveDataSkippingIndexTypeClause(indexType, safeProperties,
indexName),
+ granularity);
}
private String resolveDataSkippingIndexTypeClause(
@@ -2470,12 +2722,143 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return buildBloomFilterTypeClause(properties, DATA_SKIPPING_NGRAMBFV1,
indexName);
case DATA_SKIPPING_TOKENBFV1:
return buildBloomFilterTypeClause(properties, DATA_SKIPPING_TOKENBFV1,
indexName);
+ case DATA_SKIPPING_VECTOR_SIMILARITY:
+ return buildVectorSimilarityTypeClause(properties, indexName);
default:
throw new IllegalArgumentException(
"Gravitino ClickHouse doesn't support index : " + indexType);
}
}
+ private static String buildVectorSimilarityTypeClause(
+ Map<String, String> properties, String indexName) {
+ Map<String, String> safeProperties = properties == null ?
Collections.emptyMap() : properties;
+ for (String propertyName : safeProperties.keySet()) {
+ Preconditions.checkArgument(
+ VECTOR_SIMILARITY_PROPERTIES.contains(propertyName),
+ "Unsupported property '%s' for vector_similarity index '%s'",
+ propertyName,
+ indexName);
+ }
+
+ String type =
+ requireVectorSimilarityProperty(safeProperties,
VECTOR_SIMILARITY_TYPE, indexName);
+ Preconditions.checkArgument(
+ StringUtils.equalsIgnoreCase(DEFAULT_VECTOR_SIMILARITY_TYPE, type),
+ "Unsupported %s '%s' for vector_similarity index '%s'",
+ VECTOR_SIMILARITY_TYPE,
+ type,
+ indexName);
+
+ String distanceFunction =
+ requireVectorSimilarityProperty(
+ safeProperties, VECTOR_SIMILARITY_DISTANCE_FUNCTION, indexName);
+ Preconditions.checkArgument(
+ VECTOR_SIMILARITY_DISTANCE_FUNCTIONS.contains(distanceFunction),
+ "Unsupported %s '%s' for vector_similarity index '%s'",
+ VECTOR_SIMILARITY_DISTANCE_FUNCTION,
+ distanceFunction,
+ indexName);
+
+ String dimensions =
+ requireIntWithMin(
+ safeProperties.get(VECTOR_SIMILARITY_DIMENSIONS),
+ VECTOR_SIMILARITY_DIMENSIONS,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 1);
+
+ String quantization = DEFAULT_VECTOR_SIMILARITY_QUANTIZATION;
+ if (safeProperties.containsKey(VECTOR_SIMILARITY_QUANTIZATION)) {
+ quantization =
+ requireVectorSimilarityProperty(
+ safeProperties, VECTOR_SIMILARITY_QUANTIZATION, indexName);
+ Preconditions.checkArgument(
+ VECTOR_SIMILARITY_QUANTIZATIONS.contains(quantization),
+ "Unsupported %s '%s' for vector_similarity index '%s'",
+ VECTOR_SIMILARITY_QUANTIZATION,
+ quantization,
+ indexName);
+ }
+
+ String maxConnections = null;
+ if (safeProperties.containsKey(HNSW_MAX_CONNECTIONS_PER_LAYER)) {
+ maxConnections =
+ requireIntWithMin(
+ safeProperties.get(HNSW_MAX_CONNECTIONS_PER_LAYER),
+ HNSW_MAX_CONNECTIONS_PER_LAYER,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 0);
+ }
+
+ String candidateListSize = null;
+ if (safeProperties.containsKey(HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION))
{
+ candidateListSize =
+ requireIntWithMin(
+ safeProperties.get(HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION),
+ HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 0);
+ }
+
+ List<String> parameters =
+ new ArrayList<>(List.of("'hnsw'", "'" + distanceFunction + "'",
dimensions));
+ boolean includeQuantization =
+ !DEFAULT_VECTOR_SIMILARITY_QUANTIZATION.equals(quantization)
+ || hasNonDefaultInteger(maxConnections,
DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER)
+ || hasNonDefaultInteger(
+ candidateListSize,
DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION);
+ if (includeQuantization) {
+ parameters.add("'" + quantization + "'");
+ }
+ boolean includeMaxConnections =
+ hasNonDefaultInteger(maxConnections,
DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER)
+ || hasNonDefaultInteger(
+ candidateListSize,
DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION);
+ if (includeMaxConnections) {
+ parameters.add(
+ hasNonDefaultInteger(maxConnections,
DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER)
+ ? maxConnections
+ : String.valueOf(DEFAULT_HNSW_MAX_CONNECTIONS_PER_LAYER));
+ }
+ if (hasNonDefaultInteger(
+ candidateListSize, DEFAULT_HNSW_CANDIDATE_LIST_SIZE_FOR_CONSTRUCTION))
{
+ parameters.add(candidateListSize);
+ }
+ return DATA_SKIPPING_VECTOR_SIMILARITY + "(" + String.join(", ",
parameters) + ")";
+ }
+
+ private static boolean hasNonDefaultInteger(String value, int defaultValue) {
+ return value != null && !value.equals("0") &&
!value.equals(String.valueOf(defaultValue));
+ }
+
+ private static String requireVectorSimilarityProperty(
+ Map<String, String> properties, String propertyName, String indexName) {
+ String value = properties.get(propertyName);
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(value),
+ "%s is required for vector_similarity index '%s'",
+ propertyName,
+ indexName);
+ return value.trim();
+ }
+
+ private static int resolveVectorSimilarityGranularity(
+ Map<String, String> properties, String indexName) {
+ if (!properties.containsKey(GRANULARITY)) {
+ return DEFAULT_VECTOR_SIMILARITY_GRANULARITY;
+ }
+ return Integer.parseInt(
+ requireIntWithMin(
+ properties.get(GRANULARITY),
+ GRANULARITY,
+ DATA_SKIPPING_VECTOR_SIMILARITY,
+ indexName,
+ 1));
+ }
+
private String buildDataSkippingIndexDdl(
String indexName, String fieldStr, String typeName, int granularity) {
Preconditions.checkArgument(
@@ -2640,4 +3023,13 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return sqlBuilder;
}
+
+ private static final class UnsupportedVectorSimilarityIndexException
+ extends IllegalArgumentException {
+ private static final long serialVersionUID = 1L;
+
+ private UnsupportedVectorSimilarityIndexException(String message) {
+ super(message);
+ }
+ }
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
index 0cee91a224..6dd44dd032 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
@@ -636,6 +636,91 @@ public class CatalogClickHouseClusterIT extends BaseIT {
TableChange.deleteIndex("idx_token_alter", false));
}
+ @Test
+ public void testVectorSimilarityIndexOnCluster() {
+ String tableName = GravitinoITUtils.genRandomName("ck_cluster_vector_idx");
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName, tableName);
+ String vectorColumn = "embedding";
+ String customVectorColumn = "embedding_custom";
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+ Map<String, String> defaultProperties =
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3");
+
+ tableCatalog.createTable(
+ tableIdentifier,
+ new Column[] {
+ Column.of("id", Types.IntegerType.get(), "id", false, false,
DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ vectorColumn,
+ Types.ExternalType.of("Array(Float32)"),
+ "embedding",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ customVectorColumn,
+ Types.ExternalType.of("Array(Float32)"),
+ "custom embedding",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ },
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("id"),
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_default",
+ new String[][] {{vectorColumn}},
+ defaultProperties)
+ });
+
+ Table loaded = tableCatalog.loadTable(tableIdentifier);
+ Index defaultIndex =
+ Arrays.stream(loaded.index())
+ .filter(index -> Objects.equals(index.name(),
"idx_vector_default"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
defaultIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{vectorColumn}},
defaultIndex.fieldNames());
+ Assertions.assertEquals(defaultProperties, defaultIndex.properties());
+
+ Map<String, String> customProperties =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "3",
+ "quantization", "f16",
+ "hnsw_max_connections_per_layer", "16",
+ "hnsw_candidate_list_size_for_construction", "64",
+ "granularity", "7");
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_custom",
+ new String[][] {{customVectorColumn}},
+ customProperties));
+
+ Table altered = tableCatalog.loadTable(tableIdentifier);
+ Index customIndex =
+ Arrays.stream(altered.index())
+ .filter(index -> Objects.equals(index.name(), "idx_vector_custom"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
customIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{customVectorColumn}},
customIndex.fieldNames());
+ Assertions.assertEquals(customProperties, customIndex.properties());
+
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.deleteIndex("idx_vector_default", false),
+ TableChange.deleteIndex("idx_vector_custom", false));
+ }
+
private void assertIndexMetadata(
Index[] indexes, String name, Index.IndexType type, Map<String, String>
properties) {
Index index =
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index 09878fb407..c52f77715b 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -91,6 +91,7 @@ import org.apache.gravitino.utils.RandomNameUtils;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
@@ -1688,7 +1689,11 @@ public class CatalogClickHouseIT extends BaseIT {
Assertions.assertEquals(Types.DateType.get(), column.dataType());
break;
case "time_col":
- Assertions.assertEquals(Types.LongType.get(), column.dataType());
+ Assertions.assertEquals(
+
StringUtils.isBlank(System.getenv("GRAVITINO_CLICKHOUSE_TEST_IMAGE"))
+ ? Types.LongType.get()
+ : Types.ExternalType.of("Time"),
+ column.dataType());
break;
case "timestamp_col":
Assertions.assertEquals(Types.TimestampType.withoutTimeZone(0),
column.dataType());
@@ -3891,4 +3896,161 @@ public class CatalogClickHouseIT extends BaseIT {
normalizedCreateSql.contains("tokenbf_v1(256,2,0)"),
"SHOW CREATE TABLE should retain tokenbf_v1 parameters: " + createSql);
}
+
+ @Test
+ @Tag("gravitino-docker-test")
+ void testVectorSimilarityIndex() {
+ String tableName =
GravitinoITUtils.genRandomName("ch_vector_similarity_idx_");
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName, tableName);
+ String vectorColumn = "embedding";
+ String customVectorColumn = "embedding_custom";
+ String zeroVectorColumn = "embedding_zero";
+ Column[] columns =
+ new Column[] {
+ Column.of(
+ CLICKHOUSE_COL_NAME1,
+ Types.IntegerType.get(),
+ "id",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ vectorColumn,
+ Types.ExternalType.of("Array(Float32)"),
+ "embedding",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ customVectorColumn,
+ Types.ExternalType.of("Array(Float32)"),
+ "custom embedding",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ zeroVectorColumn,
+ Types.ExternalType.of("Array(Float32)"),
+ "zero parameter embedding",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ };
+ Map<String, String> defaultProperties =
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3");
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ tableCatalog.createTable(
+ tableIdentifier,
+ columns,
+ table_comment,
+ createProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders(CLICKHOUSE_COL_NAME1),
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_default",
+ new String[][] {{vectorColumn}},
+ defaultProperties)
+ });
+
+ Table loaded = tableCatalog.loadTable(tableIdentifier);
+ Index defaultIndex =
+ Arrays.stream(loaded.index())
+ .filter(index -> Objects.equals(index.name(),
"idx_vector_default"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
defaultIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{vectorColumn}},
defaultIndex.fieldNames());
+ Assertions.assertEquals(defaultProperties, defaultIndex.properties());
+
+ Map<String, String> customProperties =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "3",
+ "quantization", "i8",
+ "hnsw_max_connections_per_layer", "16",
+ "hnsw_candidate_list_size_for_construction", "64",
+ "granularity", "7");
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_custom",
+ new String[][] {{customVectorColumn}},
+ customProperties));
+
+ clickhouseService.executeQuery(
+ "ALTER TABLE `%s`.`%s` ADD INDEX `idx_vector_zero` `%s` TYPE "
+ .formatted(schemaName, tableName, zeroVectorColumn)
+ + "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 0, 0) "
+ + "GRANULARITY 100000000");
+
+ Table altered = tableCatalog.loadTable(tableIdentifier);
+ Index customIndex =
+ Arrays.stream(altered.index())
+ .filter(index -> Objects.equals(index.name(), "idx_vector_custom"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
customIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{customVectorColumn}},
customIndex.fieldNames());
+ Assertions.assertEquals(customProperties, customIndex.properties());
+
+ Index zeroValueIndex =
+ Arrays.stream(altered.index())
+ .filter(index -> Objects.equals(index.name(), "idx_vector_zero"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertArrayEquals(new String[][] {{zeroVectorColumn}},
zeroValueIndex.fieldNames());
+ Assertions.assertEquals(
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3"),
+ zeroValueIndex.properties());
+
+ String createSql =
+ clickhouseService.executeQueryForResult(
+ String.format("SHOW CREATE TABLE `%s`.`%s`", schemaName,
tableName));
+ String normalizedCreateSql = createSql.replaceAll("\\s+", "");
+ Assertions.assertTrue(
+
normalizedCreateSql.contains("vector_similarity('hnsw','L2Distance',3)"),
createSql);
+ Assertions.assertTrue(
+
normalizedCreateSql.contains("vector_similarity('hnsw','cosineDistance',3,'i8',16,64)"),
+ createSql);
+
Assertions.assertTrue(normalizedCreateSql.contains("GRANULARITY100000000"),
createSql);
+ Assertions.assertTrue(normalizedCreateSql.contains("GRANULARITY7"),
createSql);
+ }
+
+ @Test
+ @Tag("gravitino-docker-test")
+ void testLoadTableWithUnsupportedNativeDotProductVectorIndex() {
+ String[] version = clickhouseService.executeQueryForResult("SELECT
version()").split("\\.");
+ int majorVersion = Integer.parseInt(version[0]);
+ int minorVersion = Integer.parseInt(version[1]);
+ Assumptions.assumeTrue(
+ majorVersion > 26 || (majorVersion == 26 && minorVersion >= 4),
+ "Native dotProduct vector indexes require ClickHouse 26.4 or later");
+
+ String nativeTableName =
GravitinoITUtils.genRandomName("ch_native_dot_vector_idx_");
+ clickhouseService.executeQuery(
+ ("CREATE TABLE `%s`.`%s` ("
+ + "id Int32, embedding_l2 Array(Float32), embedding_dot
Array(Float32), "
+ + "INDEX idx_minmax id TYPE minmax GRANULARITY 1, "
+ + "INDEX idx_l2 embedding_l2 TYPE "
+ + "vector_similarity('hnsw', 'L2Distance', 3) GRANULARITY
100000000, "
+ + "INDEX idx_dot embedding_dot TYPE "
+ + "vector_similarity('hnsw', 'dotProduct', 3) GRANULARITY
100000000) "
+ + "ENGINE = MergeTree ORDER BY id")
+ .formatted(schemaName, nativeTableName));
+
+ Table loaded =
+ catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName,
nativeTableName));
+ Assertions.assertTrue(
+ Arrays.stream(loaded.index()).anyMatch(index ->
"idx_minmax".equals(index.name())));
+ Assertions.assertTrue(
+ Arrays.stream(loaded.index()).anyMatch(index ->
"idx_l2".equals(index.name())));
+ Assertions.assertFalse(
+ Arrays.stream(loaded.index()).anyMatch(index ->
"idx_dot".equals(index.name())));
+ }
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
index fb915f4136..1759dce3df 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
@@ -94,6 +94,11 @@ public class TestClickHouseTableOperationsUnit {
}
String callGenerateCreateTableSql(JdbcColumn[] columns, Map<String,
String> properties) {
+ return callGenerateCreateTableSql(columns, properties,
Indexes.EMPTY_INDEXES);
+ }
+
+ String callGenerateCreateTableSql(
+ JdbcColumn[] columns, Map<String, String> properties, Index[] indexes)
{
return generateCreateTableSql(
"test_table",
columns,
@@ -101,7 +106,7 @@ public class TestClickHouseTableOperationsUnit {
properties,
Transforms.EMPTY_TRANSFORM,
Distributions.NONE,
- Indexes.EMPTY_INDEXES,
+ indexes,
getSortOrders("id"));
}
@@ -1360,6 +1365,85 @@ public class TestClickHouseTableOperationsUnit {
.isEmpty());
}
+ @Test
+ void testParseVectorSimilarityPropertiesAndNormalizesDefaults() {
+ Assertions.assertEquals(
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3"),
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ " vector_similarity ( 'hnsw' , 'L2Distance' , 003 ) ",
"idx_vector"));
+ Assertions.assertEquals(
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "768",
+ "quantization", "i8"),
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ "vector_similarity('hnsw', 'cosineDistance', 768, 'i8')",
"idx_vector"));
+ Assertions.assertEquals(
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "hnsw_max_connections_per_layer", "16"),
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 16)",
"idx_vector"));
+ for (String quantization : List.of("f64", "f32", "f16", "i8", "b1")) {
+ Assertions.assertEquals(
+ quantization,
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ "vector_similarity('hnsw', 'L2Distance', 3, '" +
quantization + "')",
+ "idx_vector")
+ .get("quantization"));
+ }
+ Assertions.assertEquals(
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3"),
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 0, 0)",
"idx_vector"));
+ ExposedClickHouseTableOperations ops = newOps();
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ ops.getClickHouseIndexType("vector_similarity"));
+ Assertions.assertEquals(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ ops.getClickHouseIndexType("vector_similarity('hnsw', 'L2Distance',
3)"));
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
ops.getClickHouseIndexType("annoy"));
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
ops.getClickHouseIndexType("usearch"));
+ }
+
+ @Test
+ void
testParseVectorSimilarityPropertiesRejectsMalformedAndUnsupportedParameters() {
+ List<String> invalidTypeFullValues =
+ List.of(
+ "vector_similarity",
+ "wrong_type('hnsw', 'L2Distance', 3)",
+ "vector_similarity('ivf', 'L2Distance', 3)",
+ "vector_similarity('hnsw', 'euclidean', 3)",
+ "vector_similarity('hnsw', 'dotProduct', 384, 'bf16', 16, 64)",
+ "vector_similarity('hnsw', , 3)",
+ "vector_similarity('hnsw', 'L2Distance', 0)",
+ "vector_similarity('hnsw', 'L2Distance', not_an_integer)",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'float8')",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', -1)",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 32, -1)",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 32, 128, 1)",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 32,
2147483648)",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 32",
+ "vector_similarity('hnsw', 'L2Distance', 3, 'bf16', 32, 128)
trailing");
+
+ for (String typeFull : invalidTypeFullValues) {
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ClickHouseTableOperations.parseVectorSimilarityProperties(
+ typeFull, "idx_vector_bad"),
+ typeFull);
+ Assertions.assertTrue(exception.getMessage().contains("idx_vector_bad"),
typeFull);
+ }
+ }
+
@Test
void testParseSetPropertiesAcceptsIntegerMaxValue() {
Assertions.assertEquals(
@@ -1636,6 +1720,373 @@ public class TestClickHouseTableOperationsUnit {
Assertions.assertTrue(bare.properties().isEmpty());
}
+ @Test
+ void testGetIndexesFallsBackAndReadsLegacyVectorSimilarityParameters()
throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement modernSecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ PreparedStatement legacySecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ ResultSet legacySecondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(modernSecondaryStmt.executeQuery())
+ .thenThrow(new SQLException("Unknown identifier 'type_full'"));
+
Mockito.when(legacySecondaryStmt.executeQuery()).thenReturn(legacySecondaryRs);
+ Mockito.when(legacySecondaryRs.next()).thenReturn(true, true, true, false);
+ Mockito.when(legacySecondaryRs.getString("name"))
+ .thenReturn("idx_legacy_vector", "idx_legacy_vector_custom",
"idx_legacy_vector_bare");
+ Mockito.when(legacySecondaryRs.getString("type"))
+ .thenReturn(
+ "vector_similarity('hnsw', 'cosineDistance', 3, 'i8', 16, 64)",
+ "vector_similarity('hnsw', 'L2Distance', 3)",
+ "vector_similarity");
+ Mockito.when(legacySecondaryRs.getString("expr"))
+ .thenReturn("embedding", "embedding_custom", "embedding_bare");
+ Mockito.when(legacySecondaryRs.getLong("granularity"))
+ .thenReturn(100_000_000L, 7L, 100_000_000L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(modernSecondaryStmt)
+ .thenReturn(legacySecondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(2, indexes.size());
+ Index legacyIndex =
+ indexes.stream()
+ .filter(index -> "idx_legacy_vector".equals(index.name()))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
legacyIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
legacyIndex.fieldNames());
+ Assertions.assertEquals(
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "3",
+ "quantization", "i8",
+ "hnsw_max_connections_per_layer", "16",
+ "hnsw_candidate_list_size_for_construction", "64"),
+ legacyIndex.properties());
+
+ Index customGranularityIndex =
+ indexes.stream()
+ .filter(index -> "idx_legacy_vector_custom".equals(index.name()))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "granularity", "7"),
+ customGranularityIndex.properties());
+ }
+
+ @Test
+ void
testGetIndexesReadsVectorSimilarityTypeFullAndUsesItsDefaultGranularity()
throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name"))
+ .thenReturn("idx_vector_default", "idx_vector_custom",
"idx_vector_expression");
+ Mockito.when(secondaryRs.getString("type"))
+ .thenReturn("vector_similarity", "vector_similarity",
"vector_similarity");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn(
+ "vector_similarity('hnsw', 'L2Distance', 3)",
+ "vector_similarity('hnsw', 'cosineDistance', 3, 'i8', 16, 64)",
+ "vector_similarity('hnsw', 'L2Distance', 3)");
+ Mockito.when(secondaryRs.getString("expr"))
+ .thenReturn("embedding", "embedding_custom", "lower(embedding)");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(100_000_000L,
7L, 100_000_000L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(2, indexes.size());
+ Index defaultIndex =
+ indexes.stream()
+ .filter(index -> "idx_vector_default".equals(index.name()))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
defaultIndex.type());
+ Assertions.assertArrayEquals(new String[][] {{"embedding"}},
defaultIndex.fieldNames());
+ Assertions.assertEquals(
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3"),
+ defaultIndex.properties());
+
+ Index customIndex =
+ indexes.stream()
+ .filter(index -> "idx_vector_custom".equals(index.name()))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "3",
+ "quantization", "i8",
+ "hnsw_max_connections_per_layer", "16",
+ "hnsw_candidate_list_size_for_construction", "64",
+ "granularity", "7"),
+ customIndex.properties());
+ Assertions.assertFalse(
+ indexes.stream().anyMatch(index ->
"idx_vector_expression".equals(index.name())));
+ }
+
+ @Test
+ void
testGetIndexesSkipsUnsupportedVectorSimilarityWithoutDroppingOtherIndexes()
+ throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name")).thenReturn("idx_dot",
"idx_l2", "idx_minmax");
+ Mockito.when(secondaryRs.getString("type"))
+ .thenReturn("vector_similarity", "vector_similarity", "minmax");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn(
+ "vector_similarity('hnsw', 'dotProduct', 3)",
+ "vector_similarity('hnsw', 'L2Distance', 3)",
+ "minmax");
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("embedding_dot",
"embedding_l2", "id");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(100_000_000L,
100_000_000L, 1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+ Assertions.assertEquals(2, indexes.size());
+ Assertions.assertEquals("idx_l2", indexes.get(0).name());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
indexes.get(0).type());
+ Assertions.assertEquals("idx_minmax", indexes.get(1).name());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_MINMAX,
indexes.get(1).type());
+ }
+
+ @Test
+ void testGetIndexesRejectsMalformedVectorSimilarityMetadata() throws
Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name")).thenReturn("idx_malformed");
+
Mockito.when(secondaryRs.getString("type")).thenReturn("vector_similarity");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn("vector_similarity('hnsw', 'L2Distance', not_an_integer)");
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("embedding");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(100_000_000L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
ops.callGetIndexes(connection, "db", "tbl"));
+ Assertions.assertTrue(exception.getMessage().contains("idx_malformed"));
+ Assertions.assertTrue(exception.getMessage().contains("type_full"));
+ }
+
+ @Test
+ void testVectorSimilarityCreateAndAlterDdlUseTheSameTypeClause() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build(),
+ JdbcColumn.builder()
+ .withName("embedding")
+ .withType(Types.StringType.get())
+ .withNullable(true)
+ .build(),
+ };
+ Map<String, String> defaultProperties =
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3");
+ Map<String, String> customProperties =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "cosineDistance",
+ "dimensions", "3",
+ "quantization", "i8",
+ "hnsw_candidate_list_size_for_construction", "64",
+ "granularity", "7");
+ Map<String, String> zeroHnswProperties =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "hnsw_max_connections_per_layer", "0",
+ "hnsw_candidate_list_size_for_construction", "0");
+
+ String createSql =
+ newOps()
+ .callGenerateCreateTableSql(
+ columns,
+ Map.of(),
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_default",
+ new String[][] {{"embedding"}},
+ defaultProperties),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX, "idx_minmax", new
String[][] {{"id"}})
+ });
+ ExposedClickHouseTableOperations alterOps = newAlterOps(Map.of());
+ String alterSql =
+ alterOps.callGenerateAlterTableSql(
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_custom",
+ new String[][] {{"embedding"}},
+ customProperties));
+ String zeroHnswAlterSql =
+ alterOps.callGenerateAlterTableSql(
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_zero_hnsw",
+ new String[][] {{"embedding"}},
+ zeroHnswProperties));
+
+ Assertions.assertTrue(
+ createSql.contains(
+ "INDEX `idx_vector_default` `embedding` TYPE "
+ + "vector_similarity('hnsw', 'L2Distance', 3) GRANULARITY
100000000"),
+ createSql);
+ Assertions.assertTrue(
+ createSql.contains("INDEX `idx_minmax` `id` TYPE minmax GRANULARITY
1"), createSql);
+ Assertions.assertTrue(
+ alterSql.contains(
+ "ADD INDEX `idx_vector_custom` `embedding` TYPE "
+ + "vector_similarity('hnsw', 'cosineDistance', 3, 'i8', 32,
64) GRANULARITY 7"),
+ alterSql);
+ Assertions.assertTrue(
+ zeroHnswAlterSql.contains(
+ "ADD INDEX `idx_vector_zero_hnsw` `embedding` TYPE "
+ + "vector_similarity('hnsw', 'L2Distance', 3) GRANULARITY
100000000"),
+ zeroHnswAlterSql);
+ }
+
+ @Test
+ void testVectorSimilarityInvalidPropertiesFailBeforeCreateOrAlterDdl() {
+ Map<String, String> missingRequiredProperty = Map.of("type", "hnsw",
"dimensions", "3");
+ Map<String, String> unsupportedDistance =
+ Map.of("type", "hnsw", "distance_function", "dotProduct",
"dimensions", "3");
+ Map<String, String> unsupportedQuantization =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "quantization", "float8");
+ Map<String, String> negativeHnswValue =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "hnsw_max_connections_per_layer", "-1");
+ Map<String, String> zeroGranularity =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "granularity", "0");
+ Map<String, String> unknownProperty =
+ Map.of(
+ "type", "hnsw",
+ "distance_function", "L2Distance",
+ "dimensions", "3",
+ "unexpected_property", "value");
+
+ List<Map.Entry<String, Map<String, String>>> invalidCases =
+ List.of(
+ Map.entry("distance_function", missingRequiredProperty),
+ Map.entry("distance_function", unsupportedDistance),
+ Map.entry("quantization", unsupportedQuantization),
+ Map.entry("hnsw_max_connections_per_layer", negativeHnswValue),
+ Map.entry("granularity", zeroGranularity),
+ Map.entry("unexpected_property", unknownProperty));
+ for (Map.Entry<String, Map<String, String>> invalid : invalidCases) {
+ Map<String, String> properties = invalid.getValue();
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_invalid",
+ new String[][] {{"embedding"}},
+ properties)
+ };
+ IllegalArgumentException createException =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ newOps()
+ .callGenerateCreateTableSql(
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build(),
+ JdbcColumn.builder()
+ .withName("embedding")
+ .withType(Types.StringType.get())
+ .withNullable(true)
+ .build(),
+ },
+ Map.of(),
+ indexes));
+
Assertions.assertTrue(createException.getMessage().contains("idx_vector_invalid"));
+
Assertions.assertTrue(createException.getMessage().contains(invalid.getKey()));
+
+ IllegalArgumentException alterException =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ newAlterOps(Map.of())
+ .callGenerateAlterTableSql(
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "idx_vector_invalid",
+ new String[][] {{"embedding"}},
+ properties)));
+
Assertions.assertTrue(alterException.getMessage().contains("idx_vector_invalid"));
+
Assertions.assertTrue(alterException.getMessage().contains(invalid.getKey()));
+ }
+ }
+
@Test
void testGetIndexesDoesNotFallbackForOtherSqlErrors() throws Exception {
ExposedClickHouseTableOperations ops = newOps();
diff --git a/clients/client-python/gravitino/api/rel/indexes/index.py
b/clients/client-python/gravitino/api/rel/indexes/index.py
index 777e3b4813..d55929a040 100644
--- a/clients/client-python/gravitino/api/rel/indexes/index.py
+++ b/clients/client-python/gravitino/api/rel/indexes/index.py
@@ -24,14 +24,14 @@ from typing import List, Optional
class Index(ABC):
"""The Index interface defines methods for implementing table index
columns.
- Currently, settings for `PRIMARY_KEY` and `UNIQUE_KEY` are provided.
+ Catalogs may provide key and connector-specific index types.
"""
@unique
class IndexType(str, Enum):
"""The enum IndexType defines the type of the index.
- Currently, `PRIMARY_KEY` and `UNIQUE_KEY` are supported.
+ Catalogs may support generic key types and connector-specific index
types.
"""
PRIMARY_KEY = "PRIMARY_KEY"
@@ -50,6 +50,9 @@ class Index(ABC):
columns that are not designated as the primary key. The uniqueness
constraint imposed by
UNIQUE KEY helps in avoiding redundancy and ensuring data accuracy in
the database."""
+ DATA_SKIPPING_VECTOR_SIMILARITY = "DATA_SKIPPING_VECTOR_SIMILARITY"
+ """ClickHouse vector similarity data skipping index."""
+
@abstractmethod
def type(self) -> IndexType:
"""Returns the type of the index. eg: `PRIMARY_KEY` and `UNIQUE_KEY`.
@@ -79,3 +82,7 @@ class Index(ABC):
List[List[str]]: The field name under the table contained in the
index.
"""
pass # pragma: no cover
+
+ def properties(self) -> dict[str, str]:
+ """Returns the index properties, or an empty map when none are
defined."""
+ return {}
diff --git a/clients/client-python/gravitino/dto/rel/indexes/index_dto.py
b/clients/client-python/gravitino/dto/rel/indexes/index_dto.py
index 4842806e07..dbfa3240c3 100644
--- a/clients/client-python/gravitino/dto/rel/indexes/index_dto.py
+++ b/clients/client-python/gravitino/dto/rel/indexes/index_dto.py
@@ -70,10 +70,13 @@ class IndexDTO(Index):
self._index_type is other.type()
and self._name == other.name()
and self._field_names == other.field_names()
+ and self._properties == other.properties()
)
def __hash__(self) -> int:
- initial_hash = hash((self._index_type, self._name))
+ initial_hash = hash(
+ (self._index_type, self._name, frozenset(self._properties.items()))
+ )
return reduce(
lambda result, field_name: 31 * result + hash(tuple(field_name)),
self._field_names,
diff --git
a/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
b/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
index 6110a09409..eeca69bca2 100644
---
a/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
+++
b/clients/client-python/gravitino/dto/rel/indexes/json_serdes/index_serdes.py
@@ -31,6 +31,8 @@ class IndexSerdes(SerdesUtilsBase, JsonSerializable[Index]):
if value.name() is not None:
result[cls.INDEX_NAME] = value.name()
result[cls.INDEX_FIELD_NAMES] = value.field_names()
+ if value.properties():
+ result["properties"] = value.properties()
return result
@@ -51,5 +53,8 @@ class IndexSerdes(SerdesUtilsBase, JsonSerializable[Index]):
index_type = Index.IndexType(data[cls.INDEX_TYPE].upper())
return IndexDTO(
- index_type, data.get(cls.INDEX_NAME), data[cls.INDEX_FIELD_NAMES]
+ index_type,
+ data.get(cls.INDEX_NAME),
+ data[cls.INDEX_FIELD_NAMES],
+ data.get("properties"),
)
diff --git a/clients/client-python/tests/unittests/dto/rel/test_index_dto.py
b/clients/client-python/tests/unittests/dto/rel/test_index_dto.py
index 9a17862a23..a28926c1cb 100644
--- a/clients/client-python/tests/unittests/dto/rel/test_index_dto.py
+++ b/clients/client-python/tests/unittests/dto/rel/test_index_dto.py
@@ -70,3 +70,29 @@ class TestIndexDTO(unittest.TestCase):
dto_dict[slack_dto] = len(names)
self.assertEqual(len(dto_dict), len(names))
self.assertEqual(dto_dict[slack_dto], len(names))
+
+ def test_index_dto_properties_are_part_of_equality_and_hash(self):
+ field_names = [["embedding"]]
+ l2_index = IndexDTO(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "vector_index",
+ field_names,
+ {"distance_function": "L2Distance", "dimensions": "3"},
+ )
+ cosine_index = IndexDTO(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "vector_index",
+ field_names,
+ {"distance_function": "cosineDistance", "dimensions": "3"},
+ )
+ same_l2_index = IndexDTO(
+ Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY,
+ "vector_index",
+ field_names,
+ {"dimensions": "3", "distance_function": "L2Distance"},
+ )
+
+ self.assertNotEqual(l2_index, cosine_index)
+ self.assertEqual(l2_index, same_l2_index)
+ self.assertEqual(hash(l2_index), hash(same_l2_index))
+ self.assertEqual(len({l2_index, cosine_index}), 2)
diff --git a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
index 790f102a24..18b960b87c 100644
--- a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
+++ b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
@@ -116,3 +116,32 @@ class TestIndexSerdes(unittest.TestCase):
json_dict = json.loads(json_string)
serialized_dict = json.loads(mock_data_class.to_json())
self.assertDictEqual(json_dict, serialized_dict)
+
+ def test_vector_similarity_index_properties_round_trip(self):
+ json_string = """
+ {
+ "indexes": [
+ {
+ "indexType": "DATA_SKIPPING_VECTOR_SIMILARITY",
+ "name": "idx_vector",
+ "fieldNames": [["embedding"]],
+ "properties": {
+ "type": "hnsw",
+ "distance_function": "L2Distance",
+ "dimensions": "3"
+ }
+ }
+ ]
+ }
+ """
+
+ index = MockDataClass.from_json(json_string).indexes[0]
+ self.assertEqual(index.type(),
Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
+ self.assertEqual(
+ index.properties(),
+ {"type": "hnsw", "distance_function": "L2Distance", "dimensions":
"3"},
+ )
+ self.assertDictEqual(
+ json.loads(json_string),
+ json.loads(MockDataClass(indexes=[index]).to_json()),
+ )
diff --git a/clients/client-python/tests/unittests/test_relational_catalog.py
b/clients/client-python/tests/unittests/test_relational_catalog.py
index 86a3edc269..21e779980d 100644
--- a/clients/client-python/tests/unittests/test_relational_catalog.py
+++ b/clients/client-python/tests/unittests/test_relational_catalog.py
@@ -15,6 +15,7 @@
# specific language governing permissions and limitations
# under the License.
+import json
import unittest
from http.client import HTTPResponse
from unittest.mock import Mock, patch
@@ -22,6 +23,7 @@ from unittest.mock import Mock, patch
from gravitino.api.authorization.privileges import Privilege
from gravitino.api.rel.column import Column
from gravitino.api.rel.dialects import Dialects
+from gravitino.api.rel.indexes.index import Index
from gravitino.api.rel.sql_representation import SQLRepresentation
from gravitino.api.rel.table_change import TableChange
from gravitino.api.rel.types.types import Types
@@ -151,6 +153,37 @@ class TestRelationalCatalog(unittest.TestCase):
table = self.catalog.load_table(self.table_identifier)
self.assertEqual(table.name(), self.table_dto.name())
+ def test_load_table_with_vector_similarity_index(self):
+ response_json = json.loads(TableResponse(0, self.table_dto).to_json())
+ response_json["table"]["indexes"] = [
+ {
+ "indexType": "DATA_SKIPPING_VECTOR_SIMILARITY",
+ "name": "idx_vector",
+ "fieldNames": [["id"]],
+ "properties": {
+ "type": "hnsw",
+ "distance_function": "L2Distance",
+ "dimensions": "3",
+ },
+ }
+ ]
+ mock_resp = self._get_mock_http_resp(json.dumps(response_json))
+
+ with patch(
+ "gravitino.utils.http_client.HTTPClient.get",
return_value=mock_resp
+ ):
+ table = self.catalog.load_table(self.table_identifier)
+
+ self.assertEqual(table.name(), self.table_dto.name())
+ self.assertEqual(len(table.index()), 1)
+ self.assertEqual(
+ table.index()[0].type(),
Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY
+ )
+ self.assertEqual(
+ table.index()[0].properties(),
+ {"type": "hnsw", "distance_function": "L2Distance", "dimensions":
"3"},
+ )
+
def test_load_table_with_required_privilege_names(self):
resp_body = TableResponse(0, self.table_dto)
mock_resp = self._get_mock_http_resp(resp_body.to_json())
diff --git a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
index 2649667c4b..4d836e4bf6 100644
--- a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
+++ b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
@@ -511,6 +511,22 @@ public class TestJsonUtils {
Assertions.assertEquals(
objectMapper.readValue(expected, IndexDTO.class),
objectMapper.readValue(jsonValue, IndexDTO.class));
+
+ Index vectorIndex =
+ IndexDTO.builder()
+ .withIndexType(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
+ .withName("idx_vector")
+ .withFieldNames(new String[][] {{"embedding"}})
+ .withProperties(
+ Map.of("type", "hnsw", "distance_function", "L2Distance",
"dimensions", "3"))
+ .build();
+ jsonValue = JsonUtils.objectMapper().writeValueAsString(vectorIndex);
+ expected =
+
"{\"indexType\":\"DATA_SKIPPING_VECTOR_SIMILARITY\",\"name\":\"idx_vector\","
+ +
"\"fieldNames\":[[\"embedding\"]],\"properties\":{\"dimensions\":\"3\","
+ + "\"distance_function\":\"L2Distance\",\"type\":\"hnsw\"}}";
+ Assertions.assertEquals(expected, jsonValue);
+ Assertions.assertEquals(vectorIndex, objectMapper.readValue(jsonValue,
IndexDTO.class));
}
@Test
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index abbf4d6b8b..fc86b677aa 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -37,6 +37,8 @@ ClickHouse catalog is not included in the standard Gravitino
server distribution
|--------------------------------|-------------------------------------------|
| `24.8.x` (including `24.8.14`) | `0.7.1 ~ 0.8.4` |
+The `DATA_SKIPPING_VECTOR_SIMILARITY` index requires ClickHouse `25.8` or
later. This feature-specific requirement does not change the compatibility
baseline for other catalog capabilities.
+
:::tip
For other ClickHouse versions (not 24.8.x), the required JDBC driver version
may be different.
Use your staging validation results and the official ClickHouse documentation
as the final reference.
@@ -172,7 +174,7 @@ See [Manage Catalogs and
Schemas](./manage-catalogs-and-schemas.md#schema-operat
| Mapping | Gravitino table maps to a ClickHouse table
|
| Engines | **MergeTree family** (`MergeTree` default,
`ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`,
`CollapsingMergeTree`, `VersionedCollapsingMergeTree`, `GraphiteMergeTree`):
fully supported, data persists across restarts. **Log family** (`TinyLog`,
`StripeLog`, `Log`): supported, data and table definition persist across
restarts. **`Null`**: supported, table persists, data is always discarded by
design. **`Set`**: supported, table definition persists. [...]
| Ordering/Partition | MergeTree-family requires exactly one `ORDER BY`
column; `PARTITION BY` supports single-column identity and the function
expressions listed below. Other engines reject `ORDER BY`/`PARTITION BY`.
|
-| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, `DATA_SKIPPING_SET`,
`DATA_SKIPPING_NGRAMBFV1`, and `DATA_SKIPPING_TOKENBFV1` (configurable
granularity via `Index.properties()`).
[...]
+| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, `DATA_SKIPPING_SET`,
`DATA_SKIPPING_NGRAMBFV1`, `DATA_SKIPPING_TOKENBFV1`, and
`DATA_SKIPPING_VECTOR_SIMILARITY` (configurable granularity via
`Index.properties()`).
[...]
| Distribution | Gravitino enforces `Distributions.NONE`; no custom
distribution strategies.
|
| Column defaults | Supported.
|
| Unsupported | Engine and connector-owned property changes after
creation; mixing table setting changes with schema changes; auto-increment
columns.
|
@@ -291,11 +293,20 @@ The `engine_parameters` property applies to
`ReplacingMergeTree`, `SummingMergeT
- `DATA_SKIPPING_SET` (default `GRANULARITY 1`, plus configurable `set(N)`
max values)
- `DATA_SKIPPING_NGRAMBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `ngram_size`, `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
- `DATA_SKIPPING_TOKENBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
+ - `DATA_SKIPPING_VECTOR_SIMILARITY` (requires ClickHouse 25.8+; default
`GRANULARITY 100000000`; requires `type=hnsw`, `distance_function`, and
`dimensions` in `Index.properties()`)
+
+ `DATA_SKIPPING_VECTOR_SIMILARITY` supports `distance_function` values
`L2Distance` and `cosineDistance` on ClickHouse 25.8+. Optional properties are
`quantization` (`f64`, `f32`, `f16`, `bf16`, `i8`, or `b1`; default `bf16`),
`hnsw_max_connections_per_layer` (default 32), and
`hnsw_candidate_list_size_for_construction` (default 128). A value of 0 for
either HNSW numeric property requests the ClickHouse default. Custom
`granularity` must be a positive integer. See the [ClickHouse vector [...]
+
+ When adding a data-skipping index to an existing table through Gravitino's
`ALTER TABLE ADD INDEX`, existing data parts are not synchronously backfilled,
and Gravitino does not issue a materialization command. To materialize the
index on existing data, run `ALTER TABLE <table> MATERIALIZE INDEX
<index_name>` directly in ClickHouse. This can take time, especially for vector
indexes; ClickHouse may also materialize indexes during later background
merges, depending on its settings.
Custom `GRANULARITY` can be specified via the `Index.properties()` API (key
`granularity`, value must be a positive integer). For `DATA_SKIPPING_SET`, the
max unique values can be configured via `set_max_values` (non-negative
integer). If not specified, the defaults above apply.
On ClickHouse versions without `system.data_skipping_indices.type_full`,
Gravitino falls back to the legacy metadata query. If the legacy `type` value
does not include the bloom-filter parameters, the index type and fields are
preserved but the required parameter properties cannot be reconstructed;
provide the properties explicitly before recreating the table.
+ For `DATA_SKIPPING_VECTOR_SIMILARITY`, Gravitino restores parameters from
the legacy `type` value when present. If the legacy value is bare and cannot
provide the required parameters, the index is skipped with a warning.
+
+ ClickHouse 26.4 and later can create native `vector_similarity` indexes with
`dotProduct`. Gravitino skips native vector indexes with unsupported parameter
values, including `dotProduct`, and logs a warning when loading the table.
Skipped indexes are absent from the returned table metadata and will not be
recreated from it. Creating or adding a `dotProduct` vector index through
Gravitino remains unsupported.
+
ClickHouse data-skipping indexes whose field expressions cannot be
represented as Gravitino field names, such as `lower(name)` or `name + 1`, are
skipped with a warning when the table is loaded and are not recreated. Direct
column references and tuples containing only column references remain supported.
### Partitioning, Sorting, and Distribution
diff --git a/docs/open-api/indexes.yaml b/docs/open-api/indexes.yaml
index af250b8967..cbcdbde96a 100644
--- a/docs/open-api/indexes.yaml
+++ b/docs/open-api/indexes.yaml
@@ -40,6 +40,7 @@ components:
- "data_skipping_set"
- "data_skipping_ngrambfv1"
- "data_skipping_tokenbfv1"
+ - "data_skipping_vector_similarity"
name:
type: string
description: The name of the index
diff --git a/docs/table-partitioning-distribution-sort-order-indexes.md
b/docs/table-partitioning-distribution-sort-order-indexes.md
index 65025ec836..6ce5b419ec 100644
--- a/docs/table-partitioning-distribution-sort-order-indexes.md
+++ b/docs/table-partitioning-distribution-sort-order-indexes.md
@@ -253,6 +253,7 @@ To define an indexed table, you should utilize the
following four components to
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|---------------------------|
| PRIMARY_KEY | The PRIMARY KEY is a column or set of columns that uniquely
identifies each row in a table. It enforces uniqueness and ensures that no two
rows have the same values in the specified columns. Additionally, the PRIMARY
KEY constraint automatically creates a unique index on the specified columns. |
`PRIMARY_KEY` | `IndexType.PRIMARY_KEY` |
| UNIQUE_KEY | The UNIQUE KEY constraint ensures that all values in a
specified column or set of columns are unique across the entire table. Unlike
the PRIMARY KEY constraint, a table can have multiple UNIQUE KEY constraints,
allowing for unique values in multiple columns or sets of columns.
| `UNIQUE_KEY` | `IndexType.UNIQUE_KEY` |
+| DATA_SKIPPING_VECTOR_SIMILARITY | ClickHouse HNSW index for approximate
nearest-neighbor search. See the [ClickHouse catalog
guide](./jdbc-clickhouse-catalog.md#table-indexes) for its properties and
server-version requirement. | `DATA_SKIPPING_VECTOR_SIMILARITY` |
`IndexType.DATA_SKIPPING_VECTOR_SIMILARITY` |
- Name. It defines the name of the index.
diff --git
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/ContainerSuite.java
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/ContainerSuite.java
index 9adff2730d..88e4cae939 100644
---
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/ContainerSuite.java
+++
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/ContainerSuite.java
@@ -29,12 +29,15 @@ import com.google.common.collect.ImmutableSet;
import java.io.Closeable;
import java.io.File;
import java.io.IOException;
+import java.io.InputStream;
import java.net.InetAddress;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileSystems;
import java.nio.file.Files;
+import java.nio.file.Path;
import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.PosixFilePermission;
import java.util.ArrayList;
import java.util.EnumMap;
@@ -66,6 +69,7 @@ public class ContainerSuite implements Closeable {
private static final String NETWORK_NAME = "gravitino-ci-network";
private static final int CLICKHOUSE_CLUSTER_SIZE = 3;
private static final String CLICKHOUSE_CLUSTER_HOST_PREFIX =
"gravitino-ci-clickhouse-cluster-";
+ private static final String CLICKHOUSE_TEST_IMAGE_ENV =
"GRAVITINO_CLICKHOUSE_TEST_IMAGE";
private static Network network = null;
private static volatile HiveContainer hiveContainer;
@@ -585,6 +589,8 @@ public class ContainerSuite implements Closeable {
// Start ClickHouse container
ClickHouseContainer.Builder clickHouseBuilder =
ClickHouseContainer.builder()
+ .withImage(resolveClickHouseImage())
+ .withFilesToMount(resolveClickHouseTestFiles())
.withHostName("gravitino-ci-clickhouse")
.withEnvVars(
ImmutableMap.<String, String>builder()
@@ -617,6 +623,8 @@ public class ContainerSuite implements Closeable {
for (int node = 1; node <= CLICKHOUSE_CLUSTER_SIZE; node++) {
ClickHouseContainer.Builder clickHouseBuilder =
ClickHouseContainer.builder()
+ .withImage(resolveClickHouseImage())
+ .withFilesToMount(resolveClickHouseTestFiles())
.withHostName(CLICKHOUSE_CLUSTER_HOST_PREFIX + node)
.withEnvVars(
ImmutableMap.<String, String>builder()
@@ -982,4 +990,31 @@ public class ContainerSuite implements Closeable {
LOG.error("Failed to close ContainerEnvironment", e);
}
}
+
+ private static String resolveClickHouseImage() {
+ String configuredImage = System.getenv(CLICKHOUSE_TEST_IMAGE_ENV);
+ return configuredImage == null || configuredImage.isBlank()
+ ? ClickHouseContainer.DEFAULT_IMAGE
+ : configuredImage.trim();
+ }
+
+ private static Map<String, String> resolveClickHouseTestFiles() {
+ // The feature-specific ClickHouse test image is 25.8+, where Time types
require this setting.
+ String configuredImage = System.getenv(CLICKHOUSE_TEST_IMAGE_ENV);
+ if (configuredImage == null || configuredImage.isBlank()) {
+ return ImmutableMap.of();
+ }
+
+ try (InputStream settings =
+ Objects.requireNonNull(
+
ContainerSuite.class.getResourceAsStream("/clickhouse-test-users.xml"))) {
+ Path settingsPath =
Files.createTempFile("gravitino-clickhouse-test-users-", ".xml");
+ Files.copy(settings, settingsPath, StandardCopyOption.REPLACE_EXISTING);
+ settingsPath.toFile().deleteOnExit();
+ return ImmutableMap.of(
+ "/etc/clickhouse-server/users.d/gravitino-test-settings.xml",
settingsPath.toString());
+ } catch (IOException e) {
+ throw new IllegalStateException("Failed to resolve ClickHouse test user
settings", e);
+ }
+ }
}
diff --git
a/integration-test-common/src/test/resources/clickhouse-test-users.xml
b/integration-test-common/src/test/resources/clickhouse-test-users.xml
new file mode 100644
index 0000000000..8cd8901c74
--- /dev/null
+++ b/integration-test-common/src/test/resources/clickhouse-test-users.xml
@@ -0,0 +1,25 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<clickhouse>
+ <profiles>
+ <default>
+ <enable_time_time64_type>1</enable_time_time64_type>
+ </default>
+ </profiles>
+</clickhouse>
