This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new d947f9823c [#11934] feat(clickhouse): read text index metadata (#13419)
d947f9823c is described below
commit d947f9823c2a46794c57278893eb61cc4cce2f85
Author: Thunderkeg <[email protected]>
AuthorDate: Wed Oct 28 08:36:54 2026 +0800
[#11934] feat(clickhouse): read text index metadata (#13419)
### What changes were proposed in this pull request?
This change adds read-only loading of native ClickHouse text
data-skipping index metadata. It recognizes the `text`, `gin`,
`full_text`, and `inverted` type names, restores supported `tokenizer`
and `ngram_size` properties from `type_full`, and preserves fallback to
legacy `type` metadata. For the `ngrams` tokenizer, it preserves sizes
from 1 through 8. The new `DATA_SKIPPING_TEXT` enum is covered by
JSON/OpenAPI updates and catalog documentation.
### Why are the changes needed?
Loading a ClickHouse table through Gravitino should preserve its native
text-index type and supported tokenizer metadata through the existing
table API.
Fix: #11934
### Does this PR introduce any user-facing change?
Yes. Loaded table metadata exposes the `DATA_SKIPPING_TEXT` type and
supported tokenizer values (`tokens` or `ngrams`); `ngram_size` values
from 1 through 8 are exposed for `ngrams`. Creating or altering text
indexes through Gravitino remains unsupported by this read-only change.
### How was this patch tested?
* `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
org.apache.gravitino.catalog.clickhouse.operations.TestClickHouseTableOperationsUnit
-PskipITs` — 67 tests passed on the final read-path commit, including
modern `ngrams(1)`, legacy `inverted(1)`, and rejection of sizes 0 and
9.
* `./gradlew :common:test --tests
org.apache.gravitino.json.TestJsonUtils` — 13 tests passed.
* `./gradlew :docs:build` — passed.
* `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
org.apache.gravitino.catalog.clickhouse.integration.test.CatalogClickHouseIT.testLoadNativeTextDataSkippingIndexes
-PskipDockerTests=false` — 1 test passed.
* `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
org.apache.gravitino.catalog.clickhouse.integration.test.CatalogClickHouseClusterIT
-PskipDockerTests=false` — 20 tests passed; the Docker integration
results were run before the final parser-boundary-only follow-up.
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.../org/apache/gravitino/rel/indexes/Index.java | 7 +-
.../catalog/clickhouse/ClickHouseConstants.java | 21 ++
.../operations/ClickHouseTableOperations.java | 319 ++++++++++++++++++++-
.../integration/test/CatalogClickHouseIT.java | 40 +++
.../TestClickHouseTableOperationsUnit.java | 169 +++++++++++
.../gravitino/api/rel/indexes/index.py | 8 +-
.../tests/unittests/dto/rel/test_index_serdes.py | 33 +++
.../tests/unittests/dto/rel/test_table_dto.py | 24 ++
.../org/apache/gravitino/json/TestJsonUtils.java | 11 +
docs/jdbc-clickhouse-catalog.md | 1 +
docs/open-api/indexes.yaml | 1 +
11 files changed, 623 insertions(+), 11 deletions(-)
diff --git a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
index e7cf8b429c..b8e3fcc543 100644
--- a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
+++ b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
@@ -54,8 +54,8 @@ public interface Index {
}
/**
- * The index type. Catalogs may support generic key types as well as
connector-specific index
- * types.
+ * Identifies an index type in the Gravitino table model. Catalogs may
support generic key types
+ * and connector-specific index types, and supported operations may vary by
catalog.
*/
enum IndexType {
/**
@@ -148,6 +148,9 @@ public interface Index {
*/
DATA_SKIPPING_TOKENBFV1,
+ /** Full-text data skipping index in ClickHouse. */
+ DATA_SKIPPING_TEXT,
+
/**
* Vector similarity data skipping index for approximate nearest-neighbor
search in ClickHouse.
*/
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
index 6263ac27c8..9e2b6b5753 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
@@ -84,6 +84,21 @@ public class ClickHouseConstants {
/** The name of the data skipping index type for tokenbf_v1 in ClickHouse.
*/
public static final String DATA_SKIPPING_TOKENBFV1 = "tokenbf_v1";
+ /** The current name of ClickHouse's full-text data skipping index. */
+ public static final String DATA_SKIPPING_TEXT = "text";
+
+ /** Historical name of ClickHouse's full-text data skipping index. */
+ public static final String DATA_SKIPPING_GIN = "gin";
+
+ /** Historical name of ClickHouse's full-text data skipping index. */
+ public static final String DATA_SKIPPING_FULL_TEXT = "full_text";
+
+ /** Historical name of ClickHouse's full-text data skipping index. */
+ public static final String DATA_SKIPPING_INVERTED = "inverted";
+
+ /** Property key for the ClickHouse full-text tokenizer mode. */
+ public static final String TOKENIZER = "tokenizer";
+
/** The name of the vector similarity data skipping index type in
ClickHouse. */
public static final String DATA_SKIPPING_VECTOR_SIMILARITY =
"vector_similarity";
@@ -138,5 +153,11 @@ public class ClickHouseConstants {
/** Property key for the n-gram size in ngrambf_v1 index properties. */
public static final String NGRAM_SIZE = "ngram_size";
+
+ /** Gravitino tokenizer property value for ClickHouse word-token mode. */
+ public static final String TEXT_INDEX_TOKENS = "tokens";
+
+ /** Gravitino tokenizer property value for ClickHouse n-gram mode. */
+ public static final String TEXT_INDEX_NGRAMS = "ngrams";
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index 8c8660f8fd..f8ca0ffc5b 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -24,9 +24,13 @@ import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexC
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.CLICKHOUSE_TYPE_FULL;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_ANNOY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_BLOOM_FILTER;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_FULL_TEXT;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_GIN;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_INVERTED;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_MINMAX_VALUE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_NGRAMBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_SET;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_TEXT;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_TOKENBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_USEARCH;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_VECTOR_SIMILARITY;
@@ -37,6 +41,9 @@ import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexC
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.NGRAM_SIZE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.RANDOM_SEED;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.SET_MAX_VALUES;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.TEXT_INDEX_NGRAMS;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.TEXT_INDEX_TOKENS;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.TOKENIZER;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_DISTANCE_FUNCTION;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.USEARCH_SCALAR_KIND;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.VECTOR_SIMILARITY_DIMENSIONS;
@@ -2115,6 +2122,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
if (indexType == Index.IndexType.DATA_SKIPPING_SET
|| isParameterizedBloomFilterIndex(indexType)
+ || indexType == Index.IndexType.DATA_SKIPPING_TEXT
|| isLegacyAnnUsearchIndex(indexType)
|| indexType == Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
{
try {
@@ -2177,6 +2185,28 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
databaseName,
tableName);
}
+ if (indexType == Index.IndexType.DATA_SKIPPING_TEXT) {
+ if (parameterProperties.isEmpty()) {
+ LOG.warn(
+ "ClickHouse text index '{}' on {}.{} has no supported
tokenizer metadata in "
+ + "{} '{}'; loaded Index.properties() is incomplete",
+ name,
+ databaseName,
+ tableName,
+ parameterSourceName,
+ parameterSource);
+ } else if
(TEXT_INDEX_NGRAMS.equals(parameterProperties.get(TOKENIZER))
+ && !parameterProperties.containsKey(NGRAM_SIZE)) {
+ LOG.warn(
+ "ClickHouse text index '{}' on {}.{} uses ngrams but {} '{}'
does not expose "
+ + "ngram_size; loaded Index.properties() is incomplete",
+ name,
+ databaseName,
+ tableName,
+ parameterSourceName,
+ parameterSource);
+ }
+ }
properties.putAll(parameterProperties);
secondaryIndexes.add(Indexes.of(indexType, name, fields,
properties));
}
@@ -2287,6 +2317,8 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_TOKENBFV1:
return parseBloomFilterPropertiesForQuery(
indexType, parameterSource, indexName, allowBareLegacyType);
+ case DATA_SKIPPING_TEXT:
+ return parseTextIndexPropertiesForQuery(parameterSource, indexName);
case DATA_SKIPPING_ANNOY:
case DATA_SKIPPING_USEARCH:
return parseLegacyAnnUsearchProperties(indexType, parameterSource,
indexName);
@@ -2577,6 +2609,271 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
|| indexType == Index.IndexType.DATA_SKIPPING_TOKENBFV1;
}
+ @VisibleForTesting
+ static Map<String, String> parseTextIndexPropertiesForQuery(String typeFull,
String indexName) {
+ String normalizedTypeFull = StringUtils.trimToEmpty(typeFull);
+ int paramsStart = normalizedTypeFull.indexOf('(');
+ if (paramsStart < 0) {
+ return Collections.emptyMap();
+ }
+ Preconditions.checkArgument(
+ normalizedTypeFull.endsWith(")"),
+ "Invalid text-index metadata '%s' for index '%s'",
+ typeFull,
+ indexName);
+
+ String clickHouseType = normalizedTypeFull.substring(0,
paramsStart).trim();
+ Preconditions.checkArgument(
+ isClickHouseTextIndexTypeName(clickHouseType),
+ "Text-index metadata '%s' has an unsupported type for index '%s'",
+ typeFull,
+ indexName);
+
+ String rawParameters =
+ normalizedTypeFull.substring(paramsStart + 1,
normalizedTypeFull.length() - 1);
+ if (isLegacyTextIndexTypeName(clickHouseType)) {
+ List<String> parameters = splitTextIndexParameters(rawParameters,
typeFull, indexName);
+ Preconditions.checkArgument(
+ parameters.size() == 1,
+ "Invalid %s metadata '%s' for index '%s': expected one tokenizer
parameter",
+ clickHouseType,
+ typeFull,
+ indexName);
+ return parseLegacyTextTokenizer(parameters.get(0), typeFull,
clickHouseType, indexName);
+ }
+
+ Map<String, String> namedParameters = new HashMap<>();
+ List<String> unsupportedParameters = new ArrayList<>();
+ for (String parameter : splitTextIndexParameters(rawParameters, typeFull,
indexName)) {
+ int assignmentIndex = parameter.indexOf('=');
+ if (assignmentIndex <= 0) {
+ unsupportedParameters.add(parameter.trim());
+ continue;
+ }
+ String key = StringUtils.lowerCase(parameter.substring(0,
assignmentIndex).trim());
+ String value = parameter.substring(assignmentIndex + 1).trim();
+ if (!TOKENIZER.equals(key) && !NGRAM_SIZE.equals(key)) {
+ unsupportedParameters.add(key);
+ continue;
+ }
+ Preconditions.checkArgument(
+ !namedParameters.containsKey(key),
+ "Duplicate text-index parameter '%s' in '%s' for index '%s'",
+ key,
+ typeFull,
+ indexName);
+ namedParameters.put(key, value);
+ }
+ if (!unsupportedParameters.isEmpty()) {
+ LOG.warn(
+ "ClickHouse text index '{}' contains unsupported metadata parameters
{}; "
+ + "loaded Index.properties() is incomplete",
+ indexName,
+ unsupportedParameters);
+ }
+
+ String tokenizer =
stripTextParameterQuotes(namedParameters.get(TOKENIZER));
+ if (StringUtils.isBlank(tokenizer)) {
+ return Collections.emptyMap();
+ }
+ if (StringUtils.equalsAnyIgnoreCase(tokenizer, "default", "tokens",
"splitByNonAlpha")) {
+ Preconditions.checkArgument(
+ !namedParameters.containsKey(NGRAM_SIZE),
+ "ngram_size is only valid for ngrams tokenizer in text-index
metadata '%s' for index '%s'",
+ typeFull,
+ indexName);
+ return Map.of(TOKENIZER, TEXT_INDEX_TOKENS);
+ }
+
+ String embeddedNgramSize = null;
+ String normalizedTokenizer = tokenizer;
+ int tokenizerParamsStart = tokenizer.indexOf('(');
+ if (tokenizerParamsStart >= 0 && tokenizer.endsWith(")")) {
+ normalizedTokenizer = tokenizer.substring(0,
tokenizerParamsStart).trim();
+ embeddedNgramSize =
+ tokenizer.substring(tokenizerParamsStart + 1, tokenizer.length() -
1).trim();
+ }
+ if (StringUtils.equalsAnyIgnoreCase(normalizedTokenizer, "ngram",
"ngrams")) {
+ String configuredNgramSize = namedParameters.get(NGRAM_SIZE);
+ Preconditions.checkArgument(
+ embeddedNgramSize == null || configuredNgramSize == null,
+ "ngram_size is specified twice in text-index metadata '%s' for index
'%s'",
+ typeFull,
+ indexName);
+ String ngramSize = embeddedNgramSize == null ? configuredNgramSize :
embeddedNgramSize;
+ Map<String, String> properties = new HashMap<>();
+ properties.put(TOKENIZER, TEXT_INDEX_NGRAMS);
+ if (StringUtils.isNotBlank(ngramSize)) {
+ properties.put(NGRAM_SIZE, requireTextNgramSize(ngramSize, typeFull,
indexName));
+ }
+ return Map.copyOf(properties);
+ }
+
+ return Collections.emptyMap();
+ }
+
+ private static Map<String, String> parseLegacyTextTokenizer(
+ String rawParameter, String typeFull, String clickHouseType, String
indexName) {
+ String parameter = stripTextParameterQuotes(rawParameter);
+ Preconditions.checkArgument(
+ StringUtils.isNumeric(parameter),
+ "Invalid %s metadata '%s' for index '%s': tokenizer parameter '%s'
must be an integer",
+ clickHouseType,
+ typeFull,
+ indexName,
+ parameter);
+ int value;
+ try {
+ value = Integer.parseInt(parameter);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(
+ "Invalid text-index metadata '%s' for index '%s': tokenizer
parameter '%s' is outside the supported integer range"
+ .formatted(typeFull, indexName, parameter),
+ e);
+ }
+
+ if (value == 0) {
+ return Map.of(TOKENIZER, TEXT_INDEX_TOKENS);
+ }
+ Preconditions.checkArgument(
+ value >= 1 && value <= 8,
+ "Invalid text-index metadata '%s' for index '%s': ngram_size %s is
outside the supported range [1, 8]",
+ typeFull,
+ indexName,
+ value);
+ return Map.of(TOKENIZER, TEXT_INDEX_NGRAMS, NGRAM_SIZE,
String.valueOf(value));
+ }
+
+ private static String requireTextNgramSize(String value, String typeFull,
String indexName) {
+ String normalizedValue = stripTextParameterQuotes(value);
+ Preconditions.checkArgument(
+ StringUtils.isNumeric(normalizedValue),
+ "Invalid text-index metadata '%s' for index '%s': ngram_size '%s' must
be an integer",
+ typeFull,
+ indexName,
+ normalizedValue);
+ int ngramSize;
+ try {
+ ngramSize = Integer.parseInt(normalizedValue);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(
+ "Invalid text-index metadata '%s' for index '%s': ngram_size '%s' is
outside the supported integer range"
+ .formatted(typeFull, indexName, normalizedValue),
+ e);
+ }
+ Preconditions.checkArgument(
+ ngramSize >= 1 && ngramSize <= 8,
+ "Invalid text-index metadata '%s' for index '%s': ngram_size %s is
outside the supported range [1, 8]",
+ typeFull,
+ indexName,
+ ngramSize);
+ return String.valueOf(ngramSize);
+ }
+
+ private static List<String> splitTextIndexParameters(
+ String rawParameters, String typeFull, String indexName) {
+ List<String> parameters = new ArrayList<>();
+ StringBuilder current = new StringBuilder();
+ int parenthesesDepth = 0;
+ int bracketsDepth = 0;
+ char quote = 0;
+ boolean escaped = false;
+ for (int i = 0; i < rawParameters.length(); i++) {
+ char character = rawParameters.charAt(i);
+ if (quote != 0) {
+ current.append(character);
+ if (escaped) {
+ escaped = false;
+ } else if (character == '\\') {
+ escaped = true;
+ } else if (character == quote) {
+ if (i + 1 < rawParameters.length() && rawParameters.charAt(i + 1) ==
quote) {
+ current.append(rawParameters.charAt(++i));
+ } else {
+ quote = 0;
+ }
+ }
+ continue;
+ }
+ if (character == '\'' || character == '"') {
+ quote = character;
+ current.append(character);
+ } else if (character == '(') {
+ parenthesesDepth++;
+ current.append(character);
+ } else if (character == ')') {
+ parenthesesDepth--;
+ Preconditions.checkArgument(
+ parenthesesDepth >= 0,
+ "Invalid text-index metadata '%s' for index '%s'",
+ typeFull,
+ indexName);
+ current.append(character);
+ } else if (character == '[') {
+ bracketsDepth++;
+ current.append(character);
+ } else if (character == ']') {
+ bracketsDepth--;
+ Preconditions.checkArgument(
+ bracketsDepth >= 0,
+ "Invalid text-index metadata '%s' for index '%s'",
+ typeFull,
+ indexName);
+ current.append(character);
+ } else if (character == ',' && parenthesesDepth == 0 && bracketsDepth ==
0) {
+ parameters.add(current.toString().trim());
+ current.setLength(0);
+ } else {
+ current.append(character);
+ }
+ }
+ Preconditions.checkArgument(
+ quote == 0 && parenthesesDepth == 0 && bracketsDepth == 0 && !escaped,
+ "Invalid text-index metadata '%s' for index '%s'",
+ typeFull,
+ indexName);
+ if (current.length() > 0 || !rawParameters.isEmpty()) {
+ parameters.add(current.toString().trim());
+ }
+ return parameters;
+ }
+
+ private static String stripTextParameterQuotes(String value) {
+ String normalizedValue = StringUtils.trimToEmpty(value);
+ if (normalizedValue.length() >= 2
+ && ((normalizedValue.startsWith("'") && normalizedValue.endsWith("'"))
+ || (normalizedValue.startsWith("\"") &&
normalizedValue.endsWith("\"")))) {
+ normalizedValue = normalizedValue.substring(1, normalizedValue.length()
- 1);
+ }
+ return normalizedValue.replace("\\'", "'").replace("''", "'");
+ }
+
+ private static boolean isClickHouseTextIndexTypeName(String typeName) {
+ return StringUtils.equalsAnyIgnoreCase(
+ StringUtils.trimToEmpty(typeName),
+ DATA_SKIPPING_TEXT,
+ DATA_SKIPPING_GIN,
+ DATA_SKIPPING_FULL_TEXT,
+ DATA_SKIPPING_INVERTED);
+ }
+
+ private static boolean isLegacyTextIndexTypeName(String typeName) {
+ return StringUtils.equalsAnyIgnoreCase(
+ StringUtils.trimToEmpty(typeName),
+ DATA_SKIPPING_GIN,
+ DATA_SKIPPING_FULL_TEXT,
+ DATA_SKIPPING_INVERTED);
+ }
+
+ private static boolean isClickHouseTextIndexType(String rawType) {
+ if (StringUtils.isBlank(rawType)) {
+ return false;
+ }
+ int paramsStart = rawType.indexOf('(');
+ String typeName = paramsStart < 0 ? rawType.trim() : rawType.substring(0,
paramsStart).trim();
+ return isClickHouseTextIndexTypeName(typeName);
+ }
+
private static Map<String, String> parseLegacyAnnUsearchProperties(
Index.IndexType indexType, String typeFull, String indexName) {
String expectedType = legacyAnnUsearchTypeName(indexType);
@@ -2725,12 +3022,12 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
/**
* Maps a ClickHouse data skipping index type string to the corresponding
Gravitino {@link
* Index.IndexType}. Returns {@code DATA_SKIPPING_MINMAX} for blank/null
input (ClickHouse
- * default). Also handles parameterized formats such as {@code set(N)},
{@code annoy(N)}, {@code
- * usearch(distanceFunction)}, and {@code vector_similarity(...)} that some
ClickHouse versions
- * may return from {@code system.data_skipping_indices}.
+ * default). Also handles parameterized formats returned from {@code
+ * system.data_skipping_indices}, including text, legacy Annoy/USearch, and
vector similarity
+ * indexes.
*
- * @param rawType the index type string from ClickHouse metadata (e.g.
"minmax", "bloom_filter",
- * "set", "set(0)", or "vector_similarity")
+ * @param rawType the index type string from ClickHouse metadata (e.g.
"minmax", "set(0)",
+ * "inverted(0)", "text(tokenizer = ngrams(3))", or "vector_similarity")
* @return the corresponding Gravitino IndexType
* @throws IllegalArgumentException if the type is not supported
*/
@@ -2751,6 +3048,11 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Index.IndexType.DATA_SKIPPING_NGRAMBFV1;
case DATA_SKIPPING_TOKENBFV1:
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
+ case DATA_SKIPPING_TEXT:
+ case DATA_SKIPPING_GIN:
+ case DATA_SKIPPING_FULL_TEXT:
+ case DATA_SKIPPING_INVERTED:
+ return Index.IndexType.DATA_SKIPPING_TEXT;
case DATA_SKIPPING_ANNOY:
return Index.IndexType.DATA_SKIPPING_ANNOY;
case DATA_SKIPPING_USEARCH:
@@ -2758,8 +3060,8 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
case DATA_SKIPPING_VECTOR_SIMILARITY:
return Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY;
default:
- // ClickHouse may return type with parameters in some versions. Match
on prefix to handle
- // both bare and parameterized formats.
+ // ClickHouse may return types with parameters in some versions. Match
the parameterized
+ // format for data skipping indexes whose raw type is not covered by
an exact case above.
if (rawType.startsWith(DATA_SKIPPING_SET + "(")) {
return Index.IndexType.DATA_SKIPPING_SET;
}
@@ -2769,6 +3071,9 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
if (rawType.startsWith(DATA_SKIPPING_TOKENBFV1 + "(")) {
return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
}
+ if (isClickHouseTextIndexType(rawType)) {
+ return Index.IndexType.DATA_SKIPPING_TEXT;
+ }
if (isParameterizedLegacyAnnUsearchType(rawType, DATA_SKIPPING_ANNOY))
{
return Index.IndexType.DATA_SKIPPING_ANNOY;
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index 8d7444acb8..fdcb6f1c44 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -3986,6 +3986,46 @@ public class CatalogClickHouseIT extends BaseIT {
"SHOW CREATE TABLE should contain Enum16 definition: " + createSql);
}
+ @Test
+ @Tag("gravitino-docker-test")
+ void testLoadNativeTextDataSkippingIndexes() {
+ String tableName = GravitinoITUtils.genRandomName("ch_text_idx_");
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName, tableName);
+
+ // ClickHouse 24.8 requires the experimental text-index setting for this
table creation.
+ clickhouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` (\n"
+ + " `id` UInt64,\n"
+ + " `token_text` String,\n"
+ + " `ngram_text` String,\n"
+ + " INDEX `idx_tokens` `token_text` TYPE text(tokenizer =
'default') GRANULARITY 1,\n"
+ + " INDEX `idx_ngrams` `ngram_text` TYPE text(tokenizer =
'ngram', ngram_size = 3) GRANULARITY 1\n"
+ + ") ENGINE = MergeTree ORDER BY `id` COMMENT 'text index IT' "
+ + "SETTINGS allow_experimental_full_text_index = 1",
+ schemaName, tableName));
+
+ Table loaded = catalog.asTableCatalog().loadTable(tableIdentifier);
+ Index loadedTokens =
+ Arrays.stream(loaded.index())
+ .filter(index -> Objects.equals(index.name(), "idx_tokens"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
loadedTokens.type());
+ Assertions.assertArrayEquals(new String[][] {{"token_text"}},
loadedTokens.fieldNames());
+ Assertions.assertEquals(Map.of("tokenizer", "tokens"),
loadedTokens.properties());
+
+ Index loadedNgrams =
+ Arrays.stream(loaded.index())
+ .filter(index -> Objects.equals(index.name(), "idx_ngrams"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
loadedNgrams.type());
+ Assertions.assertArrayEquals(new String[][] {{"ngram_text"}},
loadedNgrams.fieldNames());
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "3"),
loadedNgrams.properties());
+ }
+
@Test
@Tag("gravitino-docker-test")
void testCreateAndLoadTableWithNgrambfAndTokenbfIndexes() {
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
index f06dd53bd3..5eb19b1702 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
@@ -1851,6 +1851,175 @@ public class TestClickHouseTableOperationsUnit {
indexes.get(0).properties());
}
+ @Test
+ void testGetClickHouseTextIndexTypeAliases() {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ for (String type :
+ List.of(
+ "text",
+ "gin",
+ "full_text",
+ "inverted",
+ "inverted(0)",
+ "full_text(3)",
+ "text(tokenizer = ngrams(3))")) {
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
ops.getClickHouseIndexType(type));
+ }
+ }
+
+ @Test
+ void testParseTextIndexProperties() {
+ Assertions.assertEquals(
+ Map.of("tokenizer", "tokens"),
+
ClickHouseTableOperations.parseTextIndexPropertiesForQuery("inverted(0)",
"idx_tokens"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "tokens"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery("gin(0)",
"idx_tokens"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "tokens"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = 'default')", "idx_tokens"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "tokens"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = splitByNonAlpha)", "idx_tokens"));
+
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "2"),
+
ClickHouseTableOperations.parseTextIndexPropertiesForQuery("full_text(2)",
"idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "5"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery("gin(5)",
"idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "8"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = ngrams(8))", "idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "1"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = ngrams(1))", "idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "1"),
+
ClickHouseTableOperations.parseTextIndexPropertiesForQuery("inverted(1)",
"idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "4"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = 'ngram', ngram_size = 4, "
+ + "preprocessor = lower(concat(body, ', ')))",
+ "idx_ngrams"));
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams"),
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = ngrams)", "idx_ngrams_incomplete"));
+ Assertions.assertTrue(
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery(
+ "text(tokenizer = sparseGrams(3, 100))", "idx_unsupported")
+ .isEmpty());
+ Assertions.assertTrue(
+ ClickHouseTableOperations.parseTextIndexPropertiesForQuery("inverted",
"idx_legacy_bare")
+ .isEmpty());
+ }
+
+ @Test
+ void testParseTextIndexPropertiesRejectsMalformedNgramSize() {
+ for (String typeFull :
+ List.of(
+ "text(tokenizer = ngrams(0))",
+ "text(tokenizer = 'ngram', ngram_size = 9)",
+ "text(tokenizer = ngrams(abc))")) {
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+
ClickHouseTableOperations.parseTextIndexPropertiesForQuery(typeFull,
"idx_bad"));
+ Assertions.assertTrue(exception.getMessage().contains("idx_bad"));
+ }
+ }
+
+ @Test
+ void testGetIndexesReadsTextIndexPropertiesFromTypeFull() throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, true, true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name"))
+ .thenReturn(
+ "idx_inverted_tokens", "idx_full_text_ngrams", "idx_text_tokens",
"idx_text_ngrams");
+ Mockito.when(secondaryRs.getString("type")).thenReturn("inverted",
"full_text", "text", "text");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn(
+ "inverted(0)",
+ "full_text(3)",
+ "text(tokenizer = 'default')",
+ "text(tokenizer = 'ngram', ngram_size = 4, "
+ + "preprocessor = lower(concat(body, ', ')))");
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("body", "body",
"body", "body");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(1L, 1L, 1L,
1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(4, indexes.size());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
indexes.get(0).type());
+ Assertions.assertArrayEquals(new String[][] {{"body"}},
indexes.get(0).fieldNames());
+ Assertions.assertEquals(Map.of("tokenizer", "tokens"),
indexes.get(0).properties());
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "3"),
indexes.get(1).properties());
+ Assertions.assertEquals(Map.of("tokenizer", "tokens"),
indexes.get(2).properties());
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "4"),
indexes.get(3).properties());
+ }
+
+ @Test
+ void testGetIndexesFallsBackAndReadsLegacyTextMetadata() throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement modernSecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ PreparedStatement legacySecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ ResultSet legacySecondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(modernSecondaryStmt.executeQuery())
+ .thenThrow(new SQLException("Unknown identifier 'type_full'"));
+
Mockito.when(legacySecondaryStmt.executeQuery()).thenReturn(legacySecondaryRs);
+ Mockito.when(legacySecondaryRs.next()).thenReturn(true, true, false);
+ Mockito.when(legacySecondaryRs.getString("name"))
+ .thenReturn("idx_legacy_ngrams", "idx_legacy_bare");
+
Mockito.when(legacySecondaryRs.getString("type")).thenReturn("inverted(3)",
"inverted");
+ Mockito.when(legacySecondaryRs.getString("expr")).thenReturn("body",
"body");
+ Mockito.when(legacySecondaryRs.getLong("granularity")).thenReturn(1L, 1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(modernSecondaryStmt)
+ .thenReturn(legacySecondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(2, indexes.size());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
indexes.get(0).type());
+ Assertions.assertEquals(
+ Map.of("tokenizer", "ngrams", "ngram_size", "3"),
indexes.get(0).properties());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TEXT,
indexes.get(1).type());
+ Assertions.assertTrue(indexes.get(1).properties().isEmpty());
+ }
+
@Test
void testGetIndexesPreservesLegacyAnnUsearchMetadata() throws Exception {
ExposedClickHouseTableOperations ops = newOps();
diff --git a/clients/client-python/gravitino/api/rel/indexes/index.py
b/clients/client-python/gravitino/api/rel/indexes/index.py
index 2803423490..995c72be4e 100644
--- a/clients/client-python/gravitino/api/rel/indexes/index.py
+++ b/clients/client-python/gravitino/api/rel/indexes/index.py
@@ -24,7 +24,8 @@ from typing import List, Optional
class Index(ABC):
"""The Index interface defines methods for implementing table index
columns.
- Catalogs may provide key and connector-specific index types.
+ Catalogs may provide generic key and connector-specific index types;
supported operations
+ depend on the catalog.
"""
@unique
@@ -67,6 +68,9 @@ class Index(ABC):
DATA_SKIPPING_USEARCH = "DATA_SKIPPING_USEARCH"
"""Legacy ClickHouse USearch index metadata. This type is
metadata-only."""
+ DATA_SKIPPING_TEXT = "DATA_SKIPPING_TEXT"
+ """ClickHouse full-text data-skipping index loaded from native table
metadata."""
+
DATA_SKIPPING_VECTOR_SIMILARITY = "DATA_SKIPPING_VECTOR_SIMILARITY"
"""ClickHouse vector similarity data skipping index."""
@@ -101,5 +105,5 @@ class Index(ABC):
pass # pragma: no cover
def properties(self) -> dict[str, str]:
- """Returns the index properties, or an empty map when none are
defined."""
+ """Returns optional index properties, empty when none are defined."""
return {}
diff --git a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
index 263270ecd4..b78c11eecb 100644
--- a/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
+++ b/clients/client-python/tests/unittests/dto/rel/test_index_serdes.py
@@ -132,6 +132,39 @@ class TestIndexSerdes(unittest.TestCase):
serialized_dict = json.loads(mock_data_class.to_json())
self.assertDictEqual(json_dict, serialized_dict)
+ def test_text_index_properties_round_trip(self):
+ index_json = {
+ "indexType": "DATA_SKIPPING_TEXT",
+ "name": "idx_text",
+ "fieldNames": [["body"]],
+ "properties": {"tokenizer": "ngrams", "ngram_size": "3"},
+ }
+
+ index = IndexSerdes.deserialize(index_json)
+ self.assertIs(index.type(), Index.IndexType.DATA_SKIPPING_TEXT)
+ self.assertEqual(index.properties(), index_json["properties"])
+ self.assertEqual(IndexSerdes.serialize(index), index_json)
+
+ def test_legacy_index_without_properties(self):
+ class LegacyIndex(Index):
+ def type(self):
+ return Index.IndexType.PRIMARY_KEY
+
+ def name(self):
+ return "PRIMARY"
+
+ def field_names(self):
+ return [["id"]]
+
+ self.assertEqual(
+ IndexSerdes.serialize(LegacyIndex()),
+ {
+ "indexType": "PRIMARY_KEY",
+ "name": "PRIMARY",
+ "fieldNames": [["id"]],
+ },
+ )
+
def test_index_serdes_legacy_clickhouse_metadata(self):
annoy_properties = {
"annoy_trees": "100",
diff --git a/clients/client-python/tests/unittests/dto/rel/test_table_dto.py
b/clients/client-python/tests/unittests/dto/rel/test_table_dto.py
index bd78ee1cf5..7cd6da2b1c 100644
--- a/clients/client-python/tests/unittests/dto/rel/test_table_dto.py
+++ b/clients/client-python/tests/unittests/dto/rel/test_table_dto.py
@@ -35,6 +35,7 @@ from gravitino.dto.rel.partitioning.identity_partitioning_dto
import (
from gravitino.dto.rel.partitioning.partitioning import SingleFieldPartitioning
from gravitino.dto.rel.sort_order_dto import SortOrderDTO
from gravitino.dto.rel.table_dto import TableDTO
+from gravitino.dto.responses.table_response import TableResponse
from gravitino.exceptions.base import IllegalArgumentException
from tests.unittests.fixtures.table_fixtures import TABLE_DTO_JSON_STRING
@@ -66,6 +67,29 @@ class TestTableDTO(unittest.TestCase):
self.assertDictEqual(dto.properties(), {"format": "ORC"})
+ def test_table_response_with_text_index(self):
+ table_json = json.loads(self.complete_json)
+ table_json["indexes"] = [
+ {
+ "indexType": "DATA_SKIPPING_TEXT",
+ "name": "idx_text",
+ "fieldNames": [["name"]],
+ "properties": {"tokenizer": "ngrams", "ngram_size": "3"},
+ }
+ ]
+
+ response = TableResponse.from_json(
+ json.dumps({"code": 0, "table": table_json}), infer_missing=True
+ )
+ response.validate()
+ index = response.table().index()[0]
+ self.assertIs(index.type(), Index.IndexType.DATA_SKIPPING_TEXT)
+ self.assertEqual(index.properties(), {"tokenizer": "ngrams",
"ngram_size": "3"})
+ self.assertEqual(
+ json.loads(response.table().to_json())["indexes"][0]["properties"],
+ index.properties(),
+ )
+
def test_table_dto_deserialize_required_fields_success(self):
json_string = """
{
diff --git a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
index dd9dca0026..b653e67a4f 100644
--- a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
+++ b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
@@ -526,6 +526,17 @@ public class TestJsonUtils {
objectMapper.readValue(expected, IndexDTO.class),
objectMapper.readValue(jsonValue, IndexDTO.class));
+ Index textIndex =
+ IndexDTO.builder()
+ .withIndexType(Index.IndexType.DATA_SKIPPING_TEXT)
+ .withName("idx_text")
+ .withFieldNames(new String[][] {{"body"}})
+ .withProperties(Map.of("tokenizer", "ngrams", "ngram_size", "3"))
+ .build();
+ String textJson = JsonUtils.objectMapper().writeValueAsString(textIndex);
+
Assertions.assertTrue(textJson.contains("\"indexType\":\"DATA_SKIPPING_TEXT\""));
+ Assertions.assertEquals(textIndex, objectMapper.readValue(textJson,
IndexDTO.class));
+
Index vectorIndex =
IndexDTO.builder()
.withIndexType(Index.IndexType.DATA_SKIPPING_VECTOR_SIMILARITY)
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index 4b6e64db22..9806b08b5e 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -297,6 +297,7 @@ The `engine_parameters` property applies to
`ReplacingMergeTree`, `SummingMergeT
- `DATA_SKIPPING_SET` (default `GRANULARITY 1`, plus configurable `set(N)`
max values)
- `DATA_SKIPPING_NGRAMBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `ngram_size`, `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
- `DATA_SKIPPING_TOKENBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
+ - `DATA_SKIPPING_TEXT` (native ClickHouse text indexes are recognized when
loading a table; `tokenizer` is exposed as `tokens` or `ngrams`, with
`ngram_size` from 1 to 8 for `ngrams`; creating or altering text indexes
through Gravitino is not supported yet)
- `DATA_SKIPPING_VECTOR_SIMILARITY` (requires ClickHouse 25.8+; default
`GRANULARITY 100000000`; requires `type=hnsw`, `distance_function`, and
`dimensions` in `Index.properties()`)
`DATA_SKIPPING_VECTOR_SIMILARITY` supports `distance_function` values
`L2Distance` and `cosineDistance` on ClickHouse 25.8+. Optional properties are
`quantization` (`f64`, `f32`, `f16`, `bf16`, `i8`, or `b1`; default `bf16`),
`hnsw_max_connections_per_layer` (default 32), and
`hnsw_candidate_list_size_for_construction` (default 128). A value of 0 for
either HNSW numeric property requests the ClickHouse default. Custom
`granularity` must be a positive integer. See the [ClickHouse vector [...]
diff --git a/docs/open-api/indexes.yaml b/docs/open-api/indexes.yaml
index 894761b776..ba594994df 100644
--- a/docs/open-api/indexes.yaml
+++ b/docs/open-api/indexes.yaml
@@ -42,6 +42,7 @@ components:
- "data_skipping_tokenbfv1"
- "data_skipping_annoy"
- "data_skipping_usearch"
+ - "data_skipping_text"
- "data_skipping_vector_similarity"
name:
type: string
