This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new f7f37af71e [#12916] fix(doris): fail closed when loading NGRAM_BF
indexes (#12919)
f7f37af71e is described below
commit f7f37af71eefeb2e646c5735c232b0b74b472af6
Author: StormSpirit <[email protected]>
AuthorDate: Sun Sep 20 21:00:03 2026 +0800
[#12916] fix(doris): fail closed when loading NGRAM_BF indexes (#12919)
### What changes were proposed in this pull request?
This pull request makes the Doris metadata read path fail closed for
native NGRAM_BF indexes instead of mapping them to unrelated existing
index types.
- Add an explicit NGRAM_BF branch in `mapDorisIndexType()` that throws
`UnsupportedOperationException` with the index name and normalized
native type.
- Read `Index_type` once in `getIndexes()` and give NGRAM_BF detection
precedence over the legacy `PRIMARY` name mapping, while preserving that
mapping for every other native type.
- Add focused unit coverage and real Doris 3.0.6.2 and 4.0.6 integration
coverage for fail-closed behavior, safe error context, unchanged native
index type and properties, and the existing INVERTED mapping.
- Document the temporary unsupported boundary.
### Why are the changes needed?
Doris exposes NGRAM_BF as a distinct native index type with `gram_size`
and `bf_size` properties. Gravitino currently has no matching index
representation, so an ordinary index name falls through to INVERTED
while the legacy `PRIMARY` name mapping returns PRIMARY_KEY. Both paths
drop the native semantics and properties. Rejecting the recognized but
unrepresentable metadata prevents callers from receiving a fabricated
index definition while keeping unrelated compatibility behavior
unchanged.
Fix: #12916
### Does this PR introduce _any_ user-facing change?
Yes. Loading a Doris table that contains a native NGRAM_BF index now
throws `UnsupportedOperationException` instead of returning an unrelated
existing index type. The error identifies the schema, table, index, and
native type without including native index properties.
There is no public API, enum, property-key, CREATE TABLE, or ALTER
behavior change. Existing mappings for BTREE, INVERTED, BITMAP,
BLOOMFILTER, and ANN remain unchanged, as does the generic fallback for
unrelated unknown Doris index types.
### How was this patch tested?
- `./gradlew :catalogs:catalog-jdbc-doris:spotlessCheck` — passed.
- `./gradlew rat` — passed.
- `./gradlew :catalogs:catalog-jdbc-doris:test -PskipITs` — 35 tests
passed, 0 skipped, 0 failures, 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:test --tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris3xIT.testNativeNgramBfIndexFailsClosed'
--tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris3xIT.testCreateTableWithInvertedIndex'
-PskipDockerTests=false -PdorisMultiVersionTest` — Doris 3.0.6.2: 2
tests passed, 0 skipped, 0 failures, 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:test --tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris4xIT.testNativeNgramBfIndexFailsClosed'
--tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris4xIT.testCreateTableWithInvertedIndex'
-PskipDockerTests=false -PdorisMultiVersionTest` — Doris 4.0.6: 2 tests
passed, 0 skipped, 0 failures, 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:build -x test` — passed.
Signed-off-by: jiangxt2 <[email protected]>
---
.../doris/operation/DorisTableOperations.java | 10 +++-
.../doris/integration/test/CatalogDoris3xIT.java | 70 ++++++++++++++++++++++
.../doris/integration/test/CatalogDoris4xIT.java | 70 ++++++++++++++++++++++
.../TestDorisTableOperationsSqlGeneration.java | 14 +++--
docs/jdbc-doris-catalog.md | 1 +
5 files changed, 158 insertions(+), 7 deletions(-)
diff --git
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
index d093c30613..8c25e45f56 100644
---
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
+++
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
@@ -575,14 +575,14 @@ public class DorisTableOperations extends
JdbcTableOperations {
String indexName = resultSet.getString("Key_name");
String columnName = resultSet.getString("Column_name");
String dorisIndexType = hasIndexType ?
resultSet.getString("Index_type") : null;
- // Doris always names the primary key index "PRIMARY"; detect it first.
+ // Preserve the legacy PRIMARY mapping unless authoritative metadata
identifies NGRAM_BF,
+ // which must fail closed.
Index.IndexType gravitinoIndexType;
- if ("PRIMARY".equals(indexName)) {
+ if ("PRIMARY".equals(indexName) &&
!"NGRAM_BF".equalsIgnoreCase(dorisIndexType)) {
gravitinoIndexType = Index.IndexType.PRIMARY_KEY;
} else if (hasIndexType) {
gravitinoIndexType = mapDorisIndexType(dorisIndexType, indexName);
} else {
- // Doris 1.2.x: no Index_type column, infer from index name
gravitinoIndexType = mapDorisIndexType(null, indexName);
}
Map<String, String> indexProperties = Collections.emptyMap();
@@ -633,6 +633,10 @@ public class DorisTableOperations extends
JdbcTableOperations {
return Index.IndexType.DATA_SKIPPING_BLOOM_FILTER;
case "ANN":
return Index.IndexType.VECTOR;
+ case "NGRAM_BF":
+ throw new UnsupportedOperationException(
+ String.format(
+ "Doris index '%s' uses unsupported native index type
'NGRAM_BF'", indexName));
default:
LOG.warn(
"Unknown Doris index type '{}' for index '{}', falling back to
INVERTED",
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
index 8c5a6ba0ba..fc5e250f90 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
@@ -33,11 +33,13 @@ import com.google.common.collect.Maps;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
+import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
@@ -517,6 +519,60 @@ public class CatalogDoris3xIT extends BaseIT {
assertEquals(colName2, t.index()[0].fieldNames()[0][0]);
}
+ @Test
+ void testNativeNgramBfIndexFailsClosed() throws Exception {
+ TableCatalog tc = catalog.asTableCatalog();
+ String tableName = GravitinoITUtils.genRandomName("t_ngram_bf");
+ NameIdentifier tid = NameIdentifier.of(schemaName, tableName);
+ String indexName = "PRIMARY";
+
+ tc.createTable(
+ tid,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ null);
+
+ DorisContainer dorisContainer =
containerSuite.getDorisContainer(DorisImageName.VERSION_3_0);
+ String jdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%d/%s",
+ dorisContainer.getContainerIpAddress(),
dorisContainer.getFeMysqlPort(), schemaName);
+ try (Connection connection =
+ DriverManager.getConnection(
+ jdbcUrl, DorisContainer.USER_NAME, DorisContainer.PASSWORD);
+ Statement statement = connection.createStatement()) {
+ statement.execute(
+ String.format(
+ "CREATE INDEX `%s` ON `%s` (`%s`) USING NGRAM_BF "
+ + "PROPERTIES(\"gram_size\"=\"3\", \"bf_size\"=\"256\")",
+ indexName, tableName, colName2));
+
+ Awaitility.await()
+ .atMost(MAX_WAIT_IN_SECONDS, TimeUnit.SECONDS)
+ .pollInterval(WAIT_INTERVAL_IN_SECONDS, TimeUnit.SECONDS)
+ .untilAsserted(
+ () ->
+ assertEquals(
+ "NGRAM_BF", getNativeIndexMetadata(statement, tableName,
indexName).get(0)));
+
+ List<String> metadataBefore = getNativeIndexMetadata(statement,
tableName, indexName);
+ UnsupportedOperationException exception =
+ assertThrows(UnsupportedOperationException.class, () ->
tc.loadTable(tid));
+
+ assertTrue(exception.getMessage().contains(schemaName));
+ assertTrue(exception.getMessage().contains(tableName));
+ assertTrue(exception.getMessage().contains(indexName));
+ assertTrue(exception.getMessage().contains("NGRAM_BF"));
+ assertFalse(exception.getMessage().contains("gram_size"));
+ assertFalse(exception.getMessage().contains("bf_size"));
+ assertEquals(metadataBefore, getNativeIndexMetadata(statement,
tableName, indexName));
+ }
+ }
+
@Test
void testExternalTypeRoundTrip() {
// Verify ExternalType columns survive the create → Doris 3.0 → load
round-trip.
@@ -773,6 +829,20 @@ public class CatalogDoris3xIT extends BaseIT {
"light_schema_change=true should appear after ALTER TABLE
SET"));
}
+ private List<String> getNativeIndexMetadata(
+ Statement statement, String tableName, String indexName) throws
SQLException {
+ try (ResultSet resultSet =
+ statement.executeQuery(String.format("SHOW INDEX FROM `%s`",
tableName))) {
+ while (resultSet.next()) {
+ if (indexName.equals(resultSet.getString("Key_name"))) {
+ return Arrays.asList(
+ resultSet.getString("Index_type"),
resultSet.getString("Properties"));
+ }
+ }
+ }
+ throw new AssertionError("Index not found: " + indexName);
+ }
+
private void executeSql(String sql) throws SQLException {
try (Connection connection =
DriverManager.getConnection(
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
index 46e80b6de5..c54154e511 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
@@ -33,11 +33,13 @@ import com.google.common.collect.Maps;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
+import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
@@ -552,6 +554,60 @@ public class CatalogDoris4xIT extends BaseIT {
assertEquals(colName2, t.index()[0].fieldNames()[0][0]);
}
+ @Test
+ void testNativeNgramBfIndexFailsClosed() throws Exception {
+ TableCatalog tc = catalog.asTableCatalog();
+ String tableName = GravitinoITUtils.genRandomName("t_ngram_bf");
+ NameIdentifier tid = NameIdentifier.of(schemaName, tableName);
+ String indexName = "PRIMARY";
+
+ tc.createTable(
+ tid,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ null);
+
+ DorisContainer dorisContainer =
containerSuite.getDorisContainer(DorisImageName.VERSION_4_0);
+ String jdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%d/%s",
+ dorisContainer.getContainerIpAddress(),
dorisContainer.getFeMysqlPort(), schemaName);
+ try (Connection connection =
+ DriverManager.getConnection(
+ jdbcUrl, DorisContainer.USER_NAME, DorisContainer.PASSWORD);
+ Statement statement = connection.createStatement()) {
+ statement.execute(
+ String.format(
+ "CREATE INDEX `%s` ON `%s` (`%s`) USING NGRAM_BF "
+ + "PROPERTIES(\"gram_size\"=\"3\", \"bf_size\"=\"256\")",
+ indexName, tableName, colName2));
+
+ Awaitility.await()
+ .atMost(MAX_WAIT_IN_SECONDS, TimeUnit.SECONDS)
+ .pollInterval(WAIT_INTERVAL_IN_SECONDS, TimeUnit.SECONDS)
+ .untilAsserted(
+ () ->
+ assertEquals(
+ "NGRAM_BF", getNativeIndexMetadata(statement, tableName,
indexName).get(0)));
+
+ List<String> metadataBefore = getNativeIndexMetadata(statement,
tableName, indexName);
+ UnsupportedOperationException exception =
+ assertThrows(UnsupportedOperationException.class, () ->
tc.loadTable(tid));
+
+ assertTrue(exception.getMessage().contains(schemaName));
+ assertTrue(exception.getMessage().contains(tableName));
+ assertTrue(exception.getMessage().contains(indexName));
+ assertTrue(exception.getMessage().contains("NGRAM_BF"));
+ assertFalse(exception.getMessage().contains("gram_size"));
+ assertFalse(exception.getMessage().contains("bf_size"));
+ assertEquals(metadataBefore, getNativeIndexMetadata(statement,
tableName, indexName));
+ }
+ }
+
@Test
void testExternalTypeRoundTrip() {
// Verify ExternalType columns survive the create → Doris 4.0 → load
round-trip.
@@ -809,6 +865,20 @@ public class CatalogDoris4xIT extends BaseIT {
"light_schema_change=true should appear after ALTER TABLE
SET"));
}
+ private List<String> getNativeIndexMetadata(
+ Statement statement, String tableName, String indexName) throws
SQLException {
+ try (ResultSet resultSet =
+ statement.executeQuery(String.format("SHOW INDEX FROM `%s`",
tableName))) {
+ while (resultSet.next()) {
+ if (indexName.equals(resultSet.getString("Key_name"))) {
+ return Arrays.asList(
+ resultSet.getString("Index_type"),
resultSet.getString("Properties"));
+ }
+ }
+ }
+ throw new AssertionError("Index not found: " + indexName);
+ }
+
private void executeSql(String sql) throws SQLException {
try (Connection connection =
DriverManager.getConnection(
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
index 9b9b7de262..164e9bb6b1 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
@@ -460,6 +460,13 @@ public class TestDorisTableOperationsSqlGeneration {
DorisTableOperations.mapDorisIndexType("BLOOMFILTER", "idx_name"));
Assertions.assertEquals(
Index.IndexType.VECTOR, DorisTableOperations.mapDorisIndexType("ANN",
"idx_name"));
+ UnsupportedOperationException exception =
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> DorisTableOperations.mapDorisIndexType("NGRAM_BF",
"idx_ngram"));
+ Assertions.assertEquals(
+ "Doris index 'idx_ngram' uses unsupported native index type
'NGRAM_BF'",
+ exception.getMessage());
// Unknown type should fall back to INVERTED
Assertions.assertEquals(
Index.IndexType.INVERTED,
@@ -529,10 +536,9 @@ public class TestDorisTableOperationsSqlGeneration {
Mockito.when(metaData.getColumnName(4)).thenReturn("Properties");
Mockito.when(metaData.getColumnName(5)).thenReturn("Comment");
Mockito.when(resultSet.next()).thenReturn(true, true, true, false);
- Mockito.when(resultSet.getString("Key_name"))
- .thenReturn("idx_first", "idx_ngram", "idx_second");
- Mockito.when(resultSet.getString("Column_name")).thenReturn("text_a",
"text_ngram", "text_b");
- Mockito.when(resultSet.getString("Index_type")).thenReturn("INVERTED",
"NGRAM_BF", "INVERTED");
+ Mockito.when(resultSet.getString("Key_name")).thenReturn("idx_first",
"idx_ann", "idx_second");
+ Mockito.when(resultSet.getString("Column_name")).thenReturn("text_a",
"text_ann", "text_b");
+ Mockito.when(resultSet.getString("Index_type")).thenReturn("INVERTED",
"ANN", "INVERTED");
Mockito.when(resultSet.getString("Properties"))
.thenReturn("(\"parser\" = \"english\")", "(\"support_phrase\" =
\"true\")");
diff --git a/docs/jdbc-doris-catalog.md b/docs/jdbc-doris-catalog.md
index 0a92de67fd..3a315ee69c 100644
--- a/docs/jdbc-doris-catalog.md
+++ b/docs/jdbc-doris-catalog.md
@@ -243,6 +243,7 @@ The Doris catalog supports the following index types. Each
index applies to a si
- `SHOW INDEX` does not escape embedded double quotes in property keys or
values. Gravitino therefore does not guarantee their round-trip and rejects
metadata that falls outside the supported flat quoted-pair format.
- Index comments are not currently represented by the Gravitino `Index` API
and are not preserved on round-trip.
- `BITMAP` is a write-only legacy type for backward compatibility with Doris
1.2.x. The write path generates a bare `INDEX` (no USING clause), but the read
path maps it back to `INVERTED` because Doris 4.0.6 removed BITMAP from the
grammar. Creating a BITMAP index and reading it back will show `INVERTED`.
+- Native Doris `NGRAM_BF` indexes are detected during table loading but are
not currently representable by a Gravitino index type. Loading a table that
contains one fails with `UnsupportedOperationException` instead of mapping it
to an unrelated Gravitino index type. Creating or altering NGRAM_BF indexes
through Gravitino and preserving their `gram_size` or `bf_size` properties are
not supported.
:::
**Primary Key example:**
