This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 0d40e22f96 [#12914] fix(doris): preserve inverted index properties
(#12920)
0d40e22f96 is described below
commit 0d40e22f96eeef4671cd0f2153f976019273d4c4
Author: StormSpirit <[email protected]>
AuthorDate: Tue Sep 15 20:03:16 2026 +0800
[#12914] fix(doris): preserve inverted index properties (#12920)
### What changes were proposed in this pull request?
This pull request preserves Apache Doris INVERTED index properties
across CREATE TABLE, ALTER TABLE ADD INDEX, native metadata adoption,
and load/recreate.
The write path validates property keys and values, sorts keys for
deterministic SQL, and reuses the existing Doris `PROPERTIES` serializer
for both CREATE and ALTER. The read path parses the flat `SHOW
INDEX.Properties` representation and attaches the result only when the
raw Doris index type is `INVERTED`; missing or empty metadata keeps the
existing empty-map behavior.
The parser preserves commas and equals signs inside quoted values and
preserves literal backslashes while rejecting malformed or duplicate
entries. Properties are attached only to raw Doris INVERTED rows; raw
ANN, NGRAM_BF, BITMAP, and unknown rows retain empty property maps. The
Doris catalog documentation now describes the property contract,
server-added effective defaults, unsupported index comments, and the
embedded-double-quote round-trip boundary.
### Why are the changes needed?
The public Gravitino index and table-change APIs already carry property
maps, but the JDBC Doris catalog currently discards them in both DDL
generation and metadata loading. Property-bearing INVERTED indexes can
therefore lose parser, phrase-search, and related full-text behavior
during create, load, or recreate without raising an error.
Fix: #12914
### Does this PR introduce _any_ user-facing change?
Yes. Doris INVERTED properties supplied during CREATE TABLE or ALTER
TABLE ADD INDEX are now emitted in DDL, and effective properties
reported by Doris are available through `Index.properties()` when the
table is loaded.
Property availability, values, and server-added defaults remain
controlled by the Doris version. Arbitrary property values containing
embedded double quotes are not guaranteed to round-trip because Doris
does not escape them in `SHOW INDEX`; Gravitino rejects metadata outside
the supported flat quoted-pair format. No public API or property key is
added or removed.
Index comments, ANN/VECTOR properties, NGRAM_BF representation, and
index build lifecycle remain unsupported by this change.
### How was this patch tested?
- `./gradlew :catalogs:catalog-jdbc-doris:spotlessCheck` — passed.
- `./gradlew rat` — passed.
- `./gradlew :catalogs:catalog-jdbc-doris:test -PskipITs` — passed with
43 tests, 0 skipped, 0 failures, and 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:test --tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris3xIT.testInvertedIndexPropertiesRoundTrip'
-PskipDockerTests=false -PdorisMultiVersionTest` — passed against Doris
3.0.6.2 with 1 test, 0 skipped, 0 failures, and 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:test --tests
'org.apache.gravitino.catalog.doris.integration.test.CatalogDoris4xIT.testInvertedIndexPropertiesRoundTrip'
-PskipDockerTests=false -PdorisMultiVersionTest` — passed against Doris
4.0.6 with 1 test, 0 skipped, 0 failures, and 0 errors.
- `./gradlew :catalogs:catalog-jdbc-doris:build -x test` — passed.
- `git diff --check` — passed.
Signed-off-by: jiangxt2 <[email protected]>
---
.../doris/operation/DorisTableOperations.java | 108 ++++++++++-
.../doris/integration/test/CatalogDoris3xIT.java | 108 ++++++++++-
.../doris/integration/test/CatalogDoris4xIT.java | 108 ++++++++++-
.../TestDorisTableOperationsSqlGeneration.java | 215 +++++++++++++++++++++
docs/jdbc-doris-catalog.md | 17 +-
5 files changed, 542 insertions(+), 14 deletions(-)
diff --git
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
index 8d0bf8464a..af8e19749e 100644
---
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
+++
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
@@ -44,9 +44,11 @@ import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.TreeMap;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
+import javax.annotation.Nullable;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.BooleanUtils;
@@ -77,6 +79,8 @@ public class DorisTableOperations extends JdbcTableOperations
{
private static final String BACK_QUOTE = "`";
private static final String DORIS_AUTO_INCREMENT = "AUTO_INCREMENT";
private static final String NEW_LINE = "\n";
+ private static final Pattern INDEX_PROPERTY_PATTERN =
+ Pattern.compile("\\s*\"([^\"]*)\"\\s*=\\s*\"([^\"]*)\"\\s*(,)?\\s*");
private static final Pattern DORIS_VERSION_PATTERN =
Pattern.compile("(\\d+\\.\\d+\\.\\d+\\.?\\d*)");
@@ -352,11 +356,14 @@ public class DorisTableOperations extends
JdbcTableOperations {
String fieldName =
requireSingleTopLevelIndexField(index.name(),
index.fieldNames());
String usingClause = mapIndexTypeToUsingClause(index.type());
+ String propertiesSql =
+ generateIndexPropertiesSql(index.type(),
index.properties());
if (usingClause.isEmpty()) {
- return String.format("INDEX `%s` (`%s`)", index.name(),
fieldName);
+ return String.format("INDEX `%s` (`%s`)", index.name(),
fieldName)
+ + propertiesSql;
}
- return String.format(
- "INDEX `%s` (`%s`) %s", index.name(), fieldName,
usingClause);
+ return String.format("INDEX `%s` (`%s`) %s", index.name(),
fieldName, usingClause)
+ + propertiesSql;
})
.collect(Collectors.joining(",\n"));
@@ -550,13 +557,15 @@ public class DorisTableOperations extends
JdbcTableOperations {
try (PreparedStatement preparedStatement =
connection.prepareStatement(sql);
ResultSet resultSet = preparedStatement.executeQuery()) {
- // Check if Index_type column exists (available in Doris 2.0+).
+ // Check which optional columns are available on this Doris version.
boolean hasIndexType = false;
+ boolean hasProperties = false;
ResultSetMetaData metaData = resultSet.getMetaData();
for (int i = 1; i <= metaData.getColumnCount(); i++) {
if ("Index_type".equals(metaData.getColumnName(i))) {
hasIndexType = true;
- break;
+ } else if ("Properties".equals(metaData.getColumnName(i))) {
+ hasProperties = true;
}
}
@@ -564,17 +573,24 @@ public class DorisTableOperations extends
JdbcTableOperations {
while (resultSet.next()) {
String indexName = resultSet.getString("Key_name");
String columnName = resultSet.getString("Column_name");
+ String dorisIndexType = hasIndexType ?
resultSet.getString("Index_type") : null;
// Doris always names the primary key index "PRIMARY"; detect it first.
Index.IndexType gravitinoIndexType;
if ("PRIMARY".equals(indexName)) {
gravitinoIndexType = Index.IndexType.PRIMARY_KEY;
} else if (hasIndexType) {
- gravitinoIndexType =
mapDorisIndexType(resultSet.getString("Index_type"), indexName);
+ gravitinoIndexType = mapDorisIndexType(dorisIndexType, indexName);
} else {
// Doris 1.2.x: no Index_type column, infer from index name
gravitinoIndexType = mapDorisIndexType(null, indexName);
}
- indexes.add(Indexes.of(gravitinoIndexType, indexName, new String[][]
{{columnName}}));
+ Map<String, String> indexProperties = Collections.emptyMap();
+ if (hasProperties && "INVERTED".equalsIgnoreCase(dorisIndexType)) {
+ indexProperties =
parseIndexProperties(resultSet.getString("Properties"), indexName);
+ }
+ indexes.add(
+ Indexes.of(
+ gravitinoIndexType, indexName, new String[][] {{columnName}},
indexProperties));
}
return indexes;
} catch (SQLException e) {
@@ -1009,10 +1025,12 @@ public class DorisTableOperations extends
JdbcTableOperations {
String fieldName =
requireSingleTopLevelIndexField(addIndex.getName(),
addIndex.getFieldNames());
String usingClause = mapIndexTypeToUsingClause(addIndex.getType());
+ String propertiesSql = generateIndexPropertiesSql(addIndex.getType(),
addIndex.getProperties());
if (usingClause.isEmpty()) {
- return String.format("ADD INDEX `%s` (`%s`)", addIndex.getName(),
fieldName);
+ return String.format("ADD INDEX `%s` (`%s`)", addIndex.getName(),
fieldName) + propertiesSql;
}
- return String.format("ADD INDEX `%s` (`%s`) %s", addIndex.getName(),
fieldName, usingClause);
+ return String.format("ADD INDEX `%s` (`%s`) %s", addIndex.getName(),
fieldName, usingClause)
+ + propertiesSql;
}
static String deleteIndexDefinition(
@@ -1076,6 +1094,78 @@ public class DorisTableOperations extends
JdbcTableOperations {
return null;
}
+ @VisibleForTesting
+ static Map<String, String> parseIndexProperties(
+ @Nullable String propertiesText, String indexName) {
+ if (StringUtils.isBlank(propertiesText)) {
+ return Collections.emptyMap();
+ }
+
+ String trimmed = propertiesText.trim();
+ Preconditions.checkArgument(
+ trimmed.length() >= 2 && trimmed.startsWith("(") &&
trimmed.endsWith(")"),
+ "Malformed Properties metadata for Doris index '%s'",
+ indexName);
+
+ String entries = trimmed.substring(1, trimmed.length() - 1);
+ if (StringUtils.isBlank(entries)) {
+ return Collections.emptyMap();
+ }
+
+ Map<String, String> properties = new HashMap<>();
+ Matcher matcher = INDEX_PROPERTY_PATTERN.matcher(entries);
+ int position = 0;
+ while (position < entries.length()) {
+ matcher.region(position, entries.length());
+ Preconditions.checkArgument(
+ matcher.lookingAt(), "Malformed Properties metadata for Doris index
'%s'", indexName);
+
+ String key = matcher.group(1);
+ String value = matcher.group(2);
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(key),
+ "Malformed Properties metadata for Doris index '%s': property key
must not be blank",
+ indexName);
+ Preconditions.checkArgument(
+ !properties.containsKey(key),
+ "Malformed Properties metadata for Doris index '%s': duplicate
property key '%s'",
+ indexName,
+ key);
+ properties.put(key, value);
+
+ position = matcher.end();
+ boolean hasSeparator = matcher.group(3) != null;
+ Preconditions.checkArgument(
+ hasSeparator == (position < entries.length()),
+ "Malformed Properties metadata for Doris index '%s'",
+ indexName);
+ }
+ return Collections.unmodifiableMap(properties);
+ }
+
+ private static String generateIndexPropertiesSql(
+ Index.IndexType indexType, @Nullable Map<String, String> properties) {
+ if (indexType != Index.IndexType.INVERTED || properties == null ||
properties.isEmpty()) {
+ return "";
+ }
+
+ properties.forEach(
+ (key, value) -> {
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(key), "Doris index property key must not
be blank");
+ Preconditions.checkArgument(
+ value != null, "Doris index property '%s' must not have a null
value", key);
+ Preconditions.checkArgument(
+ key.chars().noneMatch(Character::isISOControl),
+ "Doris index property key must not contain control characters");
+ Preconditions.checkArgument(
+ value.chars().noneMatch(Character::isISOControl),
+ "Doris index property '%s' must not contain control characters",
+ key);
+ });
+ return DorisUtils.generatePropertiesSql(new TreeMap<>(properties));
+ }
+
private static String requireSingleTopLevelIndexField(String indexName,
String[][] fieldNames) {
Preconditions.checkArgument(
fieldNames != null && fieldNames.length == 1,
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
index ce51c4005b..97da5a1c03 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris3xIT.java
@@ -30,6 +30,10 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
import com.google.common.collect.Maps;
import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.sql.Statement;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
@@ -95,6 +99,7 @@ public class CatalogDoris3xIT extends BaseIT {
private GravitinoMetalake metalake;
private Catalog catalog;
+ private String jdbcUrl;
@BeforeAll
public void startup() throws IOException {
@@ -125,7 +130,7 @@ public class CatalogDoris3xIT extends BaseIT {
private void createCatalog() {
DorisContainer dorisContainer =
containerSuite.getDorisContainer(DorisImageName.VERSION_3_0);
- String jdbcUrl =
+ jdbcUrl =
String.format(
"jdbc:mysql://%s:%d/",
dorisContainer.getContainerIpAddress(),
dorisContainer.getFeMysqlPort());
@@ -181,6 +186,86 @@ public class CatalogDoris3xIT extends BaseIT {
assertEquals("idx_data", t.index()[0].name());
}
+ @Test
+ void testInvertedIndexPropertiesRoundTrip() throws SQLException {
+ TableCatalog tc = catalog.asTableCatalog();
+ Map<String, String> requestedProperties = Map.of("parser", "english",
"support_phrase", "true");
+
+ NameIdentifier createId = NameIdentifier.of(schemaName,
"t_inverted_properties_create");
+ Index[] createIndexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.INVERTED,
+ "idx_create",
+ new String[][] {{colName2}},
+ requestedProperties)
+ };
+ tc.createTable(
+ createId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ createIndexes);
+
+ Table created = tc.loadTable(createId);
+ Index createdIndex = findIndex(created, "idx_create");
+ assertContainsProperties(createdIndex, requestedProperties);
+
+ NameIdentifier recreateId = NameIdentifier.of(schemaName,
"t_inverted_properties_recreate");
+ tc.createTable(
+ recreateId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ created.index());
+ assertEquals(
+ createdIndex.properties(), findIndex(tc.loadTable(recreateId),
"idx_create").properties());
+
+ NameIdentifier alterId = NameIdentifier.of(schemaName,
"t_inverted_properties_alter");
+ tc.createTable(
+ alterId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ Indexes.EMPTY_INDEXES);
+ tc.alterTable(
+ alterId,
+ TableChange.addIndex(
+ Index.IndexType.INVERTED,
+ "idx_alter",
+ new String[][] {{colName2}},
+ requestedProperties));
+ Awaitility.await()
+ .atMost(MAX_WAIT_IN_SECONDS, TimeUnit.SECONDS)
+ .pollInterval(WAIT_INTERVAL_IN_SECONDS, TimeUnit.SECONDS)
+ .untilAsserted(
+ () ->
+ assertContainsProperties(
+ findIndex(tc.loadTable(alterId), "idx_alter"),
requestedProperties));
+
+ NameIdentifier nativeId = NameIdentifier.of(schemaName,
"t_inverted_properties_native");
+ executeSql(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + "`%s` BIGINT NOT NULL, "
+ + "`%s` VARCHAR(100), "
+ + "INDEX `idx_native` (`%s`) USING INVERTED "
+ + "PROPERTIES(\"parser\"=\"english\",
\"support_phrase\"=\"true\")"
+ + ") DISTRIBUTED BY HASH(`%s`) BUCKETS 1 "
+ + "PROPERTIES(\"replication_num\"=\"1\")",
+ schemaName, nativeId.name(), colName1, colName2, colName2,
colName1));
+ assertContainsProperties(findIndex(tc.loadTable(nativeId), "idx_native"),
requestedProperties);
+ }
+
@Test
void testAddAndDropInvertedIndex() {
TableCatalog tc = catalog.asTableCatalog();
@@ -555,6 +640,27 @@ public class CatalogDoris3xIT extends BaseIT {
"light_schema_change=true should appear after ALTER TABLE
SET"));
}
+ private void executeSql(String sql) throws SQLException {
+ try (Connection connection =
+ DriverManager.getConnection(
+ jdbcUrl, DorisContainer.USER_NAME, DorisContainer.PASSWORD);
+ Statement statement = connection.createStatement()) {
+ statement.execute(sql);
+ }
+ }
+
+ private Index findIndex(Table table, String indexName) {
+ return Arrays.stream(table.index())
+ .filter(index -> index.name().equals(indexName))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("Index not found: " +
indexName));
+ }
+
+ private void assertContainsProperties(Index index, Map<String, String>
expectedProperties) {
+ expectedProperties.forEach(
+ (key, value) -> assertEquals(value, index.properties().get(key),
"Property: " + key));
+ }
+
private Column findColumn(Table table, String columnName) {
return Arrays.stream(table.columns())
.filter(c -> c.name().equals(columnName))
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
index d438a0e8d8..6c5863dc98 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris4xIT.java
@@ -31,6 +31,10 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
import com.google.common.collect.Maps;
import java.io.IOException;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.sql.Statement;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
@@ -97,6 +101,7 @@ public class CatalogDoris4xIT extends BaseIT {
private GravitinoMetalake metalake;
private Catalog catalog;
+ private String jdbcUrl;
@BeforeAll
public void startup() throws IOException {
@@ -127,7 +132,7 @@ public class CatalogDoris4xIT extends BaseIT {
private void createCatalog() {
DorisContainer dorisContainer =
containerSuite.getDorisContainer(DorisImageName.VERSION_4_0);
- String jdbcUrl =
+ jdbcUrl =
String.format(
"jdbc:mysql://%s:%d/",
dorisContainer.getContainerIpAddress(),
dorisContainer.getFeMysqlPort());
@@ -183,6 +188,86 @@ public class CatalogDoris4xIT extends BaseIT {
assertEquals("idx_data", t.index()[0].name());
}
+ @Test
+ void testInvertedIndexPropertiesRoundTrip() throws SQLException {
+ TableCatalog tc = catalog.asTableCatalog();
+ Map<String, String> requestedProperties = Map.of("parser", "english",
"support_phrase", "true");
+
+ NameIdentifier createId = NameIdentifier.of(schemaName,
"t_inverted_properties_create");
+ Index[] createIndexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.INVERTED,
+ "idx_create",
+ new String[][] {{colName2}},
+ requestedProperties)
+ };
+ tc.createTable(
+ createId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ createIndexes);
+
+ Table created = tc.loadTable(createId);
+ Index createdIndex = findIndex(created, "idx_create");
+ assertContainsProperties(createdIndex, requestedProperties);
+
+ NameIdentifier recreateId = NameIdentifier.of(schemaName,
"t_inverted_properties_recreate");
+ tc.createTable(
+ recreateId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ created.index());
+ assertEquals(
+ createdIndex.properties(), findIndex(tc.loadTable(recreateId),
"idx_create").properties());
+
+ NameIdentifier alterId = NameIdentifier.of(schemaName,
"t_inverted_properties_alter");
+ tc.createTable(
+ alterId,
+ basicColumns(),
+ tableComment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ hashDist(),
+ null,
+ Indexes.EMPTY_INDEXES);
+ tc.alterTable(
+ alterId,
+ TableChange.addIndex(
+ Index.IndexType.INVERTED,
+ "idx_alter",
+ new String[][] {{colName2}},
+ requestedProperties));
+ Awaitility.await()
+ .atMost(MAX_WAIT_IN_SECONDS, TimeUnit.SECONDS)
+ .pollInterval(WAIT_INTERVAL_IN_SECONDS, TimeUnit.SECONDS)
+ .untilAsserted(
+ () ->
+ assertContainsProperties(
+ findIndex(tc.loadTable(alterId), "idx_alter"),
requestedProperties));
+
+ NameIdentifier nativeId = NameIdentifier.of(schemaName,
"t_inverted_properties_native");
+ executeSql(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + "`%s` BIGINT NOT NULL, "
+ + "`%s` VARCHAR(100), "
+ + "INDEX `idx_native` (`%s`) USING INVERTED "
+ + "PROPERTIES(\"parser\"=\"english\",
\"support_phrase\"=\"true\")"
+ + ") DISTRIBUTED BY HASH(`%s`) BUCKETS 1 "
+ + "PROPERTIES(\"replication_num\"=\"1\")",
+ schemaName, nativeId.name(), colName1, colName2, colName2,
colName1));
+ assertContainsProperties(findIndex(tc.loadTable(nativeId), "idx_native"),
requestedProperties);
+ }
+
@Test
void testAddAndDropInvertedIndex() {
TableCatalog tc = catalog.asTableCatalog();
@@ -574,6 +659,27 @@ public class CatalogDoris4xIT extends BaseIT {
"light_schema_change=true should appear after ALTER TABLE
SET"));
}
+ private void executeSql(String sql) throws SQLException {
+ try (Connection connection =
+ DriverManager.getConnection(
+ jdbcUrl, DorisContainer.USER_NAME, DorisContainer.PASSWORD);
+ Statement statement = connection.createStatement()) {
+ statement.execute(sql);
+ }
+ }
+
+ private Index findIndex(Table table, String indexName) {
+ return Arrays.stream(table.index())
+ .filter(index -> index.name().equals(indexName))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("Index not found: " +
indexName));
+ }
+
+ private void assertContainsProperties(Index index, Map<String, String>
expectedProperties) {
+ expectedProperties.forEach(
+ (key, value) -> assertEquals(value, index.properties().get(key),
"Property: " + key));
+ }
+
private Column findColumn(Table table, String columnName) {
return Arrays.stream(table.columns())
.filter(c -> c.name().equals(columnName))
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
index af1bc0dc47..105549ff7c 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableOperationsSqlGeneration.java
@@ -22,11 +22,15 @@ import static
org.apache.gravitino.catalog.doris.DorisTablePropertiesMetadata.RE
import static
org.apache.gravitino.catalog.doris.DorisTablePropertiesMetadata.REPLICATION_FACTOR;
import java.sql.Connection;
+import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
import java.sql.Statement;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
import java.util.Map;
import javax.sql.DataSource;
import org.apache.gravitino.catalog.doris.converter.DorisTypeConverter;
@@ -118,6 +122,11 @@ public class TestDorisTableOperationsSqlGeneration {
distribution,
indexes);
}
+
+ List<Index> indexes(Connection connection, String databaseName, String
tableName)
+ throws SQLException {
+ return getIndexes(connection, databaseName, tableName);
+ }
}
@Test
@@ -288,6 +297,112 @@ public class TestDorisTableOperationsSqlGeneration {
"Should generate INVERTED index: " + sql);
}
+ @Test
+ public void
testCreateAndAlterInvertedIndexPropertiesUseDeterministicRendering() {
+ TestableDorisTableOperations ops = new TestableDorisTableOperations();
+ JdbcColumn idCol =
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build();
+ JdbcColumn nameCol =
+ JdbcColumn.builder()
+ .withName("name")
+ .withType(Types.VarCharType.of(100))
+ .withNullable(true)
+ .build();
+ Distribution distribution = Distributions.hash(1,
NamedReference.field("id"));
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put("support_phrase", "true");
+ properties.put("parser", "english");
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(Index.IndexType.INVERTED, "idx_name", new String[][]
{{"name"}}, properties)
+ };
+
+ TestableDorisTableOperations mockOps = Mockito.spy(ops);
+ Mockito.doAnswer(a -> a.getArgument(0))
+ .when(mockOps)
+ .appendNecessaryProperties(Mockito.anyMap());
+
+ String expectedDefinition =
+ "INDEX `idx_name` (`name`) USING INVERTED PROPERTIES (\n"
+ + "\"parser\"=\"english\",\n"
+ + "\"support_phrase\"=\"true\"\n)";
+ String createSql =
+ mockOps.createTableSqlWithIndexes(
+ "test_inverted", new JdbcColumn[] {idCol, nameCol}, distribution,
indexes);
+ Assertions.assertTrue(createSql.contains(expectedDefinition), createSql);
+
+ TableChange.AddIndex addIndex =
+ (TableChange.AddIndex)
+ TableChange.addIndex(
+ Index.IndexType.INVERTED, "idx_name", new String[][]
{{"name"}}, properties);
+ Assertions.assertEquals(
+ "ADD " + expectedDefinition,
DorisTableOperations.addIndexDefinition(addIndex));
+ }
+
+ @Test
+ public void testInvertedIndexPropertiesEscapeSqlLiterals() {
+ TableChange.AddIndex addIndex =
+ (TableChange.AddIndex)
+ TableChange.addIndex(
+ Index.IndexType.INVERTED,
+ "idx_name",
+ new String[][] {{"name"}},
+ Map.of("char_filter_pattern", "owner's \"comment\"
C:\\tmp,=中文"));
+
+ Assertions.assertEquals(
+ "ADD INDEX `idx_name` (`name`) USING INVERTED PROPERTIES (\n"
+ + "\"char_filter_pattern\"=\"owner's \"\"comment\"\"
C:\\\\tmp,=中文\"\n)",
+ DorisTableOperations.addIndexDefinition(addIndex));
+ }
+
+ @Test
+ public void testInvertedIndexPropertiesRejectUnsafeInput() {
+ Map<String, String> nullKey = new HashMap<>();
+ nullKey.put(null, "value");
+ Map<String, String> blankKey = new HashMap<>();
+ blankKey.put(" ", "value");
+ Map<String, String> nullValue = new HashMap<>();
+ nullValue.put("parser", null);
+
+ for (Map<String, String> properties :
+ List.of(
+ nullKey,
+ blankKey,
+ nullValue,
+ Map.of("bad\nkey", "value"),
+ Map.of("parser", "bad\tvalue"))) {
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ DorisTableOperations.addIndexDefinition(
+ (TableChange.AddIndex)
+ TableChange.addIndex(
+ Index.IndexType.INVERTED,
+ "idx_name",
+ new String[][] {{"name"}},
+ properties)));
+ }
+ }
+
+ @Test
+ public void testNonInvertedIndexPropertiesRemainUnchanged() {
+ TableChange.AddIndex addIndex =
+ (TableChange.AddIndex)
+ TableChange.addIndex(
+ Index.IndexType.VECTOR,
+ "idx_vec",
+ new String[][] {{"embedding"}},
+ Map.of("index_type", "hnsw"));
+
+ Assertions.assertEquals(
+ "ADD INDEX `idx_vec` (`embedding`) USING ANN",
+ DorisTableOperations.addIndexDefinition(addIndex));
+ }
+
@Test
public void testCreateTableWithBitmapIndex() {
TestableDorisTableOperations ops = new TestableDorisTableOperations();
@@ -351,6 +466,106 @@ public class TestDorisTableOperationsSqlGeneration {
Index.IndexType.UNIQUE_KEY,
DorisTableOperations.mapDorisIndexType(null, "idx_name"));
}
+ @Test
+ public void testParseIndexProperties() {
+ Assertions.assertTrue(DorisTableOperations.parseIndexProperties(null,
"idx").isEmpty());
+ Assertions.assertTrue(DorisTableOperations.parseIndexProperties("",
"idx").isEmpty());
+ Assertions.assertTrue(DorisTableOperations.parseIndexProperties("( )",
"idx").isEmpty());
+
+ Map<String, String> properties =
+ DorisTableOperations.parseIndexProperties(
+ "( \"support_phrase\" = \"true\", "
+ + "\"char_filter_pattern\" = \"._=:,\", "
+ + "\"path\" = \"C:\\tmp\" )",
+ "idx");
+
+ Assertions.assertEquals(
+ Map.of(
+ "support_phrase", "true",
+ "char_filter_pattern", "._=:,",
+ "path", "C:\\tmp"),
+ properties);
+ }
+
+ @Test
+ public void testParseIndexPropertiesRejectsMalformedMetadata() {
+ for (String propertiesText :
+ List.of(
+ "\"parser\" = \"english\"",
+ "(\"parser\" \"english\")",
+ "(\"parser\" = \"english\",)",
+ "(\"parser\" = \"english\" trailing)",
+ "(\"parser\" = \"english)",
+ "(\"parser\" = \"english\", \"parser\" = \"unicode\")")) {
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> DorisTableOperations.parseIndexProperties(propertiesText,
"idx_name"));
+ }
+ }
+
+ @Test
+ public void testGetIndexesReadsOnlyInvertedProperties() throws Exception {
+ TestableDorisTableOperations ops = new TestableDorisTableOperations();
+ Connection connection = Mockito.mock(Connection.class);
+ PreparedStatement statement = Mockito.mock(PreparedStatement.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+ ResultSetMetaData metaData = Mockito.mock(ResultSetMetaData.class);
+
+ Mockito.when(connection.prepareStatement("SHOW INDEX FROM `table` FROM
`database`"))
+ .thenReturn(statement);
+ Mockito.when(statement.executeQuery()).thenReturn(resultSet);
+ Mockito.when(resultSet.getMetaData()).thenReturn(metaData);
+ Mockito.when(metaData.getColumnCount()).thenReturn(5);
+ Mockito.when(metaData.getColumnName(1)).thenReturn("Key_name");
+ Mockito.when(metaData.getColumnName(2)).thenReturn("Column_name");
+ Mockito.when(metaData.getColumnName(3)).thenReturn("Index_type");
+ Mockito.when(metaData.getColumnName(4)).thenReturn("Properties");
+ Mockito.when(metaData.getColumnName(5)).thenReturn("Comment");
+ Mockito.when(resultSet.next()).thenReturn(true, true, true, false);
+ Mockito.when(resultSet.getString("Key_name"))
+ .thenReturn("idx_first", "idx_ngram", "idx_second");
+ Mockito.when(resultSet.getString("Column_name")).thenReturn("text_a",
"text_ngram", "text_b");
+ Mockito.when(resultSet.getString("Index_type")).thenReturn("INVERTED",
"NGRAM_BF", "INVERTED");
+ Mockito.when(resultSet.getString("Properties"))
+ .thenReturn("(\"parser\" = \"english\")", "(\"support_phrase\" =
\"true\")");
+
+ List<Index> indexes = ops.indexes(connection, "database", "table");
+
+ Assertions.assertEquals(3, indexes.size());
+ Assertions.assertEquals(Map.of("parser", "english"),
indexes.get(0).properties());
+ Assertions.assertTrue(indexes.get(1).properties().isEmpty());
+ Assertions.assertEquals(Map.of("support_phrase", "true"),
indexes.get(2).properties());
+ Mockito.verify(resultSet, Mockito.times(2)).getString("Properties");
+ }
+
+ @Test
+ public void testGetIndexesWithoutPropertiesColumnKeepsEmptyMap() throws
Exception {
+ TestableDorisTableOperations ops = new TestableDorisTableOperations();
+ Connection connection = Mockito.mock(Connection.class);
+ PreparedStatement statement = Mockito.mock(PreparedStatement.class);
+ ResultSet resultSet = Mockito.mock(ResultSet.class);
+ ResultSetMetaData metaData = Mockito.mock(ResultSetMetaData.class);
+
+ Mockito.when(connection.prepareStatement("SHOW INDEX FROM `table` FROM
`database`"))
+ .thenReturn(statement);
+ Mockito.when(statement.executeQuery()).thenReturn(resultSet);
+ Mockito.when(resultSet.getMetaData()).thenReturn(metaData);
+ Mockito.when(metaData.getColumnCount()).thenReturn(3);
+ Mockito.when(metaData.getColumnName(1)).thenReturn("Key_name");
+ Mockito.when(metaData.getColumnName(2)).thenReturn("Column_name");
+ Mockito.when(metaData.getColumnName(3)).thenReturn("Index_type");
+ Mockito.when(resultSet.next()).thenReturn(true, false);
+ Mockito.when(resultSet.getString("Key_name")).thenReturn("idx_name");
+ Mockito.when(resultSet.getString("Column_name")).thenReturn("text");
+ Mockito.when(resultSet.getString("Index_type")).thenReturn("INVERTED");
+
+ List<Index> indexes = ops.indexes(connection, "database", "table");
+
+ Assertions.assertEquals(1, indexes.size());
+ Assertions.assertTrue(indexes.get(0).properties().isEmpty());
+ Mockito.verify(resultSet, Mockito.never()).getString("Properties");
+ }
+
@Test
public void testCreateTableWithAutoIncrement() {
TestableDorisTableOperations ops = new TestableDorisTableOperations();
diff --git a/docs/jdbc-doris-catalog.md b/docs/jdbc-doris-catalog.md
index d914f310e8..0a92de67fd 100644
--- a/docs/jdbc-doris-catalog.md
+++ b/docs/jdbc-doris-catalog.md
@@ -232,13 +232,16 @@ The Doris catalog supports the following index types.
Each index applies to a si
|----------------------|------------------------------------------------------------------------|---------------|
| `PRIMARY_KEY` | `` INDEX `PRIMARY` (col) `` (in the INDEX clause, no
USING) | 1.2+ |
| `UNIQUE_KEY` | `UNIQUE KEY(col)` (in the table model section, not
INDEX clause) | 1.2+ |
-| `INVERTED` | `INDEX name (col) USING INVERTED`
| 3.0+ |
+| `INVERTED` | `INDEX name (col) USING INVERTED [PROPERTIES(...)]`
| 3.0+ |
| `BITMAP` | `INDEX name (col)` (bare, no USING clause;
write-only, see note below) | 1.2+ |
| `VECTOR` | `INDEX name (col) USING ANN`
| 4.0.6+ |
:::note
- `PRIMARY_KEY` stays in the INDEX clause as a bare index (e.g. `` INDEX
`PRIMARY` (`id`) ``), with no USING clause.
- `UNIQUE_KEY` is emitted as a table model declaration (e.g. `` UNIQUE
KEY(`id`) ``), outside the INDEX clause.
+- `INVERTED` properties can be supplied through `Index.properties()` for
CREATE TABLE or `TableChange.AddIndex.getProperties()` for ALTER TABLE. Loaded
native and Gravitino-created INVERTED indexes expose the effective Doris
properties through `Index.properties()`. Supported keys, values, and
server-added defaults depend on the Doris version and are validated by Doris.
+- `SHOW INDEX` does not escape embedded double quotes in property keys or
values. Gravitino therefore does not guarantee their round-trip and rejects
metadata that falls outside the supported flat quoted-pair format.
+- Index comments are not currently represented by the Gravitino `Index` API
and are not preserved on round-trip.
- `BITMAP` is a write-only legacy type for backward compatibility with Doris
1.2.x. The write path generates a bare `INDEX` (no USING clause), but the read
path maps it back to `INVERTED` because Doris 4.0.6 removed BITMAP from the
grammar. Creating a BITMAP index and reading it back will show `INVERTED`.
:::
@@ -282,7 +285,11 @@ Index[] indexes = new Index[] {
{
"indexType": "inverted",
"name": "idx_name",
- "fieldNames": [["name"]]
+ "fieldNames": [["name"]],
+ "properties": {
+ "parser": "english",
+ "support_phrase": "true"
+ }
}
]
}
@@ -293,7 +300,11 @@ Index[] indexes = new Index[] {
```java
Index[] indexes = new Index[] {
- Indexes.of(IndexType.INVERTED, "idx_name", new String[][]{{"name"}},
Map.of())
+ Indexes.of(
+ IndexType.INVERTED,
+ "idx_name",
+ new String[][]{{"name"}},
+ Map.of("parser", "english", "support_phrase", "true"))
};
```
