jiangxt2 opened a new pull request, #12026:
URL: https://github.com/apache/gravitino/pull/12026
<!--
1. Title: [#11933] feat(clickhouse): support ngrambf_v1 and tokenbf_v1 data
skipping index types
-->
### What changes were proposed in this pull request?
Add support for `ngrambf_v1` and `tokenbf_v1` bloom filter data skipping
index types in the ClickHouse catalog. These indexes accelerate text search on
string columns via n-gram tokenization (`ngrambf_v1`) or non-alphanumeric
delimiter tokenization (`tokenbf_v1`). Both have been available since
ClickHouse v19.x.
- Add `DATA_SKIPPING_NGRAMBFV1` and `DATA_SKIPPING_TOKENBFV1` to
`Index.IndexType` enum
- Add ClickHouse type-name constants (`ngrambf_v1`, `tokenbf_v1`) and
`CLICKHOUSE_INDEX_TYPE_KEY` infrastructure constant in `ClickHouseConstants`
(shared infra for #11934/#11935)
- Extend read path: map `ngrambf_v1` / `tokenbf_v1` (including parameterized
formats like `ngrambf_v1(3, 512, 3, 0)`) to the new enum values
- Extend CREATE TABLE write path: generate `INDEX ... TYPE ngrambf_v1(n,
size, hash, seed)` / `tokenbf_v1(size, hash, seed)` DDL from
`Index.properties()`
- Add parameter validation helpers: `requirePositiveInt` (>= 1),
`requireNonNegativeInt` (>= 0 for random_seed); reuse `resolveGranularity` from
#11806 for GRANULARITY resolution
- ALTER TABLE ADD INDEX for these types throws
`UnsupportedOperationException` — the `TableChange.AddIndex` API does not yet
carry custom properties (#12013)
- Update `docs/jdbc-clickhouse-catalog.md` to list the new types with their
required properties
### Why are the changes needed?
Tables with `ngrambf_v1` or `tokenbf_v1` indexes currently load without
error in Gravitino but their index metadata is silently dropped. Creating
tables with these index types throws `IllegalArgumentException`. This PR
enables full read (preserve metadata) and CREATE TABLE write support, matching
the coverage already available for `minmax`, `bloom_filter`, and `set`.
### Does this PR introduce _any_ user-facing change?
Yes — two new `Index.IndexType` enum values:
- `DATA_SKIPPING_NGRAMBFV1` — requires `ngram_size`, `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`; optional `granularity`
(default 1)
- `DATA_SKIPPING_TOKENBFV1` — requires `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`; optional `granularity`
(default 1)
ALTER TABLE ADD INDEX for these types is not yet supported (throws
`UnsupportedOperationException`). Use CREATE TABLE instead.
### How was this patch tested?
- Unit tests: type mapping (exact + parameterized format), DDL generation
with all parameter combinations, missing/invalid parameter validation, ALTER
TABLE `UnsupportedOperationException`
- Docker integration test: CREATE TABLE with both index types + round-trip
verification (ClickHouse 24.8.14 in Testcontainers)
```
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:spotlessCheck ✅
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test -PskipITs ✅
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test \
--tests "CatalogClickHouseIT" -PskipDockerTests=false ✅
```
Closes: #11933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
