TimothyDing opened a new pull request, #9963:
URL: https://github.com/apache/gravitino/pull/9963
### What changes were proposed in this pull request?
This PR adds full Hologres JDBC catalog support to Apache Gravitino under
the `catalogs-contrib` module. Hologres is a real-time data warehouse service
provided by Alibaba Cloud, fully compatible with the PostgreSQL protocol.
The changes include:
#### 1. Backend — Hologres Catalog Implementation
(`catalogs-contrib/catalog-jdbc-hologres`)
- **HologresCatalog / HologresCatalogOperations**: Core catalog entry point,
wired with Hologres-specific converters and operations.
- **HologresTableOperations** (~1,089 lines): Full DDL support including:
- `CREATE TABLE` with Hologres-specific `WITH` clause properties
(`orientation`, `clustering_key`, `segment_key`, `bitmap_columns`,
`dictionary_encoding_columns`, `distribution_key`, `time_to_live_in_seconds`,
`table_group`, `storage_format`, `binlog_level`, `binlog_ttl`).
- HASH distribution via `distribution_key`.
- LIST partitioning — both physical (`PARTITION BY LIST`) and logical
(`LOGICAL PARTITION BY LIST`, Hologres V3.1+).
- PRIMARY KEY index support in CREATE TABLE.
- Generated (stored computed) column support (`GENERATED ALWAYS AS (expr)
STORED`).
- Column default value support.
- Filtering of views (`v`) and foreign tables (`f`) from table listing.
- Reading table properties from `hologres.hg_table_properties` system
table.
- Reading partition info from `pg_partitioned_table` and `pg_attribute`.
- ALTER TABLE support for: `RenameTable`, `UpdateComment`, `AddColumn`
(type and comment only), `DeleteColumn`, `RenameColumn`, `UpdateColumnComment`.
- Explicit rejection (throws `IllegalArgumentException`) for unsupported
ALTER operations: `UpdateColumnType`, `UpdateColumnDefaultValue`,
`UpdateColumnNullability`, `UpdateColumnPosition`, `UpdateColumnAutoIncrement`,
`AddIndex`, `DeleteIndex`, `SetProperty`, `RemoveProperty`.
- Auto-increment column creation is blocked in both CREATE TABLE and ALTER
TABLE.
- **HologresSchemaOperations**: Schema CRUD with comment support and system
schema filtering (11 system schemas: `pg_toast`, `pg_catalog`,
`information_schema`, `hologres`, `hg_internal`, `hg_recyclebin`,
`hologres_object_table`, `hologres_sample`, `hologres_streaming_mv`,
`hologres_statistic`, `hologres_sample_internal`).
- **HologresTypeConverter**: Bidirectional type mapping for 16 base types +
6 array types. Handles Hologres-specific behaviors like no precision suffix for
`TIMESTAMP`/`TIMESTAMPTZ`, array types via `_` prefix, and unmapped types as
`ExternalType`.
- **HologresColumnDefaultValueConverter**: Handles literal, function, and
unparsed default value expressions.
- **HologresExceptionConverter**: Maps PostgreSQL error codes to Gravitino
exceptions.
- **HologresCatalogCapability**: Naming rules (max 63 chars,
letter/underscore start) and reserved schema names.
#### 2. Unit Tests (`catalogs-contrib/catalog-jdbc-hologres/src/test`)
- **TestHologresTableOperations** (~833 lines): Comprehensive tests covering
CREATE TABLE DDL generation (with properties, distribution, partitioning,
indexes, generated columns), ALTER TABLE operations (supported and rejected),
table loading, and edge cases.
- **TestHologresTypeConverter**: Full round-trip type mapping tests.
- **TestHologresColumnDefaultValueConverter**: Default value parsing tests.
- **TestHologresExceptionConverter**: Exception mapping tests.
- **TestHologresCatalogCapability**: Naming validation tests.
#### 3. Bug Fixes in Core Modules
- **JsonUtils.java** (`common`): Fixed `DistributionDTO` deserialization to
make the `number` field optional (guarded by `node.has(NUMBER)`). Previously,
omitting `number` in the JSON payload caused a missing-field exception.
- **MetadataAuthzHelper.java** (`server-common`): Added exception handling
in `preloadEntities()` for batch cache preloading. When browsing catalogs with
unregistered metadata (read-only mode), the `batchGet` call could throw,
blocking access to external catalog data. Now gracefully catches and logs the
exception.
- **TableView.js** (`web`): Fixed default value rendering to properly
display `unparsed` expression type (e.g., `CURRENT_TIMESTAMP`) and `function`
type default values, in addition to literal values.
#### 4. Web UI Integration (`web/`)
- Added Hologres provider configuration in `initial.js` (provider
definition, column type mapping, table property config).
- Added Hologres SVG icon and CSS icon class.
- Added Hologres icon mapping in `MetalakeTree.js`.
#### 5. Build Configuration
- Registered `catalog-jdbc-hologres` module in `settings.gradle.kts`.
- Added `copyLibAndConfig` task dependency in root `build.gradle.kts` for
`package-all` distribution.
#### 6. Documentation (`docs/`)
- Created `jdbc-hologres-catalog.md`: Complete Hologres catalog
documentation covering capabilities, properties, type mapping, distribution,
partitioning, indexes, and ALTER operations.
- Updated `manage-relational-metadata-using-gravitino.md`: Added Hologres
references to 6 cross-reference tables.
- Updated `webui.md`: Added Hologres to provider list and TabItem
configuration.
### Why are the changes needed?
Hologres is widely used in the Alibaba Cloud ecosystem for real-time OLAP
workloads. Adding Hologres catalog support to Gravitino enables users to:
1. **Unified metadata management**: Manage Hologres schemas and tables
alongside other data sources (MySQL, PostgreSQL, Hive, Iceberg, etc.) through a
single Gravitino interface.
2. **Read-only browsing and DDL operations**: Browse existing Hologres
metadata and perform supported DDL operations via Gravitino's REST API and Web
UI.
3. **Hologres-specific feature support**: Properly handle Hologres-unique
features such as table properties (`orientation`, `clustering_key`,
`segment_key`), logical partitioning, and PostgreSQL-compatible type system.
The core module fixes (JsonUtils, MetadataAuthzHelper, TableView) address
real bugs encountered during Hologres integration that also affect other
catalog types.
### Does this PR introduce _any_ user-facing change?
Yes:
1. **New catalog provider**: Users can now create a Hologres catalog with
provider `jdbc-hologres` via REST API or Web UI.
2. **Web UI**: Hologres appears as a selectable provider when creating
relational catalogs, with its own icon and property form.
3. **REST API**: The `number` field in distribution JSON payload is now
optional (backward compatible — existing requests with `number` still work).
4. **Default value display**: The Web UI now correctly renders `unparsed`
and `function` type default values for all catalog types.
### How was this patch tested?
1. **Unit tests**: 5 new test classes with comprehensive coverage:
- `TestHologresTableOperations` (833 lines) — DDL generation, ALTER
operations, table loading
- `TestHologresTypeConverter` — All type mappings (round-trip)
- `TestHologresColumnDefaultValueConverter` — Default value parsing
- `TestHologresExceptionConverter` — Exception mapping
- `TestHologresCatalogCapability` — Naming validation
Run with: `./gradlew :catalogs-contrib:catalog-jdbc-hologres:test`
2. **Manual testing**: Verified against a live Hologres instance:
- Created catalogs, schemas, and tables via REST API and Web UI
- Tested table listing (views and foreign tables correctly filtered)
- Tested table property reading from `hg_table_properties`
- Tested partition info retrieval (physical and logical)
- Verified ALTER TABLE operations (both supported and rejected)
- Verified Web UI default value rendering for all value types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
