jiangxt2 opened a new issue, #11975: URL: https://github.com/apache/gravitino/issues/11975
### Describe the feature The Doris catalog does not support Doris table Key Models (`PRIMARY KEY`, `UNIQUE KEY`, `AGGREGATE KEY`, `DUPLICATE KEY`). **On `loadTable`**: All four Key Models are silently lost. The table's indexes and sort orders do not reflect the key columns. There is no warning or error. **On `createTable`**: Gravitino never emits a Key Model clause, so Doris defaults to `DUPLICATE KEY(<all columns>)`. A user who loads a UNIQUE KEY table and recreates it through Gravitino will get a DUPLICATE KEY table — losing all uniqueness guarantees. ### Motivation Key Model is a core semantic property of a Doris table. It determines: - Whether duplicate rows are allowed (DUPLICATE) or rejected (UNIQUE / PRIMARY). - How data is pre-aggregated on write (AGGREGATE KEY with SUM / REPLACE / HLL functions). - Which columns define the table's sort order for data layout. Without Key Model support, tables created through Gravitino lose their intended data integrity semantics. Applications that rely on specific Key Models for correctness cannot use Gravitino as the table creation path. ### Describe the solution The Key Model information is present in `SHOW CREATE TABLE` output (e.g., `UNIQUE KEY(id, name)`) but is currently not parsed. The table-level Key Model is not available through existing index metadata, which only covers secondary indexes. The fix needs to: 1. Parse the Key Model clause from the table DDL in the load path. 2. Generate the Key Model clause in the createTable DDL generation path. 3. Map Key Model to Gravitino's table model — this may need API-level discussion since Key Model isn't a concept shared across all JDBC catalogs. The aggregate function types for AGGREGATE KEY columns (SUM, REPLACE, HLL, BITMAP_UNION) also need to be preserved during round-trip. ### Test plan | # | Scenario | Expected after fix | |---|----------|-------------------| | 1 | DUPLICATE KEY table → loadTable | indexes reflect DUPLICATE KEY columns | | 2 | UNIQUE KEY (single column) → loadTable | indexes reflect UNIQUE KEY, sort orders correct | | 3 | UNIQUE KEY (multi-column) → loadTable | all key columns present in indexes | | 4 | AGGREGATE KEY + SUM function → loadTable | KEY columns + aggregate function type preserved | | 5 | createTable via Gravitino with KEY specification | DDL contains correct KEY clause | | 6 | Round-trip: create → load → KEY matches original | no semantic loss | | 7 | Table without explicit KEY (default) | no regression, behavior unchanged | ### Additional context **Test results** (Docker + Gravitino 1.2.0, before fix): | # | Scenario | Gravitino loadTable | Native SHOW CREATE TABLE | |---|----------|:-------------------:|:------------------------:| | 1 | DUPLICATE KEY(`id`) | ❌ `indexes: []` | ✅ `DUPLICATE KEY(\`id\`)` | | 2 | UNIQUE KEY(`id`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`)` | | 3 | UNIQUE KEY(`id`, `name`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`, \`name\`)` | | 4 | AGGREGATE KEY(`site`, `city`) + `pv BIGINT SUM` | ❌ `indexes: []`, SUM lost | ✅ `AGGREGATE KEY(...)`, `pv bigint SUM` | | 5 | Gravitino createTable | ❌ no KEY → defaults to `DUPLICATE KEY(all cols)` | — | All four Key Models return empty indexes across all tested Doris versions. Multi-column keys and aggregate column functions are also lost. I'd like to pick this up. Please let me know if you have any feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
