jiangxt2 opened a new issue, #12843:
URL: https://github.com/apache/gravitino/issues/12843
### What would you like to be improved?
The ClickHouse catalog fails to load a valid MergeTree table when its
`PARTITION BY` clause uses an expression outside the current identity/date
Transform subset, for example:
```sql
PARTITION BY cityHash64(toString(sm4_cipher_msg)) % 7
```
The failure blocks ordinary table metadata access. Returning only an empty
Transform array would also be ambiguous because callers could not distinguish
an unpartitioned table from a native expression that Gravitino cannot structure.
#### Current behavior
The load path parses the SHOW CREATE statement before reading the
database-qualified `system.tables.partition_key` fallback. The strict parser
raises `UnsupportedOperationException`, although ClickHouse has already
accepted and stored the expression.
#### How to reproduce
1. Create a table:
```sql
CREATE TABLE hash_partition
(
sm4_cipher_msg String,
payload UInt32
)
ENGINE = MergeTree
ORDER BY tuple()
PARTITION BY cityHash64(toString(sm4_cipher_msg)) % 7;
```
2. Register a Gravitino JDBC ClickHouse catalog pointing to the database.
3. Load the table through the Java API:
```java
catalog.loadTable(NameIdentifier.of("default", "hash_partition"));
```
Alternatively, use the REST API:
```text
GET
/api/metalakes/{metalake}/catalogs/{catalog}/schemas/default/tables/hash_partition
```
4. Observe an error response indicating that the partition expression is
unsupported.
#### Expected behavior
- `loadTable()` succeeds.
- `table.properties().get("partition-key")` returns
`cityHash64(toString(sm4_cipher_msg)) % 7`.
- `table.partitioning()` remains empty because there is no equivalent
current Transform.
- Known identity/date expressions continue returning their existing
Transforms and also expose the canonical native property.
- The property contains ClickHouse's canonical partition expression as
returned by `system.tables.partition_key`; original whitespace and quoting are
not guaranteed to be preserved.
### How should we improve?
#### Solution overview for pre-development discussion
This solution overview is provided for maintainer discussion before
implementation. No implementation has started, and the final property name and
conversion contract are subject to reviewer agreement.
- Treat `system.tables.partition_key` as the authoritative ClickHouse-native
expression.
- Return every non-empty value through a visible, reserved `partition-key`
table property.
- Preserve the existing identity, `toDate`, `toYear`, and `toYYYYMM`
Transform mappings.
- Make structural conversion best-effort so unsupported valid expressions do
not block `loadTable()`.
- Require simple field arguments for existing date mappings so nested
expressions are not misrepresented.
- Remove the partition-specific SHOW CREATE regex path.
- Preserve the synthetic property explicitly in
`convertFromJdbcProperties()` while filtering it from
`transformToJdbcProperties()`.
- Keep the property output-only and reject create, set, and remove requests
for it.
#### Scope and boundaries
Structured conversion remains limited to the seven engines already
represented by the ClickHouse catalog: MergeTree, ReplacingMergeTree,
SummingMergeTree, AggregatingMergeTree, CollapsingMergeTree,
VersionedCollapsingMergeTree, and GraphiteMergeTree. The read-only property may
also be returned for another engine that the existing catalog already loads
successfully, but this change does not add DDL, partition, or engine support
for ReplicatedMergeTree variants, Distributed tables, or other engines.
This change does not add physical partition listing, partition mutations, a
complete ClickHouse parser, or a new public partitioning type. `partition-key`
is ClickHouse-specific fidelity metadata; it is not the generic Gravitino
partitioning API and does not represent physical partitions.
#### Compatibility and user-facing change
Yes. Loaded ClickHouse tables with a non-empty native partition key expose a
new visible, reserved `partition-key` property. Existing partitioning JSON and
supported DDL behavior remain unchanged. The property contains ClickHouse's
canonical expression rather than a byte-for-byte copy of the original CREATE
statement. It is read-only: callers cannot supply it during create, set it
during alter, or remove it.
#### Reviewer discussion points
- Is a visible, reserved `partition-key` property the appropriate
ClickHouse-specific fidelity channel?
- Is ClickHouse canonical expression text, rather than byte-for-byte CREATE
SQL, the correct contract?
- Is the asymmetric conversion contract acceptable: preserve on read
conversion and always filter on write conversion?
- Is the ClickHouse-specific two-channel behavior acceptable when generic
callers inspect only `Table.partitioning()`?
- Are the engine, physical partition, and mutation boundaries sufficiently
explicit for implementation?
#### References
- [ClickHouse
system.tables](https://clickhouse.com/docs/reference/system-tables/tables)
- [ClickHouse MergeTree table
engine](https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/mergetree)
- [Gravitino ClickHouse partition
parser](https://github.com/apache/gravitino/blob/main/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java)
- [Gravitino reserved
properties](https://github.com/apache/gravitino/blob/main/core/src/main/java/org/apache/gravitino/connector/PropertyEntry.java)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
