cwq222 opened a new pull request, #13749:
URL: https://github.com/apache/gravitino/pull/13749
**### What changes were proposed in this pull request?**
Use `system.columns.is_in_partition_key` to resolve the real partition
column for nested type-conversion expressions that the existing regex-based
parser cannot handle.
- Add `ClickHouseTableOperations.getPartitionColumns()` to query the columns
that participate in the partition key via `system.columns.is_in_partition_key`.
- Thread the resulting column list through `parsePartitioning` and
`parsePartitionExpression`. A single-argument `parsePartitioning` overload is
kept for existing callers, and `getTablePartitioning` also queries the
partition columns internally.
- Split the identifier extraction into two helpers:
- `extractSimpleIdentifier`: resolves only backtick-quoted or strict plain
column names (no fallback).
- `resolvePartitionIdentifier`: resolves a simple column name, and when
that fails and exactly one column participates in the partition, falls back to
that column.
- The granularity functions
(`toYear`/`toYYYYMM`/`toDate`/`toStartOfWeek`/`toStartOfMonth`) use the
fallback; the identity branch does not, so hash/arithmetic expressions such as
`cityHash64(x) % 7` remain unstructured.
-
**### Why are the changes needed?**
`PARTITION BY toYYYYMM(toDate(event_time))` is a valid ClickHouse pattern
whose inner expression (`toDate(event_time)`) cannot be resolved to a simple
column name, so it currently loads with an empty `partitioning()`. Using
`system.columns.is_in_partition_key` lets ClickHouse tell us the real column
(`event_time`) directly, avoiding the semantic pitfalls of unwrapping inner
expressions and handling nested/arithmetic/backtick forms for free.
Fix: #13318
**### Does this PR introduce _any_ user-facing change?**
Yes. A nested type-conversion partition expression such as
`toYYYYMM(toDate(event_time))` now structures as `month("event_time")` instead
of an empty `partitioning()`. The canonical expression is still preserved in
the read-only `partition-key` property. Expressions without a known granularity
(e.g. `cityHash64(x) % 7`) remain unstructured. Known limitation: a complex
expression inside a multi-column tuple, such as `(toDate(a + 1), b)`, still
remains unstructured because the participating columns cannot be matched to the
individual granularities.
**### How was this patch tested?**
- Unit test
`TestClickHouseTableOperationsPartitioning.testComplexExpressionFallsBackToPartitionColumn`:
nested/arithmetic expressions resolve to the participating column,
multi-column and hash expressions remain unstructured.
- `TestClickHouseTableOperationsUnit` mock adjusted for the new
`getPartitionColumns` call.
- Manual end-to-end: real ClickHouse + Gravitino server + REST, verifying
`toYYYYMM(toDate(event_time))` loads as `month("event_time")` and
`cityHash64(toString(x)) % 7` stays empty with the raw expression in
`partition-key`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
