JoegenUSTC opened a new issue, #11841:
URL: https://github.com/apache/gravitino/issues/11841
### Version
main branch
### Describe what's wrong
Two problems in the ClickHouse partition expression handling:
**1. Day transform maps to `toDate()` instead of `toYYYYMMDD()`.**
`toPartitionExpression` outputs `toDate(col)` for the `day` transform.
`toDate()` returns a `Date` type, but ClickHouse's integer day-partitioned
tables use `toYYYYMMDD()` which returns `UInt32` (e.g. `20230622`). This
mismatch means `loadTable()` + `createTable()` does not produce an equivalent
table — the partition expression type differs from what ClickHouse actually
stores.
**2. Nested partition function calls are not handled.**
`parsePartitionExpression` uses `normalizeIdentifier()` to strip backticks
from the inner argument of `toYear()`, `toYYYYMM()`, and `toDate()`. This fails
on nested expressions like `toYYYYMM(toDate(dt))` because
`normalizeIdentifier("toDate(dt)")` returns `"toDate(dt)"`, which contains
parentheses and is not a simple identifier. The function should recursively
peel wrapper functions to find the innermost column name.
### Error message and/or stacktrace
For problem 1: no error, but `loadTable` followed by `createTable` produces
a table with `PARTITION BY toDate(col)` instead of `PARTITION BY
toYYYYMMDD(col)`, meaning the round-trip does not preserve the original
partition semantics.
For problem 2: `IllegalArgumentException: Only simple identifier is
supported` when loading a table whose partition key contains nested function
wrappers.
### How to reproduce
1. Create a ClickHouse table with `PARTITION BY toYYYYMMDD(dt)`.
2. Load it via Gravitino → partition is parsed as `day("dt")`.
3. Create a new table with the same partition transform → Gravitino
generates `PARTITION BY toDate(dt)` instead of `toYYYYMMDD(dt)`.
For nested functions:
1. Create a ClickHouse table with `PARTITION BY
toYYYYMM(toDate(event_time))`.
2. Load it via Gravitino → `parsePartitionExpression` fails on
`toDate(event_time)`.
### Additional context
Both fixes are in `ClickHouseTableSqlUtils`:
- Change `toPartitionExpression` to output `toYYYYMMDD` for the `day`
transform.
- Add `extractInnermostField` to recursively unwrap nested date/time
functions (with max depth guard).
- Add `TO_DAY_PATTERN` so `toYYYYMMDD(...)` is recognized during parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
