JoegenUSTC opened a new pull request, #11842:
URL: https://github.com/apache/gravitino/pull/11842
### What changes were proposed in this pull request?
- Change `toPartitionExpression` to output `toYYYYMMDD()` for the `day`
transform instead of `toDate()`. `toYYYYMMDD` returns `UInt32` (e.g.
`20230622`), matching ClickHouse's actual integer-based day partition format.
`toDate` returns a `Date` type with different semantics, breaking round-trip
through `loadTable` + `createTable`.
- Add `extractInnermostField` method that recursively unwraps known
ClickHouse date/time function wrappers to find the innermost simple column
identifier. Used in `parsePartitionExpression` for `toYear`, `toYYYYMM`,
`toDate`, and `toYYYYMMDD` branches. `normalizeIdentifier` alone only strips
backticks and cannot handle nested calls like `toYYYYMM(toDate(dt))`.
- Add `TO_DAY_PATTERN` so `toYYYYMMDD(...)` is properly recognized during
partition parsing.
- Add comprehensive unit tests covering `toPartitionExpression` round-trip,
`extractInnermostField`
(simple/backtick-quoted/single-wrapper/nested/complex-null), and
`parsePartitioning` round-trip for all supported partition expressions
including nested forms.
### Why are the changes needed?
1. **Day transform type mismatch**: loading a table with `PARTITION BY
toYYYYMMDD(dt)` and then creating a new table via Gravitino API produces
`PARTITION BY toDate(dt)` — a different SQL with different ClickHouse semantics.
2. **Nested expression parsing**: `toYYYYMM(toDate(dt))` fails to parse
because `normalizeIdentifier("toDate(dt)")` returns `"toDate(dt)"` which
contains parentheses — not a valid simple identifier.
This change is backward-compatible: `parsePartitionExpression` still
recognizes `toDate(col)`; only the reverse (generation) side changes.
Fix: #11841
### Does this PR introduce _any_ user-facing change?
No. Partition expression generation is internal to the ClickHouse catalog.
### How was this patch tested?
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:spotlessApply` —
passes
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:build` — passes
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test` — all tests
pass, including 13 new tests:
- `toPartitionExpression` round-trip: day/month/year/identity
- `extractInnermostField`: simple identifier, backtick-quoted, single
wrapper, nested toYYYYMM(toDate(dt)), nested toYear(toDate(dt)), complex
expression returning null
- `parsePartitioning` round-trip: toYYYYMMDD, toYYYYMM, toYear, nested
toYYYYMM, nested toYear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
