This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 3a449c59d2 [#13490] feat(clickhouse): support quarter/year start
partition transforms (#13492)
3a449c59d2 is described below
commit 3a449c59d289c511ef050178fb455762e3eb4b2e
Author: Thunderkeg <[email protected]>
AuthorDate: Sat Oct 10 16:57:42 2026 +0800
[#13490] feat(clickhouse): support quarter/year start partition transforms
(#13492)
### What changes were proposed in this pull request?
Add structured read and create support for the one-column ClickHouse
partition expressions `toStartOfQuarter(column)` and
`toStartOfYear(column)`. When these tables are loaded, the catalog
preserves the function name and column reference in a function
transform; when tables are created, it emits the corresponding
ClickHouse expression.
### Why are the changes needed?
ClickHouse MergeTree tables can use these calendar boundary functions in
`PARTITION BY`, but the ClickHouse catalog currently leaves them
unstructured and exposes the native expressions only through the
read-only `partition-key` property. This prevents clients from handling
quarter- and year-based partitions through structured metadata across
load and create workflows.
Fix: #13490
### Does this PR introduce any user-facing change?
Yes. The ClickHouse catalog now supports one-column
`toStartOfQuarter(column)` and `toStartOfYear(column)` partition
transforms for MergeTree-family tables. Calls with additional arguments,
nested functions, and other unsupported expressions remain unstructured
and are preserved through the read-only `partition-key` property.
### How was this patch tested?
The equivalent Gradle 8.2 tasks passed using Corretto 17.0.18:
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test -PskipITs
--tests '*TestClickHouseTableOperationsPartitioning' --tests
'*TestClickHouseTableOperationsUnit'` — passed.
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
'org.apache.gravitino.catalog.clickhouse.integration.test.CatalogClickHouseIT.testLoadAndCreateWithStartOfDatePartitionTransforms'
-PskipDockerTests=false` — 1 test passed.
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
'org.apache.gravitino.catalog.clickhouse.integration.test.CatalogClickHouseClusterIT'
-PskipDockerTests=false` — 20 tests passed.
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:spotlessCheck
rat` — passed.
- `git diff --check` — passed.
Signed-off-by: jiangxt2 <[email protected]>
---
.../operations/ClickHouseTableSqlUtils.java | 25 ++++++++++++++
.../integration/test/CatalogClickHouseIT.java | 5 +--
.../TestClickHouseTableOperationsPartitioning.java | 38 ++++++++++++++++++++--
.../TestClickHouseTableOperationsUnit.java | 30 ++++++++++++++++-
docs/jdbc-clickhouse-catalog.md | 4 +--
5 files changed, 95 insertions(+), 7 deletions(-)
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
index 905a090fe2..1294ed6085 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
@@ -58,6 +58,10 @@ final class ClickHouseTableSqlUtils {
Pattern.compile("toStartOfWeek[(](.+)[)]", Pattern.CASE_INSENSITIVE);
private static final Pattern TO_START_OF_MONTH_PATTERN =
Pattern.compile("toStartOfMonth[(](.+)[)]", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_START_OF_QUARTER_PATTERN =
+ Pattern.compile("toStartOfQuarter[(](.+)[)]", Pattern.CASE_INSENSITIVE);
+ private static final Pattern TO_START_OF_YEAR_PATTERN =
+ Pattern.compile("toStartOfYear[(](.+)[)]", Pattern.CASE_INSENSITIVE);
private static final Pattern FUNCTION_WRAPPER_PATTERN =
Pattern.compile("^\\s*([A-Za-z0-9_]+)\\((.*)\\)\\s*$");
private static final Pattern PROJECTION_SETTING_NAME_PATTERN =
@@ -392,6 +396,10 @@ final class ClickHouseTableSqlUtils {
.formatted(quoteIdentifier(partitionFieldName(transform)));
case "tostartofmonth" -> "toStartOfMonth(%s)"
.formatted(quoteIdentifier(partitionFieldName(transform)));
+ case "tostartofquarter" -> "toStartOfQuarter(%s)"
+ .formatted(quoteIdentifier(partitionFieldName(transform)));
+ case "tostartofyear" -> "toStartOfYear(%s)"
+ .formatted(quoteIdentifier(partitionFieldName(transform)));
default -> throw new IllegalArgumentException(
"Unsupported partition transform: " + transform.name());
};
@@ -530,6 +538,23 @@ final class ClickHouseTableSqlUtils {
: Transforms.apply("toStartOfMonth", new Expression[]
{NamedReference.field(identifier)});
}
+ Matcher toStartOfQuarterMatcher =
TO_START_OF_QUARTER_PATTERN.matcher(trimmedExpression);
+ if (toStartOfQuarterMatcher.matches()) {
+ String identifier =
extractPartitionIdentifier(toStartOfQuarterMatcher.group(1));
+ return identifier == null
+ ? null
+ : Transforms.apply(
+ "toStartOfQuarter", new Expression[]
{NamedReference.field(identifier)});
+ }
+
+ Matcher toStartOfYearMatcher =
TO_START_OF_YEAR_PATTERN.matcher(trimmedExpression);
+ if (toStartOfYearMatcher.matches()) {
+ String identifier =
extractPartitionIdentifier(toStartOfYearMatcher.group(1));
+ return identifier == null
+ ? null
+ : Transforms.apply("toStartOfYear", new Expression[]
{NamedReference.field(identifier)});
+ }
+
String identifier = extractPartitionIdentifier(trimmedExpression);
return identifier == null ? null : Transforms.identity(identifier);
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index c52f77715b..f1705e2a3d 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -640,7 +640,7 @@ public class CatalogClickHouseIT extends BaseIT {
}
@Test
- void testLoadAndCreateWithStartOfWeekAndMonthPartitionTransforms() {
+ void testLoadAndCreateWithStartOfDatePartitionTransforms() {
TableCatalog tableCatalog = catalog.asTableCatalog();
Column[] columns =
new Column[] {
@@ -653,7 +653,8 @@ public class CatalogClickHouseIT extends BaseIT {
DEFAULT_VALUE_NOT_SET)
};
- for (String functionName : new String[] {"toStartOfWeek",
"toStartOfMonth"}) {
+ for (String functionName :
+ new String[] {"toStartOfWeek", "toStartOfMonth", "toStartOfQuarter",
"toStartOfYear"}) {
String nativeTableName =
GravitinoITUtils.genRandomName("native_start_partition");
String createdTableName =
GravitinoITUtils.genRandomName("created_start_partition");
String createNativeSql =
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
index a44edd8333..411403319e 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
@@ -49,9 +49,19 @@ public class TestClickHouseTableOperationsPartitioning {
Assertions.assertEquals(1, monthStartPartitions.length);
assertFunctionTransform(monthStartPartitions[0], "toStartOfMonth",
"event_time");
+ Transform[] quarterStartPartitions =
+ operations.parsePartitioning("toStartOfQuarter(event_time)");
+ Assertions.assertEquals(1, quarterStartPartitions.length);
+ assertFunctionTransform(quarterStartPartitions[0], "toStartOfQuarter",
"event_time");
+
+ Transform[] yearStartPartitions =
operations.parsePartitioning("toStartOfYear(event_time)");
+ Assertions.assertEquals(1, yearStartPartitions.length);
+ assertFunctionTransform(yearStartPartitions[0], "toStartOfYear",
"event_time");
+
// A native expression that cannot be structured returns an empty
transform array. The raw
// expression is instead exposed through the read-only partition-key
property during load.
Assertions.assertEquals(0,
operations.parsePartitioning("cityHash64(user_id) % 16").length);
+ Assertions.assertEquals(0,
operations.parsePartitioning("toStartOfDay(event_time)").length);
Transform[] identityPartitions =
operations.parsePartitioning("metric_type");
Assertions.assertEquals(1, identityPartitions.length);
@@ -63,10 +73,14 @@ public class TestClickHouseTableOperationsPartitioning {
assertSingleFieldTransform(tuplePartitions[1],
Transforms.NAME_OF_IDENTITY, "tenant_id");
Transform[] functionTuplePartitions =
- operations.parsePartitioning("(toStartOfWeek(ts),
toStartOfMonth(created_at))");
- Assertions.assertEquals(2, functionTuplePartitions.length);
+ operations.parsePartitioning(
+ "(toStartOfWeek(ts), toStartOfMonth(created_at), "
+ + "toStartOfQuarter(created_at), toStartOfYear(ts))");
+ Assertions.assertEquals(4, functionTuplePartitions.length);
assertFunctionTransform(functionTuplePartitions[0], "toStartOfWeek", "ts");
assertFunctionTransform(functionTuplePartitions[1], "toStartOfMonth",
"created_at");
+ assertFunctionTransform(functionTuplePartitions[2], "toStartOfQuarter",
"created_at");
+ assertFunctionTransform(functionTuplePartitions[3], "toStartOfYear", "ts");
Assertions.assertEquals(0, operations.parsePartitioning("tuple()").length);
Assertions.assertEquals(0, operations.parsePartitioning(" ").length);
@@ -80,6 +94,10 @@ public class TestClickHouseTableOperationsPartitioning {
Assertions.assertEquals(0,
operations.parsePartitioning("toYear(toString(event_time))").length);
Assertions.assertEquals(
0,
operations.parsePartitioning("toStartOfMonth(toDate(event_time))").length);
+ Assertions.assertEquals(
+ 0,
operations.parsePartitioning("toStartOfQuarter(toDate(event_time))").length);
+ Assertions.assertEquals(
+ 0,
operations.parsePartitioning("toStartOfYear(toDate(event_time))").length);
}
@Test
@@ -98,6 +116,15 @@ public class TestClickHouseTableOperationsPartitioning {
Assertions.assertEquals(1, monthStartPartitions.length);
assertFunctionTransform(monthStartPartitions[0], "toStartOfMonth",
"event-time");
+ Transform[] quarterStartPartitions =
+ operations.parsePartitioning("toStartOfQuarter(`event-time`)");
+ Assertions.assertEquals(1, quarterStartPartitions.length);
+ assertFunctionTransform(quarterStartPartitions[0], "toStartOfQuarter",
"event-time");
+
+ Transform[] yearStartPartitions =
operations.parsePartitioning("toStartOfYear(`event-time`)");
+ Assertions.assertEquals(1, yearStartPartitions.length);
+ assertFunctionTransform(yearStartPartitions[0], "toStartOfYear",
"event-time");
+
Transform[] identityPartitions =
operations.parsePartitioning("`event-time`");
Assertions.assertEquals(1, identityPartitions.length);
assertSingleFieldTransform(identityPartitions[0],
Transforms.NAME_OF_IDENTITY, "event-time");
@@ -110,6 +137,13 @@ public class TestClickHouseTableOperationsPartitioning {
0, operations.parsePartitioning("toStartOfWeek(event_time, 1,
'Asia/Shanghai')").length);
}
+ @Test
+ public void testQuarterAndYearAdditionalArgumentsRemainUnstructured() {
+ Assertions.assertEquals(
+ 0, operations.parsePartitioning("toStartOfQuarter(event_time,
1)").length);
+ Assertions.assertEquals(0,
operations.parsePartitioning("toStartOfYear(event_time, 1)").length);
+ }
+
private void assertFunctionTransform(
Transform transform, String expectedName, String expectedColumn) {
Assertions.assertEquals(expectedName, transform.name());
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
index 1759dce3df..cc8c1f4c85 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
@@ -159,6 +159,16 @@ public class TestClickHouseTableOperationsUnit {
ClickHouseTableSqlUtils.toPartitionExpression(
Transforms.apply(
"toStartOfMonth", new Expression[]
{NamedReference.field("event_time")})));
+ Assertions.assertEquals(
+ "toStartOfQuarter(`event_time`)",
+ ClickHouseTableSqlUtils.toPartitionExpression(
+ Transforms.apply(
+ "toStartOfQuarter", new Expression[]
{NamedReference.field("event_time")})));
+ Assertions.assertEquals(
+ "toStartOfYear(`event_time`)",
+ ClickHouseTableSqlUtils.toPartitionExpression(
+ Transforms.apply(
+ "toStartOfYear", new Expression[]
{NamedReference.field("event_time")})));
}
@Test
@@ -168,7 +178,25 @@ public class TestClickHouseTableOperationsUnit {
() ->
ClickHouseTableSqlUtils.toPartitionExpression(
Transforms.apply(
- "toStartOfQuarter", new Expression[]
{NamedReference.field("event_time")})));
+ "toStartOfDay", new Expression[]
{NamedReference.field("event_time")})));
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ClickHouseTableSqlUtils.toPartitionExpression(
+ Transforms.apply(
+ "toStartOfQuarter",
+ new Expression[] {
+ NamedReference.field("event_time"),
NamedReference.field("tenant_id")
+ })));
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ClickHouseTableSqlUtils.toPartitionExpression(
+ Transforms.apply(
+ "toStartOfYear",
+ new Expression[] {
+ FunctionExpression.of("toDate",
NamedReference.field("event_time"))
+ })));
Assertions.assertThrows(
IllegalArgumentException.class,
() ->
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index d01c601285..ddab8ff151 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -319,10 +319,10 @@ The `engine_parameters` property applies to
`ReplacingMergeTree`, `SummingMergeT
- Accept format: `id`, `(id, name)`, `(func(id), name)`, `func(id)`;
- Reject format: `(id + 1)`, `(func(id) + 1)`, etc.
-- `PARTITION BY`: single-column identity and some functions are supported
only, and only for MergeTree-family engines. For example `PARTITION BY
created_at`, `PARTITION BY toYYYYMM(created_at)`, `PARTITION BY
toStartOfWeek(created_at)`, and `PARTITION BY toStartOfMonth(created_at)` are
supported, but `PARTITION BY (created_at + 1)` is not supported.
+- `PARTITION BY`: single-column identity and some functions are supported
only, and only for MergeTree-family engines. For example `PARTITION BY
created_at`, `PARTITION BY toYYYYMM(created_at)`, `PARTITION BY
toStartOfWeek(created_at)`, `PARTITION BY toStartOfMonth(created_at)`,
`PARTITION BY toStartOfQuarter(created_at)`, and `PARTITION BY
toStartOfYear(created_at)` are supported, but `PARTITION BY (created_at + 1)`
is not supported.
In all, the following partitioning expressions are supported:
- Identity: `PARTITION BY column_name`
- - Functions: `PARTITION BY toDate(column_name)`, `PARTITION BY
toYear(column_name)`, `PARTITION BY toYYYYMM(column_name)`, `PARTITION BY
toStartOfWeek(column_name)`, and `PARTITION BY toStartOfMonth(column_name)`.
Other function expressions are not supported as structured transforms.
+ - Functions: `PARTITION BY toDate(column_name)`, `PARTITION BY
toYear(column_name)`, `PARTITION BY toYYYYMM(column_name)`, `PARTITION BY
toStartOfWeek(column_name)`, `PARTITION BY toStartOfMonth(column_name)`,
`PARTITION BY toStartOfQuarter(column_name)`, and `PARTITION BY
toStartOfYear(column_name)`. Other function expressions are not supported as
structured transforms.
- `toStartOfWeek(column_name)` uses ClickHouse's default mode `0` (Sunday
start) and the server timezone. Calls with an explicit mode or timezone are not
structured.
- Not support: `PARTITION BY (column_name + 1)`, `PARTITION BY
(toYear(column_name) + 1)`, etc. (Note: ClickHouse itself does support
arbitrary partitioning expressions, but Gravitino supports only the above
patterns for partitioning).
