This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c1305e48b78 [enhance](auto-partition) Constrain dynamic & auto
partition use same interval unit if both enable (#30426)
c1305e48b78 is described below
commit c1305e48b786bfc05b36b6249499c37226211c22
Author: zclllyybb <[email protected]>
AuthorDate: Mon Jan 29 14:59:41 2024 +0800
[enhance](auto-partition) Constrain dynamic & auto partition use same
interval unit if both enable (#30426)
---
docs/en/docs/advanced/partition/auto-partition.md | 58 ++++++++++++++++++++++
.../docs/advanced/partition/auto-partition.md | 58 ++++++++++++++++++++++
.../org/apache/doris/analysis/PartitionDesc.java | 8 +++
.../apache/doris/datasource/InternalCatalog.java | 24 ++++++++-
.../test_auto_partition_behavior.groovy | 50 +++++++++++++++++++
5 files changed, 197 insertions(+), 1 deletion(-)
diff --git a/docs/en/docs/advanced/partition/auto-partition.md
b/docs/en/docs/advanced/partition/auto-partition.md
index d9ac42cd556..df28dea0104 100644
--- a/docs/en/docs/advanced/partition/auto-partition.md
+++ b/docs/en/docs/advanced/partition/auto-partition.md
@@ -192,6 +192,64 @@ mysql> show partitions from `DAILY_TRADE_VALUE`;
A partition created by the AUTO PARTITION function has the exact same
functional properties as a manually created partition.
+## Coupled with dynamic partitioning
+
+AUTO PARTITION is supported on the same table as [DYNAMIC
PARTITION](./dynamic-partition). for example:
+
+```sql
+CREATE TABLE tbl3
+(
+ k1 DATETIME NOT NULL,
+ col1 int
+)
+AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+DISTRIBUTED BY HASH(k1)
+PROPERTIES
+(
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "year",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+);
+```
+
+When the two functions are used in combination, neither of their original
functions is affected and they still act on the entire table. The behavior
includes but is not limited to:
+
+1. regardless of the creation method, the expired historical partitions will
be periodically cleaned up or transferred to cold storage according to the
rules specified by the DYNAMIC PARTITION properties
+2. partition ranges cannot overlap or conflict. If a new partition range that
needs to be created by DYNAMIC PARTITION has already been covered by an
automatically or manually created partition, the partition creation will fail
without affecting the business process.
+
+The principle is that AUTO PARTITION is only a complementary means introduced
to the creation of partitions, and that a partition, whether created manually,
by AUTO PARTITION, or by DYNAMIC PARTITION, will be governed by DYNAMIC
PARTITION functions.
+
+### Constraint
+
+In order to simplify the behavioral pattern of combine two partition methods,
when the AUTO PARTITION and DYNAMIC PARTITION are both used, the **partition
intervals of the two must be consistent** or the table creating will fail:
+
+```sql
+mysql > CREATE TABLE tbl3
+ (
+ k1 DATETIME NOT NULL,
+ col1 int
+ )
+ AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+ DISTRIBUTED BY HASH(k1)
+ PROPERTIES
+ (
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "HOUR",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+ );
+ERROR 1105 (HY000): errCode = 2, detailMessage = errCode = 2, detailMessage =
If support auto partition and dynamic partition at same time, they must have
the same interval unit.
+```
+
## caveat
- If a partition is created during the insertion or import of data and the
entire import process does not complete (fails or is cancelled), the created
partition is not automatically deleted.
diff --git a/docs/zh-CN/docs/advanced/partition/auto-partition.md
b/docs/zh-CN/docs/advanced/partition/auto-partition.md
index c58a1fd3606..d5b8343d694 100644
--- a/docs/zh-CN/docs/advanced/partition/auto-partition.md
+++ b/docs/zh-CN/docs/advanced/partition/auto-partition.md
@@ -192,6 +192,64 @@ mysql> show partitions from `DAILY_TRADE_VALUE`;
经过自动分区功能所创建的PARTITION，与手动创建的PARTITION具有完全一致的功能性质。
+## 与动态分区联用
+
+自动分区支持与[动态分区](./dynamic-partition)同时作用于同一张表上，例如：
+
+```sql
+CREATE TABLE tbl3
+(
+ k1 DATETIME NOT NULL,
+ col1 int
+)
+AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+DISTRIBUTED BY HASH(k1)
+PROPERTIES
+(
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "year",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+);
+```
+
+当两种功能联用时，它们的原始功能均不受影响，依旧作用于整张表上，行为包括但不限于：
+
+1. 无论创建方式如何，过期的历史分区都会按动态分区功能指定的规则定期清理或转入冷存储
+2. 分区范围不能重叠、冲突。如果动态分区需要创建的新分区范围已经被自动或手动创建的分区覆盖，则该分区创建会失败，但不影响业务过程。
+
+其原则在于，自动分区仅是对创建分区引入的一种补充手段，一个分区无论是手动、经自动分区创建，还是经动态分区创建的，均会受到动态分区的管理。
+
+### 限制
+
+为简化两种分区方式联用的行为模式，当前自动分区和动态分区联用时，两者的**分区间隔必须一致**，否则建表将会失败：
+
+```sql
+mysql > CREATE TABLE tbl3
+ (
+ k1 DATETIME NOT NULL,
+ col1 int
+ )
+ AUTO PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+ DISTRIBUTED BY HASH(k1)
+ PROPERTIES
+ (
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "HOUR",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+ );
+ERROR 1105 (HY000): errCode = 2, detailMessage = errCode = 2, detailMessage =
If support auto partition and dynamic partition at same time, they must have
the same interval unit.
+```
+
## 注意事项
- 在数据的插入或导入过程中如果创建了分区，而整个导入过程没有完成（失败或被取消），被创建的分区不会被自动删除。
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java
index 9bc2920281d..625e0de8228 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java
@@ -248,6 +248,14 @@ public class PartitionDesc {
return type;
}
+ public ArrayList<Expr> getPartitionExprs() {
+ return partitionExprs;
+ }
+
+ public boolean isAutoCreatePartitions() {
+ return isAutoCreatePartitions;
+ }
+
public String toSql() {
throw new NotImplementedException("toSql not implemented");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index 7650a7a40f5..7f1dd7a64e7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -41,6 +41,7 @@ import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.FunctionCallExpr;
import org.apache.doris.analysis.HashDistributionDesc;
import org.apache.doris.analysis.KeysDesc;
+import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.PartitionDesc;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.QueryStmt;
@@ -64,6 +65,7 @@ import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.DatabaseProperty;
import org.apache.doris.catalog.DistributionInfo;
import org.apache.doris.catalog.DistributionInfo.DistributionInfoType;
+import org.apache.doris.catalog.DynamicPartitionProperty;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.EnvFactory;
import org.apache.doris.catalog.EsTable;
@@ -2606,7 +2608,27 @@ public class InternalCatalog implements
CatalogIf<Database> {
"Only support dynamic partition properties
on range partition table");
}
}
-
+ // check the interval same between dynamic & auto range
partition
+ DynamicPartitionProperty dynamicProperty =
olapTable.getTableProperty()
+ .getDynamicPartitionProperty();
+ if (dynamicProperty.isExist() &&
dynamicProperty.getEnable()
+ && partitionDesc.isAutoCreatePartitions()) {
+ String dynamicUnit = dynamicProperty.getTimeUnit();
+ ArrayList<Expr> autoExprs =
partitionDesc.getPartitionExprs();
+ for (Expr autoExpr : autoExprs) {
+ Expr func = (FunctionCallExpr) autoExpr;
+ for (Expr child : func.getChildren()) {
+ if (child instanceof LiteralExpr) {
+ String autoUnit = ((LiteralExpr)
child).getStringValue();
+ if
(!dynamicUnit.equalsIgnoreCase(autoUnit)) {
+ throw new AnalysisException(
+ "If support auto partition and
dynamic partition at same time, "
+ + "they must have the
same interval unit.");
+ }
+ }
+ }
+ }
+ }
} catch (AnalysisException e) {
throw new DdlException(e.getMessage());
}
diff --git
a/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy
b/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy
index 1a583560a70..af81060bd21 100644
---
a/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy
+++
b/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy
@@ -254,4 +254,54 @@ suite("test_auto_partition_behavior") {
"""
exception "The auto partition column must be NOT NULL"
}
+
+ // PROHIBIT different timeunit of interval when use both auto & dynamic
partition
+ test{
+ sql "set experimental_enable_nereids_planner=true;"
+ sql """
+ CREATE TABLE tbl3
+ (
+ k1 DATETIME NOT NULL,
+ col1 int
+ )
+ auto PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+ DISTRIBUTED BY HASH(k1)
+ PROPERTIES
+ (
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "HOUR",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+ );
+ """
+ exception "If support auto partition and dynamic partition at same
time, they must have the same interval unit."
+ }
+ test{
+ sql "set experimental_enable_nereids_planner=false;"
+ sql """
+ CREATE TABLE tbl3
+ (
+ k1 DATETIME NOT NULL,
+ col1 int
+ )
+ auto PARTITION BY RANGE date_trunc(`k1`, 'year') ()
+ DISTRIBUTED BY HASH(k1)
+ PROPERTIES
+ (
+ "replication_num" = "1",
+ "dynamic_partition.create_history_partition"="true",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "HOUR",
+ "dynamic_partition.start" = "-2",
+ "dynamic_partition.end" = "2",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "8"
+ );
+ """
+ exception "If support auto partition and dynamic partition at same
time, they must have the same interval unit."
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
