pengxiangyu commented on a change in pull request #8532:
URL: https://github.com/apache/incubator-doris/pull/8532#discussion_r830728323
##########
File path: docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
##########
@@ -284,6 +284,8 @@ Syntax:
```
PROPERTIES (
"storage_medium" = "[SSD|HDD]",
+ ["storage_cold_medium" = "[S3]"],
Review comment:
默认值是HDD，线上有一个热数据转冷功能是SSD -> HDD。
##########
File path: docs/en/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
##########
@@ -292,6 +294,10 @@ Syntax:
storage_medium: SSD or HDD, The default initial storage media
can be specified by `default_storage_medium= XXX` in the fe configuration file
`fe.conf`, or, if not, by default, HDD.
Note: when FE configuration
'enable_strict_storage_medium_check' is' True ', if the corresponding storage
medium is not set in the cluster, the construction clause 'Failed to find
enough host in all backends with storage medium is SSD|HDD'.
+ storage_cold_medium: Used to specify the cold data storage medium
for this partition, currently only S3 is
Review comment:
默认值是HDD，线上有一个热数据转冷功能是SSD -> HDD。
##########
File path: docs/zh-CN/sql-reference/sql-statements/Data Definition/CREATE
TABLE.md
##########
@@ -301,14 +301,18 @@ under the License.
```
PROPERTIES (
"storage_medium" = "[SSD|HDD]",
+ ["storage_cold_medium" = "[S3]"],
+ ["remote_storage" = "xxx"],
["storage_cooldown_time" = "yyyy-MM-dd HH:mm:ss"],
["replication_num" = "3"]
["replication_allocation" = "xxx"]
)
```
storage_medium： 用于指定该分区的初始存储介质，可选择 SSD 或 HDD。默认初始存储介质可通过fe的配置文件
`fe.conf` 中指定 `default_storage_medium=xxx`，如果没有指定，则默认为 HDD。
- 注意：当FE配置项 `enable_strict_storage_medium_check`
为 `True` 时，若集群中没有设置对应的存储介质时，建表语句会报错 `Failed to find enough host in all backends
with storage medium is SSD|HDD`.
+ 注意：当FE配置项 `enable_strict_storage_medium_check`
为 `True` 时，若集群中没有设置对应的存储介质时，建表语句会报错 `Failed to find enough host in all backends
with storage medium is SSD|HDD`.
+ storage_cold_medium: 用于指定该分区的冷数据存储介质，当前只支持 S3。默认为 S3。
Review comment:
还支持HDD，现存的热转冷是SSD->HDD。其他与此相关的地方也改一下。
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
##########
@@ -1257,7 +1257,7 @@ public OlapTable selectiveCopy(Collection<String>
reservedPartitions, IndexExtSt
// set storage medium to HDD for backup job, because we want
that the backuped table
// can be able to restored to another Doris cluster without
SSD disk.
// But for other operation such as truncate table, keep the
origin storage medium.
- copied.getPartitionInfo().setDataProperty(partition.getId(),
new DataProperty(TStorageMedium.HDD));
+ copied.getPartitionInfo().setDataProperty(partition.getId(),
new DataProperty(TStorageMedium.HDD, TStorageMedium.S3));
Review comment:
第一个参数是当前storage，第二个参数是目标storage，这代表当前partition是需要冷热数据转换的，默认不配是不需要的配成不同storage的。
##########
File path:
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
##########
@@ -130,21 +139,48 @@ public static DataProperty
analyzeDataProperty(Map<String, String> properties, D
hasCooldown = true;
DateLiteral dateLiteral = new DateLiteral(value,
Type.DATETIME);
coolDownTimeStamp =
dateLiteral.unixTimestamp(TimeUtils.getTimeZone());
+ } else if (!hasColdMedium &&
key.equalsIgnoreCase(PROPERTIES_STORAGE_COLD_MEDIUM)) {
+ hasColdMedium = true;
+ if (value.equalsIgnoreCase(TStorageMedium.S3.name())) {
+ coldStorageMedium = TStorageMedium.S3;
+ } else {
+ throw new AnalysisException("Invalid cold storage medium:
" + value);
+ }
+ } else if (!hasRemoteStorage &&
key.equalsIgnoreCase(PROPERTIES_REMOTE_STORAGE)) {
+ hasRemoteStorage = true;
+ remoteStorageName = value;
}
} // end for properties
- if (!hasCooldown && !hasMedium) {
+ if (!hasCooldown && !hasMedium && !hasRemoteStorage) {
return oldDataProperty;
}
properties.remove(PROPERTIES_STORAGE_MEDIUM);
properties.remove(PROPERTIES_STORAGE_COLDOWN_TIME);
+ properties.remove(PROPERTIES_STORAGE_COLD_MEDIUM);
+ properties.remove(PROPERTIES_REMOTE_STORAGE);
+
+ if ((hasColdMedium && !hasRemoteStorage) || (!hasColdMedium &&
hasRemoteStorage)) {
+ throw new AnalysisException("Invalid data property, " +
Review comment:
SSD->HDD的情况下不需要remote storage
##########
File path: docs/zh-CN/sql-reference/sql-statements/Administration/ALTER
SYSTEM.md
##########
@@ -49,6 +49,10 @@ under the License.
ALTER SYSTEM SET LOAD ERRORS HUB PROPERTIES ("key" = "value"[,
...]);
10) 修改一个 BE 节点的属性
ALTER SYSTEM MODIFY BACKEND "host:heartbeat_port" SET ("key" =
"value"[, ...]);
+ 11）增加一个远端存储
+ ALTER SYSTEM ADD REMOTE STORAGE storage_name PROPERTIES ("key" =
"value"[, ...]);
Review comment:
需要有一个修改远端存储的功能：MODIFY REMOTE
##########
File path: docs/zh-CN/sql-reference/sql-statements/Administration/SHOW REMOTE
STORAGES.md
##########
@@ -0,0 +1,43 @@
+---
+{
+ "title": "SHOW REMOTE STORAGES",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# SHOW REMOTE STORAGES
+
+## Description
+
+ 该语句用于查看当前存在的远端存储
+ 语法：
+ SHOW REMOTE STORAGES;
+
+ 说明：
+ 1. Name：远端存储的名字
+ 2. Type：远端存储的类型
+ 3. Properties：远端存储的参数
+
+## keyword
+
+ SHOW, REMOTE STORAGES
Review comment:
加个单独REMOTE吧：SHOW, REMOTE, REMOTE STORAGES
##########
File path:
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
##########
@@ -130,21 +139,48 @@ public static DataProperty
analyzeDataProperty(Map<String, String> properties, D
hasCooldown = true;
DateLiteral dateLiteral = new DateLiteral(value,
Type.DATETIME);
coolDownTimeStamp =
dateLiteral.unixTimestamp(TimeUtils.getTimeZone());
+ } else if (!hasColdMedium &&
key.equalsIgnoreCase(PROPERTIES_STORAGE_COLD_MEDIUM)) {
+ hasColdMedium = true;
+ if (value.equalsIgnoreCase(TStorageMedium.S3.name())) {
+ coldStorageMedium = TStorageMedium.S3;
+ } else {
+ throw new AnalysisException("Invalid cold storage medium:
" + value);
+ }
+ } else if (!hasRemoteStorage &&
key.equalsIgnoreCase(PROPERTIES_REMOTE_STORAGE)) {
+ hasRemoteStorage = true;
+ remoteStorageName = value;
}
} // end for properties
- if (!hasCooldown && !hasMedium) {
+ if (!hasCooldown && !hasMedium && !hasRemoteStorage) {
return oldDataProperty;
Review comment:
如果是SSD->HDD，remote storage是空的。
##########
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##########
@@ -4980,7 +4986,7 @@ public long getNextId() {
if (dataProperty.getStorageMedium() ==
TStorageMedium.SSD
&& dataProperty.getCooldownTimeMs() <
currentTimeMs) {
// expire. change to HDD.
- partitionInfo.setDataProperty(partition.getId(),
new DataProperty(TStorageMedium.HDD));
+ partitionInfo.setDataProperty(partition.getId(),
new DataProperty(TStorageMedium.HDD, TStorageMedium.S3));
Review comment:
这里逻辑有问题，这里的逻辑是把当前storageMedium设置成目标storage，上面条件里原值是SSD，这里设置成了原值HDD，目标S3。
##########
File path: docs/zh-CN/sql-reference/sql-statements/Administration/SHOW REMOTE
STORAGES.md
##########
@@ -0,0 +1,43 @@
+---
+{
+ "title": "SHOW REMOTE STORAGES",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# SHOW REMOTE STORAGES
+
+## Description
+
+ 该语句用于查看当前存在的远端存储
+ 语法：
+ SHOW REMOTE STORAGES;
+
+ 说明：
+ 1. Name：远端存储的名字
+ 2. Type：远端存储的类型
+ 3. Properties：远端存储的参数
+
Review comment:
加个Example，参数是什么格式的，JSON？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
