This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 99e5ca937a [doc](multi-catalog)hms event sync (#15809)
99e5ca937a is described below
commit 99e5ca937a581bba3bb92e102be9a853c3ff20b7
Author: zhangdong <[email protected]>
AuthorDate: Thu Jan 12 09:23:05 2023 +0800
[doc](multi-catalog)hms event sync (#15809)
* hms event doc
* hms event doc add version
---
.../docs/ecosystem/external-table/multi-catalog.md | 55 ++++++++++++++++++++-
.../docs/ecosystem/external-table/multi-catalog.md | 56 +++++++++++++++++++++-
2 files changed, 107 insertions(+), 4 deletions(-)
diff --git a/docs/en/docs/ecosystem/external-table/multi-catalog.md
b/docs/en/docs/ecosystem/external-table/multi-catalog.md
index c19db7d417..4758e78466 100644
--- a/docs/en/docs/ecosystem/external-table/multi-catalog.md
+++ b/docs/en/docs/ecosystem/external-table/multi-catalog.md
@@ -732,11 +732,62 @@ The privilege management of Doris provides an extension
to the Cataloig level. F
## Metadata Refresh
+### Manual Refresh
+
Metadata changes of external data sources, such as creating, dropping tables,
adding or dropping columns, etc., will not be synchronized to Doris.
-Currently, users need to manually refresh metadata via the [REFRESH
CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH.md) command.
+Users need to manually refresh metadata via the [REFRESH
CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH.md) command.
+
+### Auto Refresh
+
+#### Hive MetaStore(HMS) catalog
+
+<version since="dev">
+
+The FE node can sense the change of the Hive table metadata by reading the HMS
notification event regularly. Currently, the following events are supported:
+
+</version>
+
+1. CREATE DATABASE event:Create database under the corresponding catalog.
+2. DROP DATABASE event:Drop database under the corresponding catalog.
+3. ALTER DATABASE event:The main impact of this event is to change the
attribute information, comments and default storage location of the
database,These changes do not affect the query operation of doris on the
external catalog, so this event will be ignored at present.
+4. CREATE TABLE event:Create table under the corresponding database.
+5. DROP TABLE event:Drop table under the corresponding database and invalidate
its cache.
+6. ALTER TABLE event:If renaming, delete the table with the old name first,
and then create the table with the new name, otherwise the cache of the table
will be invalidated.
+7. ADD PARTITION event:Add partition in the partitions list of the
corresponding table cache.
+8. DROP PARTITION event:Delete the partition in the partition list of the
corresponding table cache and invalidate the cache of the partition.
+9. ALTER PARTITION event:If renaming, delete the partition with the old name
first, and then create the partition with the new name, otherwise the cache of
the partition will be invalidated.
+10. When the file is changed due to importing data, the partition table will
follow the ALTER PARTITION event logic, and the non-partition table will follow
the ALTER TABLE event logic (note: if the file system is directly operated by
bypassing the HMS, the HMS will not generate the corresponding event, and Doris
will not be aware of it).
+
+The feature is controlled by the following parameters of fe:
+
+1. enable_hms_events_incremental_sync:Whether to enable automatic incremental
metadata synchronization. The default value false.
+2. hms_events_polling_interval_ms: The interval between reading events. The
default value is 10000, in milliseconds.
+3. hms_events_batch_size_per_rpc:The maximum number of events read each time.
The default value is 500.
+
+If you want to use this feature, you need to change the hive-site.xml of HMS
and restart HMS
+```
+<property>
+ <name>hive.metastore.event.db.notification.api.auth</name>
+ <value>false</value>
+</property>
+<property>
+ <name>hive.metastore.dml.events</name>
+ <value>true</value>
+</property>
+<property>
+ <name>hive.metastore.transactional.event.listeners</name>
+ <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
+</property>
+
+```
+##### Use suggestions
+
+Whether you want to change the previously created catalog to automatic refresh
or the newly created catalog, you only need to set
enable_hms_events_incremental_sync to true, restart the fe node, and do not
need to manually refresh the metadata before or after restarting.
+
+#### Other catalog
-Automatic synchronization of metadata will be supported soon.
+Not supported temporarily.
## FAQ
diff --git a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
index a216008867..7999825a55 100644
--- a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
+++ b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
@@ -731,11 +731,63 @@ Doris 的权限管理功能提供了对 Cataloig 层级的扩展，具体可参
## 元数据更新
+### 手动刷新
+
外部数据源的元数据变动，如创建、删除表，加减列等操作，不会同步给 Doris。
-目前需要用户通过 [REFRESH
CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH.md)
命令手动刷新元数据。
+用户通过 [REFRESH
CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH.md)
命令手动刷新元数据。
+
+### 自动刷新
+
+#### Hive MetaStore(HMS)数据目录
+
+<version since="dev">
+
+通过让FE节点定时读取HMS的notification event来感知Hive表元数据的变更情况，目前支持处理如下event：
+
+</version>
+
+1. CREATE DATABASE event:在对应数据目录下创建数据库。
+2. DROP DATABASE event:在对应数据目录下删除数据库。
+3. ALTER DATABASE
event:此事件的影响主要有更改数据库的属性信息，注释及默认存储位置等，这些改变不影响doris对外部数据目录的查询操作，因此目前会忽略此event。
+4. CREATE TABLE event:在对应数据库下创建表。
+5. DROP TABLE event:在对应数据库下删除表，并失效表的缓存。
+6. ALTER TABLE event:如果是重命名，先删除旧名字的表，再用新名字创建表，否则失效该表的缓存。
+7. ADD PARTITION event:在对应表缓存的分区列表里添加分区。
+8. DROP PARTITION event:在对应表缓存的分区列表里删除分区，并失效该分区的缓存。
+9. ALTER PARTITION event:如果是重命名，先删除旧名字的分区，再用新名字创建分区，否则失效该分区的缓存。
+10. 当导入数据导致文件变更,分区表会走ALTER PARTITION event逻辑，不分区表会走ALTER TABLE
event逻辑(注意：如果绕过HMS直接操作文件系统的话，HMS不会生成对应事件，doris因此也无法感知)。
+
+该特性被fe的如下参数控制：
+
+1. enable_hms_events_incremental_sync:是否开启元数据自动增量同步功能,默认关闭。
+2. hms_events_polling_interval_ms: 读取 event 的间隔时间，默认值为 10000，单位：毫秒。
+3. hms_events_batch_size_per_rpc:每次读取 event 的最大数量，默认值为 500。
+
+如果想使用该特性，需要更改HMS的hive-site.xml并重启HMS
+```
+<property>
+ <name>hive.metastore.event.db.notification.api.auth</name>
+ <value>false</value>
+</property>
+<property>
+ <name>hive.metastore.dml.events</name>
+ <value>true</value>
+</property>
+<property>
+ <name>hive.metastore.transactional.event.listeners</name>
+ <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
+</property>
+
+```
+##### 使用建议
+
+无论是之前已经创建好的catalog现在想改为自动刷新，还是新创建的catalog，都只需要把enable_hms_events_incremental_sync设置为true，重启fe节点，
+无需重启之前或之后再手动刷新元数据。
+
+#### 其它数据目录
-后续会支持元数据的自动同步。
+暂未支持。
## 常见问题
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
