lsyldliu commented on code in PR #26073: URL: https://github.com/apache/flink/pull/26073#discussion_r1948323675
########## docs/content/docs/dev/table/materialized-table/deployment.md: ########## @@ -0,0 +1,201 @@ +--- +title: Deployment +weight: 3 +type: docs +aliases: +- /dev/table/materialized-table/deployment.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Introduction + +The creation and operation of materialized tables involves the collaborative work of multiple components. This article systematically explains the complete deployment solution for Materialized Tables, covering architectural overview, environment preparation, deployment procedures, and operational practices. + +# Architecture Introduction + +- **Client**: Could be any client that can interact with [Flink SQL Gateway]({{< ref "docs/dev/table/sql-gateway/overview" >}}), such as [SQL Client]({{< ref "docs/dev/table/sqlClient" >}}), [Flink JDBC Driver]({{< ref "docs/dev/table/jdbcDriver" >}}) and so on. +- **Flink SQL Gateway**: Supports creating, altering, and dropping Materialized table. It also serves as an embedded workflow scheduler to periodically refresh full mode Materialized Table. +- **Flink Cluster**: The pipeline for refreshing Materialized Table will run on the Flink cluster. +- **Catalog**: Manages the creation, retrieval, modification, and deletion of the metadata of Materialized Table. +- **Catalog Store**: Supports catalog property persistence to automatically initialize catalogs for retrieving metadata in Materialized Table related operations. + +{{< img src="/fig/materialized-table-architecture.svg" alt="Illustration of Flink Materialized Table Architecture" width="85%" >}} + +# Deployment Preparation + +## Flink Cluster Setup + +Materialized Table refresh jobs currently support execution in these cluster environments: +* [Standalone clusters]({{<ref "docs/deployment/resource-providers/standalone/overview">}}) +* [YARN clusters]({{<ref "docs/deployment/resource-providers/yarn" >}}) +* [Kubernetes clusters]({{<ref "docs/deployment/resource-providers/native_kubernetes" >}}) + +## Flink SQL Gateway Deployment + +Materialized Tables must be created through SQL Gateway, which requires specific configurations for metadata persistence and job scheduling. + +### Configure Catalog Store + +Add catalog store configurations in `config.yaml` to persist catalog properties. + +```yaml +table: + catalog-store: + kind: file + file: + path: {path_to_catalog_store} # Replace with the actual path +``` +Refer to [Catalog Store]({{<ref "docs/dev/table/catalogs">}}#catalog-store) for details. + +### Configure Workflow Scheduler Plugin + +Add scheduler configurations in `config.yaml` for periodic refresh job scheduling. Currently, only the `embedded` scheduler is supported: + +```yaml +workflow-scheduler: + type: embedded +``` + +### Start SQL Gateway + +Start the SQL Gateway using: + +```shell +./sql-gateway.sh start +``` + +<span class="label label-danger">Note</span> +The Catalog must support the creation of materialized tables. Currently, only [Paimon Catalog](https://paimon.apache.org/docs/master/concepts/table-types/#materialized-table) supports it. Review Comment: ```suggestion The Catalog must support creating materialized tables, which is currently only supported by [Paimon Catalog](https://paimon.apache.org/docs/master/concepts/table-types/#materialized-table). ``` ########## docs/content.zh/docs/dev/table/materialized-table/deployment.md: ########## @@ -64,9 +64,9 @@ table: ``` 更多详情配置可参考 [Catalog Store]({{<ref "docs/dev/table/catalogs">}}#catalog-store)。 -### 配置调度插件 +### 配置工作流调度器插件 -在 `config.yaml` 增加配置调度器配置，用于定时调度刷新任务。 当前我们仅支持 `embedded` 调度器 +在 `config.yaml` 增加配置调度器配置，用于定时调度刷新作业。 当前我们仅支持 `embedded` 调度器 Review Comment: 增加配置调度器配置 -> 增加工作流调度器配置 ########## docs/content/docs/dev/table/materialized-table/deployment.md: ########## @@ -0,0 +1,201 @@ +--- +title: Deployment +weight: 3 +type: docs +aliases: +- /dev/table/materialized-table/deployment.html +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Introduction + +The creation and operation of materialized tables involves the collaborative work of multiple components. This article systematically explains the complete deployment solution for Materialized Tables, covering architectural overview, environment preparation, deployment procedures, and operational practices. Review Comment: ```suggestion Creating and operating materialized tables involves multiple components' collaborative work. This document will systematically explain the complete deployment solution for Materialized Tables, covering architectural overview, environment preparation, deployment procedures, and operational practices. ``` ########## docs/content.zh/docs/dev/table/materialized-table/deployment.md: ########## @@ -64,9 +64,9 @@ table: ``` 更多详情配置可参考 [Catalog Store]({{<ref "docs/dev/table/catalogs">}}#catalog-store)。 -### 配置调度插件 +### 配置工作流调度器插件 -在 `config.yaml` 增加配置调度器配置，用于定时调度刷新任务。 当前我们仅支持 `embedded` 调度器 +在 `config.yaml` 增加配置调度器配置，用于定时调度刷新作业。 当前我们仅支持 `embedded` 调度器 Review Comment: 当前我们仅支持 `embedded` 调度器 -> 当前我们仅支持 `embedded` 调度器： -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
