This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
The following commit(s) were added to refs/heads/master by this push:
new 8fd28f178 [FLINK-34680][cdc][docs] Add overview page for CDC
connectors and optimize connector docs
8fd28f178 is described below
commit 8fd28f1788f35ef6a8a2560811520170ff70a10f
Author: Qingsheng Ren <[email protected]>
AuthorDate: Mon Mar 18 19:06:08 2024 +0800
[FLINK-34680][cdc][docs] Add overview page for CDC connectors and optimize
connector docs
This closes #3160.
---
docs/content/docs/connectors/doris.md | 19 +++++-------
.../legacy-flink-cdc-sources/overview.md | 8 +++--
docs/content/docs/connectors/mysql.md | 26 +++++++----------
docs/content/docs/connectors/overview.md | 34 +++++++++++++++-------
docs/content/docs/connectors/starrocks.md | 22 +++++++-------
5 files changed, 57 insertions(+), 52 deletions(-)
diff --git a/docs/content/docs/connectors/doris.md
b/docs/content/docs/connectors/doris.md
index 101cae7d6..61b88d0df 100644
--- a/docs/content/docs/connectors/doris.md
+++ b/docs/content/docs/connectors/doris.md
@@ -24,13 +24,12 @@ specific language governing permissions and limitations
under the License.
-->
-# Doris Pipeline Connector
-
-This article introduces of Doris Pipeline Connector
+# Doris Connector
+This article introduces of Doris Connector
## Example
-----------------
+
```yaml
source:
@@ -50,8 +49,7 @@ pipeline:
```
-## Pipeline options
-----------------
+## Connector Options
<div class="highlight">
<table class="colwidths-auto docutils">
@@ -180,17 +178,16 @@ pipeline:
</tbody>
</table>
</div>
-## Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left" style="width:10%;">CDC type</th>
- <th class="text-left" style="width:30%;">Doris type<a
href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/";></a></th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:10%;">Flink CDC Type</th>
+ <th class="text-left" style="width:30%;"><a
href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/";>Doris
Type</a></th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>
diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
b/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
index babdb7931..f09ca2879 100644
--- a/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
+++ b/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
@@ -24,10 +24,12 @@ specific language governing permissions and limitations
under the License.
-->
-# CDC Connectors for Apache Flink
+# Legacy CDC sources for Apache Flink
-CDC Connectors for Apache Flink<sup>®</sup> is a set of source connectors for
<a href="https://flink.apache.org/";>Apache Flink<sup>®</sup></a>, ingesting
changes from different databases using change data capture (CDC).
-The CDC Connectors for Apache Flink<sup>®</sup> integrate Debezium as the
engine to capture data changes. So it can fully leverage the ability of
Debezium. See more about what is
[Debezium](https://github.com/debezium/debezium).
+Flink CDC sources is a set of source connectors for <a
href="https://flink.apache.org/";>Apache Flink<sup>®</sup></a>, ingesting
changes from different databases using change data capture (CDC).
+Some CDC sources integrate Debezium as the engine to capture data changes. So
it can fully leverage the ability of Debezium. See more about what is
[Debezium](https://github.com/debezium/debezium).
+
+You can also read [tutorials]({{< ref
"docs/connectors/legacy-flink-cdc-sources/tutorials/mysql-postgres-tutorial"
>}}) about how to use these sources.
{{< img src="/fig/cdc-flow.png" width="600px" alt="Flink CDC" >}}
diff --git a/docs/content/docs/connectors/mysql.md
b/docs/content/docs/connectors/mysql.md
index 3f9d27c64..455e22edf 100644
--- a/docs/content/docs/connectors/mysql.md
+++ b/docs/content/docs/connectors/mysql.md
@@ -24,16 +24,15 @@ specific language governing permissions and limitations
under the License.
-->
-# MySQL CDC Pipeline Connector
+# MySQL Connector
-The MySQL CDC Pipeline Connector allows for reading snapshot data and
incremental data from MySQL database and provides end-to-end full-database data
synchronization capabilities.
-This document describes how to setup the MySQL CDC Pipeline connector.
+MySQL connector allows reading snapshot data and incremental data from MySQL
database and provides end-to-end full-database data synchronization
capabilities.
+This document describes how to setup the MySQL connector.
-How to create Pipeline
-----------------
+## Example
-The pipeline for reading data from MySQL and sink to Doris can be defined as
follows:
+An example of the pipeline for reading data from MySQL and sink to Doris can
be defined as follows:
```yaml
source:
@@ -58,8 +57,7 @@ pipeline:
parallelism: 4
```
-Pipeline Connector Options
-----------------
+## Connector Options
<div class="highlight">
<table class="colwidths-auto docutils">
@@ -245,8 +243,7 @@ Pipeline Connector Options
</table>
</div>
-Startup Reading Position
---------
+## Startup Reading Position
The config option `scan.startup.mode` specifies the startup mode for MySQL CDC
consumer. The valid enumerations are:
@@ -259,16 +256,15 @@ The config option `scan.startup.mode` specifies the
startup mode for MySQL CDC c
- `timestamp`: Skip snapshot phase and start reading binlog events from a
specific timestamp.
-Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left" style="width:30%;">MySQL type<a
href="https://dev.mysql.com/doc/man/8.0/en/data-types.html";></a></th>
- <th class="text-left" style="width:10%;">CDC type</th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:30%;"><a
href="https://dev.mysql.com/doc/man/8.0/en/data-types.html";>MySQL type</a></th>
+ <th class="text-left" style="width:10%;">Flink CDC type</th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>
diff --git a/docs/content/docs/connectors/overview.md
b/docs/content/docs/connectors/overview.md
index 00b2bf2a3..0eee7bdfa 100644
--- a/docs/content/docs/connectors/overview.md
+++ b/docs/content/docs/connectors/overview.md
@@ -24,21 +24,33 @@ specific language governing permissions and limitations
under the License.
-->
-# Pipeline Connectors Of CDC Streaming ELT Framework
+# Connectors
+
+Flink CDC provides several source and sink connectors to interact with external
+systems. You can use these connectors out-of-box, by adding released JARs to
+your Flink CDC environment, and specifying the connector in your YAML pipeline
+definition.
## Supported Connectors
-| Connector | Database
|
-|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [doris-pipeline](doris-pipeline.md) | <li>
[Doris](https://doris.apache.org/): 1.2.x, 2.x.x
|
-| [mysql-pipeline](mysql-pipeline.md) | <li>
[MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS
MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB
MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora
MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li>
[MariaDB](https://mariadb.org): 10.x <li> [PolarDB
X](https://github.com/ApsaraDB/galaxysql): 2.0.1 |
-| [starrocks-pipeline](starrocks-pipeline.md) | <li>
[StarRocks](https://www.starrocks.io/): 2.x, 3.x
|
+| Connector | Supported Type |
External System
|
+|------------------------------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Apache Doris]({{< ref "docs/connectors/doris" >}}) | Sink | <li>
[Apache Doris](https://doris.apache.org/): 1.2.x, 2.x.x
|
+| [MySQL]({{< ref "docs/connectors/mysql" >}}) | Source | <li>
[MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS
MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB
MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora
MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li>
[MariaDB](https://mariadb.org): 10.x <li> [PolarDB
X](https://github.com/ApsaraDB/galaxysql): 2.0.1 |
+| [StarRocks]({{< ref "docs/connectors/starrocks" >}}) | Sink | <li>
[StarRocks](https://www.starrocks.io/): 2.x, 3.x
|
+
+## Develop Your Own Connector
+
+If provided connectors cannot fulfill your requirement, you can always develop
+your own connector to get your external system involved in Flink CDC pipelines.
+Check out [Flink CDC APIs]({{< ref
"docs/developer-guide/understand-flink-cdc-api" >}})
+to learn how to develop your own connectors.
-## Supported Flink Versions
-The following table shows the version mapping between Flink<sup>®</sup> CDC
Pipeline and Flink<sup>®</sup>:
+## Legacy Flink CDC Sources
-| Flink<sup>®</sup> CDC Version |
Flink<sup>®</sup>
Version
|
-|:-----------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
-| <font color="DarkCyan">3.0.*</font> | <font
color="MediumVioletRed">1.14.\*</font>, <font
color="MediumVioletRed">1.15.\*</font>, <font
color="MediumVioletRed">1.16.\*</font>, <font
color="MediumVioletRed">1.17.\*</font>, <font
color="MediumVioletRed">1.18.\*</font> |
+Flink CDC sources introduces before 3.0 are still available as normal Flink
+connector sources. You can find more details in the
+[overview page]({{< ref "docs/connectors/legacy-flink-cdc-sources/overview"
>}})
+of legacy Flink CDC sources.
{{< top >}}
diff --git a/docs/content/docs/connectors/starrocks.md
b/docs/content/docs/connectors/starrocks.md
index 64dff6689..84a4a99d1 100644
--- a/docs/content/docs/connectors/starrocks.md
+++ b/docs/content/docs/connectors/starrocks.md
@@ -24,17 +24,16 @@ specific language governing permissions and limitations
under the License.
-->
-# StarRocks Pipeline Connector
+# StarRocks Connector
-The StarRocks Pipeline connector can be used as the *Data Sink* of the
pipeline, and write data to
[StarRocks](https://github.com/StarRocks/starrocks). This document describes
how to set up the StarRocks Pipeline connector.
+StarRocks connector can be used as the *Data Sink* of the pipeline, and write
data to [StarRocks](https://github.com/StarRocks/starrocks). This document
describes how to set up the StarRocks connector.
## What can the connector do?
* Create table automatically if not exist
* Schema change synchronization
* Data synchronization
-How to create Pipeline
-----------------
+## Example
The pipeline for reading data from MySQL and sink to StarRocks can be defined
as follows:
@@ -62,8 +61,8 @@ pipeline:
parallelism: 2
```
-Pipeline Connector Options
-----------------
+## Connector Options
+
<div class="highlight">
<table class="colwidths-auto docutils">
<thead>
@@ -220,8 +219,8 @@ Pipeline Connector Options
</table>
</div>
-Usage Notes
---------
+
+## Usage Notes
* Only support StarRocks primary key table, so the source table must have
primary keys.
@@ -244,15 +243,14 @@ Usage Notes
to write data to StarRocks. You can see [sink
documentation](https://github.com/StarRocks/starrocks-connector-for-apache-flink/blob/main/docs/content/connector-sink.md)
for how it works.
-Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left">CDC type</th>
+ <th class="text-left">Flink CDC type</th>
<th class="text-left">StarRocks type</th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>
