Copilot commented on code in PR #4349:
URL: https://github.com/apache/flink-cdc/pull/4349#discussion_r2999047314
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img
src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly";
alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE";
target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache
License 2.0&color=white" alt="License">
+ <img
src="https://img.shields.io/github/license/apache/flink-cdc?color=white";
alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch
data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and
transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch
data, built on top of Apache Flink. It prioritizes efficient end-to-end data
integration and offers enhanced functionalities such as full database
synchronization, sharding table synchronization, schema evolution and data
transformation.
+
-The Flink CDC prioritizes efficient end-to-end data integration and offers
enhanced functionalities such as
-full database synchronization, sharding table synchronization, schema
evolution and data transformation.
+## API Layers
+
+Flink CDC provides three API layers for different usage scenarios:
+
+### 1. YAML API (Pipeline API)
+
+The YAML API provides a declarative, zero-code approach to define data
pipelines. Users describe the source, sink,
[routing](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/route/),
[transformation](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/transform/),
and [schema
evolution](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/schema-evolution/)
rules in a YAML file and submit it via the `flink-cdc.sh` CLI.
+
+Please refer to the [Quickstart
Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/)
for detailed setup instructions.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: root
+ password: 123456
+ tables: app_db\.\.*
Review Comment:
In the YAML example, `tables: app_db\.\.\.*` doesn’t match the table
selector format used in the project docs (e.g., `app_db.\.*` in
docs/core-concept/data-pipeline.md). As written, the escaped `\.` will be
treated differently and can prevent matching all tables under `app_db` as
intended; please align this example with the documented pattern syntax.
```suggestion
tables: app_db.\.*
```
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img
src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly";
alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE";
target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache
License 2.0&color=white" alt="License">
+ <img
src="https://img.shields.io/github/license/apache/flink-cdc?color=white";
alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch
data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and
transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch
data, built on top of Apache Flink. It prioritizes efficient end-to-end data
integration and offers enhanced functionalities such as full database
synchronization, sharding table synchronization, schema evolution and data
transformation.
Review Comment:
Minor wording: "real time" should be hyphenated as "real-time" when used as
a compound modifier (e.g., "real-time data").
```suggestion
Flink CDC is a distributed data integration tool for real-time data and
batch data, built on top of Apache Flink. It prioritizes efficient end-to-end
data integration and offers enhanced functionalities such as full database
synchronization, sharding table synchronization, schema evolution and data
transformation.
```
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img
src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly";
alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE";
target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache
License 2.0&color=white" alt="License">
+ <img
src="https://img.shields.io/github/license/apache/flink-cdc?color=white";
alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch
data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and
transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch
data, built on top of Apache Flink. It prioritizes efficient end-to-end data
integration and offers enhanced functionalities such as full database
synchronization, sharding table synchronization, schema evolution and data
transformation.
+
-The Flink CDC prioritizes efficient end-to-end data integration and offers
enhanced functionalities such as
-full database synchronization, sharding table synchronization, schema
evolution and data transformation.
+## API Layers
+
+Flink CDC provides three API layers for different usage scenarios:
+
+### 1. YAML API (Pipeline API)
+
+The YAML API provides a declarative, zero-code approach to define data
pipelines. Users describe the source, sink,
[routing](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/route/),
[transformation](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/transform/),
and [schema
evolution](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/schema-evolution/)
rules in a YAML file and submit it via the `flink-cdc.sh` CLI.
+
+Please refer to the [Quickstart
Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/)
for detailed setup instructions.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: root
+ password: 123456
+ tables: app_db\.\.*
+
+sink:
+ type: doris
+ fenodes: 127.0.0.1:8030
+ username: root
+ password: ""
+
+# Transform data on-the-fly
+transform:
+ - source-table: app_db.orders
+ projection: id, order_id, UPPER(product_name) as product_name
+ filter: id > 10 AND order_id > 100
+
+# Route source tables to different sink tables
+route:
+ - source-table: app_db.orders
+ sink-table: ods_db.ods_orders
+ - source-table: app_db.shipments
+ sink-table: ods_db.ods_shipments
+ - source-table: app_db.\.*
+ sink-table: ods_db.others
+
+pipeline:
+ name: Sync MySQL Database to Doris
+ parallelism: 2
+ schema.change.behavior: evolve # Support schema evolution
+```
-
+**Pipeline connectors:**
-### Quickstart Guide
+[Doris](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-doris)
|
[Elasticsearch](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-elasticsearch)
|
[Fluss](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-fluss)
|
[Hudi](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-hudi)
|
[Iceberg](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-iceberg)
|
[Kafka](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-kafka)
|
[MaxCompute](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-maxcompute)
|
[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-mysql)
|
[OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-oceanbase)
|
[Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline
-connector-oracle) |
[Paimon](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-paimon)
|
[PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-postgres)
|
[StarRocks](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-starrocks)
-Flink CDC provides a CdcUp CLI utility to start a playground environment and
run Flink CDC jobs.
-You will need to have a working Docker and Docker compose environment to use
it.
+See the [connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/overview/)
for a full list and configurations.
-1. Run `git clone https://github.com/apache/flink-cdc.git --depth=1` to
retrieve a copy of Flink CDC source code.
-2. Run `cd tools/cdcup/ && ./cdcup.sh init` to use the CdcUp tool to start a
playground environment.
-3. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be
ready.
-4. Run `./cdcup.sh mysql` to open a MySQL session, and create at least one
table.
+### 2. SQL API (Table/SQL API)
+
+The SQL API integrates with Flink SQL, allowing users to define CDC sources
using SQL DDL statements. Deploy the SQL connector JAR to `FLINK_HOME/lib/` and
use it directly in Flink SQL Client:
```sql
--- initialize db and table
-CREATE DATABASE cdc_playground;
-USE cdc_playground;
-CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+CREATE TABLE mysql_binlog (
+ id INT NOT NULL,
+ name STRING,
+ description STRING,
+ weight DECIMAL(10,3),
+ PRIMARY KEY(id) NOT ENFORCED
+) WITH (
+ 'connector' = 'mysql-cdc',
+ 'hostname' = 'localhost',
+ 'port' = '3306',
+ 'username' = 'flinkuser',
+ 'password' = 'flinkpw',
+ 'database-name' = 'inventory',
+ 'table-name' = 'products'
+);
+
+SELECT id, UPPER(name), description, weight FROM mysql_binlog;
+```
--- insert test data
-INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4,
'derrida');
+**Available SQL connectors** (dependencies bundled):
--- verify if it has been successfully inserted
-SELECT * FROM test_table;
-```
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mysql-cdc)
|
[PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-postgres-cdc)
|
[Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-oracle-cdc)
| [SQL
Server](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-sqlserver-cdc)
|
[MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mongodb-cdc)
|
[OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-oceanbase-cdc)
|
[TiDB](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-tidb-cdc)
|
[Db2](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-db2-cdc)
|
[Vitess](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-vitess-cdc)
-5. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline
job. You may also edit the pipeline definition file for further configurations.
-6. Run `./cdcup.sh flink` to access the Flink Web UI.
-
-### Getting Started
-
-1. Prepare a [Apache
Flink](https://nightlies.apache.org/flink/flink-docs-master/docs/try-flink/local_installation/#starting-and-stopping-a-local-cluster)
cluster and set up `FLINK_HOME` environment variable.
-2. [Download](https://github.com/apache/flink-cdc/releases) Flink CDC tar,
unzip it and put jars of pipeline connector to Flink `lib` directory.
-
-> If you're using macOS or Linux, you may use `brew install apache-flink-cdc`
to install Flink CDC and compatible connectors quickly.
-
-3. Create a **YAML** file to describe the data source and data sink, the
following example synchronizes all tables under MySQL app_db database to Doris :
- ```yaml
- source:
- type: mysql
- hostname: localhost
- port: 3306
- username: root
- password: 123456
- tables: app_db.\.*
-
- sink:
- type: doris
- fenodes: 127.0.0.1:8030
- username: root
- password: ""
-
- transform:
- - source-table: adb.web_order01
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 10 AND order_id > 100
- description: project fields and filter
- - source-table: adb.web_order02
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 20 AND order_id > 200
- description: project fields and filter
-
- route:
- - source-table: app_db.orders
- sink-table: ods_db.ods_orders
- - source-table: app_db.shipments
- sink-table: ods_db.ods_shipments
- - source-table: app_db.products
- sink-table: ods_db.ods_products
-
- pipeline:
- name: Sync MySQL Database to Doris
- parallelism: 2
- route-mode: ALL_MATCH
- user-defined-function:
- - name: addone
- classpath: com.example.functions.AddOneFunctionClass
- - name: format
- classpath: com.example.functions.FormatFunctionClass
- ```
-4. Submit pipeline job using `flink-cdc.sh` script.
- ```shell
- bash bin/flink-cdc.sh /path/mysql-to-doris.yaml
- ```
-5. View job execution status through Flink WebUI or downstream database.
-
-Try it out yourself with our more detailed
[tutorial](docs/content/docs/get-started/quickstart/mysql-to-doris.md).
-You can also see [connector
overview](docs/content/docs/connectors/pipeline-connectors/overview.md) to view
a comprehensive catalog of the
-connectors currently provided and understand more detailed configurations.
-
-### Join the Community
+See the [source connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for a full list and configurations.
-There are many ways to participate in the Apache Flink CDC community. The
-[mailing
lists](https://flink.apache.org/what-is-flink/community/#mailing-lists) are the
primary place where all Flink
-committers are present. For user support and questions use the user mailing
list. If you've found a problem of Flink CDC,
-please create a [Flink
jira](https://issues.apache.org/jira/projects/FLINK/summary) and tag it with
the `Flink CDC` tag.
-Bugs and feature requests can either be discussed on the dev mailing list or
on Jira.
+### 3. DataStream API
+The DataStream API provides programmatic access for building custom Flink
streaming applications. Add the corresponding connector as a Maven dependency:
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-mysql-cdc</artifactId>
+ <version>${flink-cdc.version}</version>
+</dependency>
+```
-### Contributing
+**Available source connectors:**
-Welcome to contribute to Flink CDC, please see our [Developer
Guide](docs/content/docs/developer-guide/contribute-to-flink-cdc.md)
-and [APIs
Guide](docs/content/docs/developer-guide/understand-flink-cdc-api.md).
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mysql-cdc)
|
[PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-postgres-cdc)
|
[Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-oracle-cdc)
| [SQL
Server](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-sqlserver-cdc)
|
[MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mongodb-cdc)
|
[OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-oceanbase-cdc)
|
[TiDB](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-tidb-cdc)
|
[Db2](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-db2-cdc)
|
[Vitess](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-vitess-cdc)
+All artifacts use group ID `org.apache.flink`. See the [DataStream API
packaging
guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/datastream-api-package-guidance/)
for a complete `pom.xml` example.
+## Flink Version Compatibility
-### License
+| Flink CDC Version | Flink Version | Notes
|
+|-------------------|---------------------------|-----------------------------------|
+| 3.6.x | 1.20.\*, 2.2.\* |
|
+| 3.5.x | 1.19.\*, 1.20.\* |
|
+| 3.4.x | 1.19.\*, 1.20.\* |
|
+| 3.3.x | 1.19.\*, 1.20.\* |
|
+| 3.2.x | 1.17.\*, 1.18.\*, 1.19.\* |
|
+| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Only 3.1.1 supports Flink
1.19.\* |
Review Comment:
The Flink version compatibility table is ambiguous about which API layer it
applies to. In the docs, DataStream/SQL connectors list 3.2.* as supporting
Flink 1.20.*, while pipeline connectors list 3.2.x only up to 1.19.*; the
README table currently presents a single mapping without clarifying this
distinction, which can mislead users. Consider splitting the matrix by API
layer (Pipeline vs Sources/SQL) or adding a clear note on scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
