This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
The following commit(s) were added to refs/heads/master by this push:
new 1ac12c396 [FLINK-36784][docs][pipeline-connector/mysql] Add op_ts
metadata docs for MySQL pipeline connector
1ac12c396 is described below
commit 1ac12c396d2f20c383fa38e75b7204631c4a6ef7
Author: Hang Ruan <[email protected]>
AuthorDate: Thu Mar 13 22:08:26 2025 +0800
[FLINK-36784][docs][pipeline-connector/mysql] Add op_ts metadata docs for
MySQL pipeline connector
This closes #3948.
---
.../docs/connectors/pipeline-connectors/mysql.md | 9 ++++++++
docs/content.zh/docs/core-concept/transform.md | 23 +++++++++++++++++++++
.../docs/connectors/pipeline-connectors/mysql.md | 20 ++++++++++++++++++
docs/content/docs/core-concept/transform.md | 24 ++++++++++++++++++++++
4 files changed, 76 insertions(+)
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md
b/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md
index 5ab0baf1c..ccbfd1fbc 100644
--- a/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md
+++ b/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md
@@ -323,6 +323,15 @@ pipeline:
这是一项实验特性，默认为 false。
</td>
</tr>
+ <tr>
+ <td>metadata.list</td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>String</td>
+ <td>
+ 可额外读取的SourceRecord中元数据的列表，后续可直接使用在transform模块，英文逗号 `,`
分割。目前可用值包含：op_ts。
+ </td>
+ </tr>
</tbody>
</table>
</div>
diff --git a/docs/content.zh/docs/core-concept/transform.md
b/docs/content.zh/docs/core-concept/transform.md
index ef87bd35d..f72df7259 100644
--- a/docs/content.zh/docs/core-concept/transform.md
+++ b/docs/content.zh/docs/core-concept/transform.md
@@ -70,6 +70,29 @@ transform:
| __table_name__ | String | Name of the table that contains the row.
|
| __data_event_type__ | String | Operation type of data change event.
|
+除了以上的元数据字段，连接器也可以解析额外的元数据并放入 DataChangeEvent 的 meta map中，这些元数据也可以在transform使用。
+例如如下的作业，MySQL 连接器可以解析 `op_ts` 元数据并在transform中使用。
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: testuser
+ password: testpwd
+ tables: testdb.customer
+ server-id: 5400-5404
+ server-time-zone: UTC
+ metadata.list: op_ts
+
+transform:
+ - source-table: testdb.customer
+ projection: \*, __namespace_name__ || '.' || __schema_name__ || '.' ||
__table_name__ AS identifier_name, __data_event_type__ AS type, op_ts AS opts
+
+sink:
+ type: values
+```
+
## Metadata relationship
| Type | Namespace | SchemaName | Table |
diff --git a/docs/content/docs/connectors/pipeline-connectors/mysql.md
b/docs/content/docs/connectors/pipeline-connectors/mysql.md
index a5e74f6e3..82e2efab2 100644
--- a/docs/content/docs/connectors/pipeline-connectors/mysql.md
+++ b/docs/content/docs/connectors/pipeline-connectors/mysql.md
@@ -343,6 +343,26 @@ pipeline:
Experimental option, defaults to false.
</td>
</tr>
+ <tr>
+ <td>scan.incremental.snapshot.unbounded-chunk-first.enabled</td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>Boolean</td>
+ <td>
+ Whether to assign the unbounded chunks first during snapshot reading
phase.<br>
+ This might help reduce the risk of the TaskManager experiencing an
out-of-memory (OOM) error when taking a snapshot of the largest unbounded
chunk.<br>
+ Experimental option, defaults to false.
+ </td>
+ </tr>
+ <tr>
+ <td>metadata.list</td>
+ <td>optional</td>
+ <td style="word-wrap: break-word;">false</td>
+ <td>String</td>
+ <td>
+ List of readable metadata from SourceRecord to be passed to downstream
and could be used in transform module, split by `,`. Available readable
metadata are: op_ts.
+ </td>
+ </tr>
</tbody>
</table>
</div>
diff --git a/docs/content/docs/core-concept/transform.md
b/docs/content/docs/core-concept/transform.md
index 26a2ec6ea..1b541e144 100644
--- a/docs/content/docs/core-concept/transform.md
+++ b/docs/content/docs/core-concept/transform.md
@@ -70,6 +70,30 @@ There are some hidden columns used to access metadata
information. They will onl
| __table_name__ | String | Name of the table that contains the row.
|
| __data_event_type__ | String | Operation type of data change event.
|
+Besides these fields, pipeline connectors could parse more metadata and put
them in the meta map of the DataChangeEvent.
+These metadata could be accessed in the transform module.
+For example, MySQL pipeline connector could parse `op_ts` and use it in the
transform module.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: testuser
+ password: testpwd
+ tables: testdb.customer
+ server-id: 5400-5404
+ server-time-zone: UTC
+ metadata.list: op_ts
+
+transform:
+ - source-table: testdb.customer
+ projection: \*, __namespace_name__ || '.' || __schema_name__ || '.' ||
__table_name__ AS identifier_name, __data_event_type__ AS type, op_ts AS opts
+
+sink:
+ type: values
+```
+
## Metadata relationship
| Type | Namespace | SchemaName | Table |
