[
https://issues.apache.org/jira/browse/FLINK-38813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tejansh Rana updated FLINK-38813:
---------------------------------
Description:
Enable Flink CDC connectors to emit Debezium heartbeat events to downstream
consumers, allowing applications to monitor database connection health and
implement custom metrics for change data capture pipelines.
*Problem Statement:*
Flink CDC connectors currently skip heartbeat events generated by Debezium,
making it difficult for downstream applications to:
* Monitor database connection health and liveness
* Track replication lag in real-time
* Implement custom metrics for data streaming pipeline health
* Detect stalled or failed binlog reading
The Flink CDC connectors process heartbeat events internally for offset
tracking but do not emit them to downstream consumers.
*Proposed Solution:*
Expose heartbeat events as optional emitted records to enable downstream
consumers to consume these events and implement their custom solutions.
This config would be false by default to allow backward compatibility. When set
true by users, the Record Emitter will emit the heartbeat events in addition to
the existing behaviour of updating the offsets.
I would be happy to create a PR for this feature should we accept this
proposal. The implementation would be similar to the feature proposed in this
ticket - https://issues.apache.org/jira/browse/FLINK-38691. Reference PR -
[https://github.com/apache/flink-cdc/pull/4170]
Draft PR to demonstrate the proposed changes in MySQL connector; others to
follow the same pattern - https://github.com/apache/flink-cdc/pull/4193
was:
Enable Flink CDC connectors to emit Debezium heartbeat events to downstream
consumers, allowing applications to monitor database connection health and
implement custom metrics for change data capture pipelines.
*Problem Statement:*
Flink CDC connectors currently skip heartbeat events generated by Debezium,
making it difficult for downstream applications to:
* Monitor database connection health and liveness
* Track replication lag in real-time
* Implement custom metrics for data streaming pipeline health
* Detect stalled or failed binlog reading
The Flink CDC connectors process heartbeat events internally for offset
tracking but do not emit them to downstream consumers.
*Proposed Solution:*
Expose heartbeat events as optional emitted records to enable downstream
consumers to consume these events and implement their custom solutions.
This config would be false by default to allow backward compatibility. When set
true by users, the Record Emitter will emit the heartbeat events in addition to
the existing behaviour of updating the offsets.
I would be happy to create a PR for this feature should we accept this
proposal. The implementation would be similar to the feature proposed in this
ticket - https://issues.apache.org/jira/browse/FLINK-38691. Reference PR -
[https://github.com/apache/flink-cdc/pull/4170]
> Enable Flink CDC connectors to emit Debezium heartbeat events to downstream
> consumers
> -------------------------------------------------------------------------------------
>
> Key: FLINK-38813
> URL: https://issues.apache.org/jira/browse/FLINK-38813
> Project: Flink
> Issue Type: New Feature
> Components: Flink CDC
> Reporter: Tejansh Rana
> Priority: Major
> Labels: pull-request-available
>
> Enable Flink CDC connectors to emit Debezium heartbeat events to downstream
> consumers, allowing applications to monitor database connection health and
> implement custom metrics for change data capture pipelines.
> *Problem Statement:*
> Flink CDC connectors currently skip heartbeat events generated by Debezium,
> making it difficult for downstream applications to:
> * Monitor database connection health and liveness
> * Track replication lag in real-time
> * Implement custom metrics for data streaming pipeline health
> * Detect stalled or failed binlog reading
> The Flink CDC connectors process heartbeat events internally for offset
> tracking but do not emit them to downstream consumers.
>
> *Proposed Solution:*
> Expose heartbeat events as optional emitted records to enable downstream
> consumers to consume these events and implement their custom solutions.
> This config would be false by default to allow backward compatibility. When
> set true by users, the Record Emitter will emit the heartbeat events in
> addition to the existing behaviour of updating the offsets.
> I would be happy to create a PR for this feature should we accept this
> proposal. The implementation would be similar to the feature proposed in this
> ticket - https://issues.apache.org/jira/browse/FLINK-38691. Reference PR -
> [https://github.com/apache/flink-cdc/pull/4170]
>
> Draft PR to demonstrate the proposed changes in MySQL connector; others to
> follow the same pattern - https://github.com/apache/flink-cdc/pull/4193
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
