[
https://issues.apache.org/jira/browse/FLINK-40732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117778#comment-18117778
]
Spoorthi Basu commented on FLINK-40732:
---------------------------------------
[~Leonard] I've opened [PR #4553|https://github.com/apache/flink-cdc/pull/4553]
for this, which wires the FLINK-40697 snapshot metadata release through the
MongoDB connector. Could you please assign this ticket to me when you have a
moment? Thanks!
> Release finished snapshot split metadata in the MongoDB CDC connector
> ---------------------------------------------------------------------
>
> Key: FLINK-40732
> URL: https://issues.apache.org/jira/browse/FLINK-40732
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: Spoorthi Basu
> Priority: Major
> Labels: pull-request-available
>
> The MongoDB CDC connector keeps the snapshot split metadata (assigned splits,
> finished offsets and table schemas) in the source coordinator for the whole
> lifetime of the job, and rebuilds it on every checkpoint. On a job with a
> very large number of snapshot splits this keeps JobManager memory high and
> inflates checkpoint size.
> That metadata is released by the incremental source framework
> ({{flink-cdc-base}}) once the source enters the stream phase, behind
> {{scan.incremental.snapshot.metadata.release.enabled}} (FLINK-40697). The
> four JDBC dialects inherit the option from {{BaseSourceConfig}} and its
> validation from {{JdbcSourceConfig}}. MongoDB implements {{SourceConfig}}
> directly, so it inherits neither and MongoDB jobs cannot enable the option.
> This proposes adding it to the MongoDB connector: the config plumbing, the
> fail-fast against {{scan.newly-added-table.enabled}}, and the option itself
> via both the DataStream builder and Flink SQL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
