[
https://issues.apache.org/jira/browse/FLINK-38782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18043026#comment-18043026
]
Joekwal edited comment on FLINK-38782 at 12/8/25 7:31 AM:
----------------------------------------------------------
Well, it occured in incremental streaming stage. The document is 27mb of size
have mentioned in exception.
Config:
'hosts' = '${ext_source.mongodb.aiagent.hosts}',
'collection' = 'aiagent.agent_runs_[0-9]{4}_[0-9]{2}',
'password' = '${ext_source.mongodb.aiagent.password}',
'database' = 'aiagent',
'connector' = 'mongodb-cdc',
'username' = '${ext_source.mongodb.aiagent.username}',
'connection.options' = '${ext_source.mongodb.aiagent.connection_options}'
jar version: flink-sql-connector-{color:#ff3333}mongo{color}db-cdc-2.4.2.jar
was (Author: JIRAUSER288930):
Well, it occured in incremental streaming stage. The document is 27mb of size
have mentioned in exception.
Config:
'hosts' = '${ext_source.mongodb.aiagent.hosts}',
'collection' = 'aiagent.agent_runs_[0-9]{4}_[0-9]{2}',
'password' = '${ext_source.mongodb.aiagent.password}',
'database' = 'aiagent',
'connector' = 'mongodb-cdc',
'username' = '${ext_source.mongodb.aiagent.username}',
'connection.options' = '${ext_source.mongodb.aiagent.connection_options}'
jar version:
flink-sql-connector-{color:#ff3333}mongo{color}db-cdc-2.4.2.jar
> Mongodb CDC need a config to filter large size column
> -----------------------------------------------------
>
> Key: FLINK-38782
> URL: https://issues.apache.org/jira/browse/FLINK-38782
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: Joekwal
> Priority: Major
>
> An error occured:
> {color:#ff3333}com.mongodb.MongoQueryException: Query failed with error code
> 10334 and error message 'Executor error during getMore :: caused by ::
> BSONObj size: 27661090 (0x1A61322) is invalid. Size must be between 0 and
> 16793600(16MB) First element: _id: { _data: "xxx{color}" }' on server
> I'v solved the source problem in mongo, but maybe the developers need a
> config to skip the large size columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
