[
https://issues.apache.org/jira/browse/FLINK-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061351#comment-17061351
]
Kurt Young commented on FLINK-16639:
------------------------------------
does current \{{format.ignore-parse-errors}} meet your requirement?
> Flink SQL Kafka source connector, add the no json format filter params when
> format.type is json
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-16639
> URL: https://issues.apache.org/jira/browse/FLINK-16639
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Reporter: LakeShen
> Priority: Minor
> Fix For: 1.10.2
>
>
> In my thought, kafka source connector is the one of most frequently used
> connector in flink sql. Flink sql kafka source connector supports the
> json,csv or other data format. But there is a problem for json format in
> kafka source connector. For example, flink sql kafka source ddl l
> like this:
> CREATE TABLE team_olap_table (
> a varchar,
> b varchar,
> )
> with (
> 'connector.type' = 'kafka',
> 'connector.version' = '0.10',
> 'connector.topic' = topics',
> 'connector.properties.0.key' = 'group.id',
> 'connector.properties.0.value' = 'hello_world',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'xxx',
> 'connector.property-version' = '1',
> 'connector.startup-mode' = 'latest-offset',
> 'format.type' = 'json',
> 'format.property-version' = '1',
> 'format.derive-schema' = 'true',
> 'update-mode' = 'append'
> );
> If the kafka topic messages are not json format ,just one or two records,the
> flink sql task will fail-over all the time .
> In order to solve this problem , if flink sql source connector use the
> json-format, I want to add the 'format.fail-on-not-json-record' param in
> flink-json module, if this param is true（default），when read the no-json
> records, the flink will fail, if this param is false, the flink sql task will
> filter no-json records,the flink task running normally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
