[
https://issues.apache.org/jira/browse/FLINK-23613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393056#comment-17393056
]
Jark Wu commented on FLINK-23613:
---------------------------------
> calculating the total number of delete or update record from mysql, or output
> different 'op' of data to different tables for batch processing.
If this is your use case, I would suggest to use "json" format instead of
"debezium-json".
"debezium-json" will convert the events into Flink changelog (you can think it
as of a materialized table equivalent to the original database table), that
means you can't get an exepcted result when calculating count/sum for
update/deletes. For example, count on deletes would always be zero, because
there is no "delete" record in the original databse table.
> debezium and canal support read medata op and type
> --------------------------------------------------
>
> Key: FLINK-23613
> URL: https://issues.apache.org/jira/browse/FLINK-23613
> Project: Flink
> Issue Type: New Feature
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Ward Harris
> Priority: Major
>
> in our scene, there will be two types of database data delivered to the data
> warehouse:
> 1. the first type is exactly the same as the online table
> 2. the second type adds two columns on the basis of the previous table,
> representing action_type and action_time respectively, which is to record
> events
> in order to solve this demand by flink sql, it is necessary to be able to
> read the action_type and action_time from debezium or canal metadata,
> action_time can read from ingestion-timestamp metadata, but can not read
> action_type from metadata.
> the database action is insert/update/delete, but there will be
> insert/update_before/update_after/delete in Flink's RowKind, so action_type
> is RowKind will be better for us. at the same time, flink needs to modify
> RowKind to insert for record this event table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
