[
https://issues.apache.org/jira/browse/FLINK-18758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692639#comment-17692639
]
Frank Lyaruu commented on FLINK-18758:
--------------------------------------
Hey all, is there any planned work on this issue? I did some demo's with
Debezium+Flink last year, which worked nicely, but now I am applying this in a
slightly more serious setting, and run immediately into the problem that
timestamps simply do not work.
If I hack flink-json to check if the date representation is a number, and in
that case consider it as epoch microseconds, it does work. I am using debezium
JSON right now with a schema attached, so it does get the info that it is a) a
microsecond timestamp b) encoded as int64, so it does have all necessary info.
I can spend time on a more real solution, but as this is a pretty old ticket,
it like to check in with the community first.
> Many data types do not work in debezium format
> -----------------------------------------------
>
> Key: FLINK-18758
> URL: https://issues.apache.org/jira/browse/FLINK-18758
> Project: Flink
> Issue Type: Sub-task
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Leonard Xu
> Priority: Major
> Labels: auto-unassigned
>
> Currently debezium json format wrapper the json format to
> serialize/deserialize the debezuim json format data.
> But debezium json format has its own data type which consists of Literal Type
> and Semantic Type[1], i.g. Date type in debezium is an integer which
> represents the number of days since epoch rather than a string with
> 'yyyy-MM-dd' pattern.
> {code:java}
> { "schema":{
> "fields":[
> {
> "fields":[
> {
> "type":"int32", //Literal Type
> "optional":false,
> "name":"io.debezium.time.Date", //semantic Type
> "version":1,
> "field":"order_date"
> },
> { "type":"int32",
> "optional":false,
> "field":"quantity"
> }
> ]
> }
> ]
> },
> "payload":{
> "before":null,
> "after":{
> "order_date":16852, // Literal Value, the number of days since epoch
> "quantity":1
> },
> "op":"c",
> "ts_ms":1596081813474
> }
> } {code}
>
> I think we need obtain the debezuim data type from schema information and
> then serialize/deserialize the data in payload.
> [1][https://debezium.io/documentation/reference/1.2/connectors/mysql.html]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
