[
https://issues.apache.org/jira/browse/FLINK-37017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-37017:
-------------------------------
Description:
https://github.com/apache/flink-cdc/pull/3434#issuecomment-2192072774
{code:java}
I was writing a custom flink cdc source connector. Input data has complex
nested types. That is how I ran into the issue of BinaryRecordData not
supporting complex types such as ARRAY and MAP, including AbstractBinaryWriter
not supporting those. Most of the code here is copied from flink source
(including relevant tests).
I was also trying to get my data eventually into paimon, using paimon sink.
Thus I have made relevant changes to PaimonWriterHelper as well to handle
complex types.
Lastly, I came across a bug due to circular dependency between
DataFieldSerializer, DataTypeSerializer, and RowTypeSerializer classes that
causes NPE for certain RowTypes like nested row type. I have baked that change
in this PR but also have a separate one here with more explanation
{code}
> Supports complex DataTypes in Flink CDC
> ---------------------------------------
>
> Key: FLINK-37017
> URL: https://issues.apache.org/jira/browse/FLINK-37017
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.1
> Reporter: Leonard Xu
> Assignee: Leonard Xu
> Priority: Major
> Fix For: cdc-3.3.0
>
>
> https://github.com/apache/flink-cdc/pull/3434#issuecomment-2192072774
> {code:java}
> I was writing a custom flink cdc source connector. Input data has complex
> nested types. That is how I ran into the issue of BinaryRecordData not
> supporting complex types such as ARRAY and MAP, including
> AbstractBinaryWriter not supporting those. Most of the code here is copied
> from flink source (including relevant tests).
> I was also trying to get my data eventually into paimon, using paimon sink.
> Thus I have made relevant changes to PaimonWriterHelper as well to handle
> complex types.
> Lastly, I came across a bug due to circular dependency between
> DataFieldSerializer, DataTypeSerializer, and RowTypeSerializer classes that
> causes NPE for certain RowTypes like nested row type. I have baked that
> change in this PR but also have a separate one here with more explanation
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
