Hi Umesh, Recently I’m working on support Confluent debezium avro format in Kafka cdc based on [1]. But the Paimon community is planning to cut 0.7.0 release branch at Jan. 25th. And I think there is not enough time for me to complete the job before the deadline for some reasons:
1. I have to modify the current CDC framework. Currently, the intermediate records from Kafka source are string type. But for debezium-avro, the intermediate records are avro objects, so we have to adjust the framework. It needs some time. 2. I noticed that you want to support some complex type in [2] which made some changes to TypeUtils. Since this util is used by many features, we should do some tests to see if the changes are compatible with other features. I think if we implement a simple version in this release which doesn’t support those complex types, this release cannot meet your situation. So I suggest that you continue to use the jar buit by yourself. Recently I’m also woking on preparing to release 0.7.0. I will continue on Debezium-avro format in 0.8.0. If you have any problems with [1], welcome to discuss with us in mailing list. Best, Zelin Yu [1] https://github.com/apache/incubator-paimon/pull/2070 [2] https://github.com/harveyyue/incubator-paimon/pull/1 > 2024年1月10日 01:21，umesh dangat <[email protected]> 写道： > > Hello, > > I am a software engineer at Yelp Inc and lead the data infrastructure > group at Yelp. We have a complex real time streaming ecosystem comprising > flink, kafka and our custom schema registry service. I am trying to > evaluate Apache Paimon as a potential replacement for a lot of our data > pipelines, involving streaming reads, joins and aggregations to help > minimize our growing operational complexity and cost. Also paimon seems to > solve the schema evolution problem better than flink sqlclient? (which we > use currently) > > One issue with integrating paimon in our ecosystem seems to be that it does > not support debezium avro format. Although Jingsong Li pointed me to this > <https://github.com/apache/incubator-paimon/pull/2070> branch that does > seem to add support for debezium avro format using confluent schema > registry. This would allow us to ingest our data from kafka into paimon and > then evaluate it. > > I wanted to know if we have plans to push this branch to master soonish. I > can help with reviewing, since I plan to consume data written using this > format for some of our production workflows. > > Thanks, > Umesh
