Hello,

 I am a software engineer at Yelp Inc and lead the data infrastructure
group at Yelp. We have a complex real time streaming ecosystem comprising
flink, kafka and our custom schema registry service. I am trying to
evaluate Apache Paimon as a potential replacement for a lot of our data
pipelines, involving streaming reads, joins and aggregations to help
minimize our growing operational complexity and cost. Also paimon seems to
solve the schema evolution problem better than flink sqlclient? (which we
use currently)

One issue with integrating paimon in our ecosystem seems to be that it does
not support debezium avro format. Although Jingsong Li pointed me to this
<https://github.com/apache/incubator-paimon/pull/2070> branch that does
seem to add support for debezium avro format using confluent schema
registry. This would allow us to ingest our data from kafka into paimon and
then evaluate it.

I wanted to know if we have plans to push this branch to master soonish. I
can help with reviewing, since I plan to consume data written using this
format for some of our production workflows.

Thanks,
Umesh

Reply via email to