kinolaev commented on PR #14797: URL: https://github.com/apache/iceberg/pull/14797#issuecomment-5317728219
Slightly off-topic, but since this PR is about the Kafka Connect sink, I built a Flink-based alternative: https://github.com/kinolaev/iceberg-flink-kafka. [Flink Dynamic Iceberg Sink](https://iceberg.apache.org/docs/latest/flink-writes/#flink-dynamic-iceberg-sink) supports upsert mode, and `iceberg-flink` has an Avro-to-Iceberg mapper. In my project I just wired them together with the [Flink Kafka Connector](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/). The project supports most of the configuration properties from Iceberg Kafka Connect to ease migration. Additionally, it has a full `compose` environment with `postgres` and `debezium-server` to see it in action. So if you're looking to migrate to Flink Dynamic Iceberg Sink and Avro is a valid serialization option for you, you're more than welcome to test out and use the project. The Flink sink allows automatic execution of maintenance tasks after data is committed to the table, and recently the [ConvertEqualityDeletes procedure](https://iceberg.apache.org/docs/nightly/flink-maintenance/#convertequalitydeletes) has been merged into the `main` branch. Combined, these two make Flink the most promising solution for CDC scenarios, in my opinion. I don't know if the coordinated maintenance is already integrated with Flink Dynamic Iceberg Sink, but I plan to look into it and provide a way to configure it in my project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
