Hi Bryan, I am very happy to see this contribution. I have recently tested this project with Nessie catalog and very much liked it.
However, I still don't know the benefits of using kafka-connect instead of directly consuming from the kafka like Delta-lake's implementation. https://github.com/delta-io/kafka-delta-ingest/blob/main/doc/DESIGN.md I am not an expert in this ingestion domain and recently got started. I hope someone will chime in and we will have detailed analysis over the design. Looking forward to this feature. Thanks, Ajantha On Tue, Oct 3, 2023 at 12:18 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Bryan > > That’s a great news ! Thanks a lot for the proposal. > > I will take a look on the PR and existing connector. > I’m sure the Iceberg community will be very happy to see this and we will > able to add new features and improvements thanks to the community feedback. > I would be more than happy to help for donation (I know that the connector > is already under Apache license but we have to double check the ICLA for > the initial contributors etc , just to be sure we are good there). > > Thanks again ! > > Let’s see what the others are thinking. > > Regards > JB > > Le lun. 2 oct. 2023 à 19:39, Bryan Keller <brya...@gmail.com> a écrit : > >> Hi all, >> >> We at Tabular would like to contribute our Kafka Connect Iceberg sink to >> the Iceberg project. It would be great to give Iceberg users another option >> for landing data from Kafka into Iceberg tables that is supported by the >> Iceberg community. Kafka Connect is a part of systems from AWS, Confluent, >> Redpanda, and so on, so it can make landing data from Kafka into Iceberg >> much easier for those without a Flink or Spark infrastructure. >> >> There are a few Iceberg sink implementations out there for Kafka Connect, >> but we feel this one covers most of the features users have requested, such >> as exactly-once processing, schema evolution, and multi-table fanout. And >> having the sink backed by the Iceberg community will help it to evolve and >> improve over time. >> >> If this sounds like something everyone would like to see added to >> Iceberg, I've opened a PR that includes some initial pieces of the sink. >> The thought was to break up the submission into parts so each could be >> reviewed more easily. Some design docs and notes can be found in the >> original repo here: >> https://github.com/tabular-io/iceberg-kafka-connect/tree/main/docs >> >> We'd like to get feedback if others approve of moving forward with this >> or not. >> >> Thanks, >> Bryan >> >>
