Re: [DISCUSS] Sink V2 API missing features

2023-09-29 Thread Péter Váry
Hi Martijn, Thanks for the prompt answer, good to hear that there is interest in the community to improve things around the APIs! *First and foremost, we still plan to use the PostCommitTopology* to do the compaction for the Iceberg Sink. That is on our roadmap, but the first step is to do the Si

Re: [DISCUSS] Sink V2 API missing features

2023-09-28 Thread Martijn Visser
Hi Peter, I actually had a session this week at Current, talking about the state of the Flink connector ecosystem. Afterwards, I had a lot of conversations with people who were interested in the compaction with Iceberg and I actually was planning to look at what the current status was on the Icebe

[DISCUSS] Sink V2 API missing features

2023-09-28 Thread Péter Váry
Hi Flink Team, I am working on implementing a new Iceberg Sink which would use the new SinkV2 API [1], [2]. The main goal is to open up the possibility for doing the incremental small files compaction inside a Flink job using the new Iceberg Sink. During the implementation I have found several pl