fapaul commented on pull request #3213: URL: https://github.com/apache/iceberg/pull/3213#issuecomment-946521587
@kbendick thanks for the introduction. I think @rdblue already made a good point noting that Flink's current FileSink is not extensible enough and can thereof not support most of the use cases. We are currently working on bringing compaction to the FileSink and as a next step try to generify it to support more usecases. One thing I am still curious about is what drove the decision that the Flink Iceberg connector is built with the low-level operator API rather than using the unified Sink interface[1]. In general, we designed the interface to allow users to support batch and streaming executions without the need to worry about the different semantics. I know there have been problems in the past about Flink's API stability and I am a bit worried that the operator model the Iceberg Sink currently uses, might be become difficult to maintain in the future. Sorry for abusing this pull request thread for some general discussion ... [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
