stevenzwu commented on issue #1438: URL: https://github.com/apache/iceberg/issues/1438#issuecomment-694977225
There is a also a broader discussion in Flink community regarding a new unified (stream and batch) sink interface, which can really help the Flink Iceberg sink: (1) simplify committer code (including the ManifestFile discussion here) (2) run committer in jobmanager which can avoid the current limitation of invalidate embarrassingly parallel DAG. If we migrate to the new sink interface, it will likely be an incompatible change (including state). I think the new sink interface is targeted for the upcoming Flink 1.12 (ETA is Nov) @rdblue @openinx @JingsongLi I am wondering if we should release Iceberg with the current V1 product and plus some blocker fixes, like [1]. If we have time for those incompatible changes, that is great. if not, we can come back for the major incompatible changes once Flink released the new sink interface in Flink 1.12. I will also try to help then. [1] https://github.com/apache/iceberg/pull/1404 (merged already) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
