Hi Jingsong, thank you for writing up the proposal. The benefits such a mechanism will bring will be very valuable! I haven't yet looked into this in detail, but one question came to my mind immediately:
The DDL for these tables seems to rely on there not being a 'connector' option. However, catalogs can provide a custom factory, and thus tables don't necessarily need to contain such an option already today. How will this interact / work with catalogs? I think there are more points regarding interaction with catalogs, e.g. if tables are dropped externally rather than through Flink SQL DDL, how would Flink be able to remove the physical storage for it. Best Ingo On Wed, Oct 20, 2021 at 11:14 AM Jingsong Li <jingsongl...@gmail.com> wrote: > Hi all, > > Kurt and I propose to introduce built-in storage support for dynamic > table, a truly unified changelog & table representation, from Flink > SQL’s perspective. We believe this kind of storage will improve the > usability a lot. > > We want to highlight some characteristics about this storage: > > - It’s a built-in storage for Flink SQL > ** Improve usability issues > ** Flink DDL is no longer just a mapping, but a real creation for these > tables > ** Masks & abstracts the underlying technical details, no annoying options > > - Supports subsecond streaming write & consumption > ** It could be backed by a service-oriented message queue (Like Kafka) > ** High throughput scan capability > ** Filesystem with columnar formats would be an ideal choice just like > iceberg/hudi does. > > - More importantly, in order to solve the cognitive bar, storage needs > to automatically address various Insert/Update/Delete inputs and table > definitions > ** Receive any type of changelog > ** Table can have primary key or no primary key > > Looking forward to your feedback. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage > > Best, > Jingsong Lee >
