Hi Ahmed, thank you for starting this discussion. In general, I am more inclined toward proposal 2 - having logic that dynamically selects implementation under the hood might introduce unexpected behaviors for users, considering that one of the options is an experimental connector with an unstable API.
Best regards, Aleksandr Pilipenko On Sun, 24 Mar 2024 at 15:19, Ahmed Hamdy <hamdy10...@gmail.com> wrote: > Hi devs, > With the delivery of FLINK-31813 > <https://issues.apache.org/jira/browse/FLINK-31813>[1] that offers a new > Datastream Kinesis Streams source paving the way to deprecate the KCL > backed `FlinkKinesisConsumer`, I would like to start working on migrating > Kinesis Source Table API to the new source (FLINK-31987 > <https://issues.apache.org/jira/browse/FLINK-31987>[2]) in a > similar approach to what I did with the sink migration. This work however > is blocked on migration of additional features like EFO support and > deaggregation which seems to be idle for some time now. I want to discuss > an alternative approach for migration ensuring backward compatibility. > > Proposal 1: Given the pre-existing dependency from > `flink-connector-kinesis` to `flink-connector-aws-kinesis-streams` I > suggest we implement a `KinesisStreamsDynamicSource` backed by the new > `KinesisStreamsSource` and add a `KinesisSourceSelector` to the existing > source factory to dynamically select a source according to features enabled > from the DDL, i.e shall the unsupported features like EFO or aggregation be > enabled the legacy dynamic source would be provided, otherwise the new > dynamic source is to be provided. Once all other features are implemented > we migrate the factory as well. > Proposal 2: We do nothing and wait for all features to be implemented. > Let me know your thoughts. > > 1- https://issues.apache.org/jira/browse/FLINK-31813 > 2- https://issues.apache.org/jira/browse/FLINK-31987 > Best Regards > Ahmed Hamdy >
