Hi Daniel, I think, it would be great to have a PubSub Lite Connector in Flink. Before you put this proposal up for a vote, though, we need feedback from a Committer who would review and help maintain it going forward. Ideally, this Committer would guide one or more contributors from Google to Committership so that Google could step up and maintain Flink's PubSub and PubSub Lite Connector in the future. For this, it would be good to understand how you envision the involvement of the PubSub Lite team at Google.
I am specifically sensitive on this topic, because the PubSub connector has lacked attention and maintenance for a long time. There was also a very short-lived interested by Google in the past to contribute a Google PubSub Connector [1]. Best, Konstantin [1] https://issues.apache.org/jira/browse/FLINK-22380 Am Mi., 8. März 2023 um 14:45 Uhr schrieb Etienne Chauchot < echauc...@apache.org>: > Hi, > > I agree with Ryan, even if clients might be totally different the > backend technologies are the same so hosting them in the same repo makes > sense. Similar thinking made us put all the Cassandra related connectors > in the same cassandra repo. > > Etienne > > Le 02/03/2023 à 14:43, Daniel Collins a écrit : > > Hello Ryan, > > > > Unfortunately there's not much shared logic between the two- the clients > > have to look fundamentally different since the Pub/Sub Lite client > exposes > > partitions to the split level for repeatable reads. > > > > I have no objection to this living in the same repo as the Pub/Sub > > connector, if this is an easier way forward than setting up a new repo, > > sounds good to me. The Pub/Sub team is organizationally close to us, and > is > > looking into providing more support for the flink connector in the near > > future. > > > > -Daniel > > > > On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba <ryan.skr...@aiven.io.invalid > > > > wrote: > > > >> Hello Daniel! Quite a while ago, I started porting the Pub/Sub > connector > >> (from an existing PR) to the new source API in the new > >> flink-connector-gcp-pubsub repository [PR2]. As Martijn mentioned, > there > >> hasn't been a lot of attention on this connector; any community > involvement > >> would be appreciated! > >> > >> Instead of considering this a new connector, is there an opportunity > here > >> to offer the two variants (Pub/Sub and Pub/Sub Lite) as different > artifacts > >> in that same repo? Is there much common logic that can be shared > between > >> the two? I'm not as familiar as I should be with Lite, but I do recall > >> that they share many concepts and _some_ dependencies. > >> > >> All my best, Ryan > >> > >> > >> On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins > >> <dpcoll...@google.com.invalid> > >> wrote: > >> > >>> Hello all, > >>> > >>> I'd like to start an official discuss thread for adding a Pub/Sub Lite > >>> Connector to Flink. We've had requests from our users to add flink > >> support, > >>> and are willing to maintain and support this connector long term from > the > >>> product team. > >>> > >>> The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what > would > >>> be > >>> people's thoughts on adding this connector? > >>> > >>> -Daniel > >>> > -- https://twitter.com/snntrable https://github.com/knaufk
