Hi Sijie, Yes, we will have to support existing old connectors and new connectors in parallel for a while. We have to take that maintenance overhead because existing connectors have been used by the users for a long time. I guess It may take at least a year for us to fully remove the old connectors.
Process wise, we can do the same for Pulsar connector. But I am not sure if we want to have the same burden on Pulsar connector, and I would like to understand the benefit of doing that. For users, the benefit of having the old Pulsar connector checked in seems limited because 1) that code base will be immediately deprecated in the next release in 3-4 months; 2) users can always use it even if it is not in the Flink code base. Admittedly it is not as convenient as having it in Flink code base, but doesn't seem super either. And after 3-4 months, users can just use the new connector in Flink repo. For Flink developers, the old connector code base is not something that we want to evolve later. Instead, these code will be deprecated and removed. So why do we want to get a beta version out to attract people to use something we don't want to maintain? Thanks, Jiangjie (Becket) Qin On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> wrote: > Thanks everyone here. Sorry for jumping into the discussion here. > > I am not very familiar about the deprecation process in Flink. If I > misunderstood the process, please fix me. > > As far as I understand, FLIP-27 is introducing a new unified API for > connectors. After it introduces the new API > and before moving all the existing connectors from old API to new API, both > old ApI and new API will co-exist > for a while until Flink moves all existing connectors to new API. So the > Pulsar connector (using old API) can > follow the deprecation process with other connector using old API and the > deprecation of old API, no? > > If that's the case, I think contributing the current connector back to > Flink rather than maintaining it outside Flink > would provide a bit more benefits. We can deprecate the existing > streamnative/pulsar-flink repo and point the users > to use the connector in Flink repo. So all the review processes will happen > within Flink for both old connector and > new connector. It also reduces the confusions for the users as the > documentation and code base happen in one place. > > Thoughts? > > - Sijie > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <becket....@gmail.com> wrote: > > > Thanks for the explanation, Stephan. I have a few questions / thoughts. > > > > So that means we will remove the old connector without a major version > > bump, is that correct? > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will > > always work because we saw some dependency class collisions in the past. > To > > make it safe we may have to maintain the old code for one more release. > > > > To be honest I am still wondering if we have to put the old connector in > > Flink repo. if we check in the old connector to Flink. We will end up in > > the following situation: > > 1. Old connector in streamnative/pulsar-flink repo. > > 2. Old connector in Flink Repo, which may be different from the one in > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12) > > 3. New connector in Flink Repo. > > > > We need to think about how to make the users in each case happy. > > - For users of (1), I assume Sijie and Yijie will have to maintain the > code > > a bit longer for its own compatibility even after we have (2). In that > > case, bugs found in old connector may or may not need to be fixed in both > > Flink and the streamnative/pulsar-flink repo. > > - For users of (2), will we provide bug fixes? If we do, it will be a > > little awkward because those bug fixes will be immediately deprecated in > > 1.11, and removed in 1.12. So we are essentially asking users to migrate > > away from the bug fix. After Flink 1.12, users may still have to switch > to > > use (3) due to the potential dependency class conflicts mentioned above. > > - Users of (3) have a much easier life and don't need to worry too much. > > > > The above story seems a little complicated to tell. I think it will be > much > > easier to not have (2) at all. > > 1. Old connector in streamnative/pulsar-flink repo. > > 3. New connector in Flink Repo. > > > > - Old connector will only be maintained in streamnative/pulsar-flink repo > > until it is fully deprecated. Users can always use the existing Pulsar > > connector in that repo. > > - New connector will be in Flink repo and maintained like the other > > connectors. > > > > This seems much simpler for users understand and they will not be blocked > > from using the old connector. If the concern is about the quality of the > > connector in streamnative/pulsar-flink repo, is it enough for us just to > > review the code in streamnative/pulsar-flink connector to make sure it > > looks good from Flink's perspective? > > > > What do you think? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <se...@apache.org> wrote: > > > > > My take would be the following: > > > > > > - If we merge the connector now and replace it with a FLIP-27 version > > > before the 1.10 release, then we need no deprecation process > > > - If we don't manage to replace it with a FLIP-27 version before the > > 1.10 > > > release, than it is good that we have the other version, so no users > get > > > blocked. > > > > > > In the latter case we can see how we want to do it. Immediate removal > of > > > the old version or deprecation label and keeping it for one more > release. > > > Given that you should be able to use a Flink 1.10 connector with Flink > > 1.11 > > > as well (stable public APIs) there is also a workaround if you need an > > old > > > connector in a newer version. So immediate removal might even be > > feasible. > > > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > Hi Stephan, > > > > > > > > Thanks for the clarification. I completely agree with you and Thomas > on > > > the > > > > process of adding connectors to Flink repo. However, I am wondering > > what > > > is > > > > the deprecation process? Given the main concern here was that we may > > have > > > > to maintain two Pulsar connector code bases until the old one is > > removed > > > > from the repo, it would be good to know how long we have to do that. > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <se...@apache.org> > wrote: > > > > > > > > > Some quick thoughts on the connector contribution process. I > > basically > > > > > reiterate here what Thomas mentioned in another thread about the > > > Kinesis > > > > > connector. > > > > > > > > > > For connectors, we should favor a low-overhead contribution > process, > > > and > > > > > accept user code and changes more readily than in the core system. > > > > > That is because connectors have both a big variety of scenarios > they > > > get > > > > > used in (only through use and many small contributions do they > become > > > > > really useful over time) and at the same time, and committers do > not > > > use > > > > > the connector themselves and usually cannot foresee too well what > is > > > > > needed. > > > > > > > > > > Further more, a missing connector (or connector feature) is often a > > > > bigger > > > > > show stopper for users than a missing API or system feature. > > > > > > > > > > Along these lines of thougt, the conclusion would be to take the > > Pulsar > > > > > connector now, focus the review on legal/dependencies/rough code > > style > > > > and > > > > > conventions, label it as "beta" (in the sense of "new code" that is > > > "not > > > > > yet tested through longer use") and go ahead. And then evolve it > > > quickly > > > > > without putting formal blockers in the way, meaning also adding a > new > > > > FLIP > > > > > 27 version when it is there. > > > > > > > > > > Best, > > > > > Stephan > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > Could you please follow the FLIP process to start a new FLIP > > > > [DISCUSSION] > > > > > > thread in the mailing list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So there > is a > > > > FLIP > > > > > > number collision. Can you change the FLIP number to 72? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <walter...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > > > > > Would you mind enabling comments/suggestions on the google doc > > > link? > > > > > This > > > > > > > way the contributors from the community can comment on the doc. > > > > > > > > > > > > > > Best, > > > > > > > Rong > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > > henry.yijies...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > I've drafted a FLIP that describes the current design of the > > > Pulsar > > > > > > > > connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yijie > > > > > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > walter...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Sorry for joining the discussion late and thanks Yijie & > > Sijie > > > > for > > > > > > > > driving > > > > > > > > > the discussion. > > > > > > > > > I also think the Pulsar connector would be a very valuable > > > > addition > > > > > > to > > > > > > > > > Flink. I can also help out a bit on the review side :-) > > > > > > > > > > > > > > > > > > Regarding the timeline, I also share concerns with Becket > on > > > the > > > > > > > > > relationship between the new Pulsar connector and FLIP-27. > > > > > > > > > There's also another discussion just started by Stephan on > > > > dropping > > > > > > > Kafka > > > > > > > > > 9/10 support on next Flink release [1]. Although the > > situation > > > > is > > > > > > > > somewhat > > > > > > > > > different, and Kafka 9/10 connector has been in Flink for > > > almost > > > > > 3-4 > > > > > > > > years, > > > > > > > > > based on the discussion I am not sure if a major version > > > release > > > > > is a > > > > > > > > > requirement for removing old connector supports. > > > > > > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree the old > > > > connector > > > > > > will > > > > > > > > be > > > > > > > > > removed once FLIP-27 based Pulsar connector is there. As > > > Stephan > > > > > > > stated, > > > > > > > > it > > > > > > > > > is easier to contribute the source sooner and adjust it > > later. > > > > > > > > > We should also ensure we clearly communicate the message: > for > > > > > > example, > > > > > > > > > putting an experimental flag on the pre-FLIP27 connector > page > > > of > > > > > the > > > > > > > > > website, documentations, etc. Any other thoughts? > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > > > becket....@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Technically speaking, removing the old connector code is > a > > > > > > backwards > > > > > > > > > > incompatible change which requires a major version bump, > > i.e. > > > > > Flink > > > > > > > > 2.x. > > > > > > > > > > Given that we don't have a clear plan on when to have the > > > next > > > > > > major > > > > > > > > > > version release, it seems unclear how long the old > > connector > > > > code > > > > > > > will > > > > > > > > be > > > > > > > > > > there if we check it in right now. Or will we remove the > > old > > > > > > > connector > > > > > > > > > > without a major version bump? In any case, it sounds not > > > quite > > > > > user > > > > > > > > > > friendly to the those who might use the old Pulsar > > > connector. I > > > > > am > > > > > > > not > > > > > > > > sure > > > > > > > > > > if it is worth these potential problems in order to have > > the > > > > > Pulsar > > > > > > > > source > > > > > > > > > > connector checked in one or two months earlier. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen < > > > se...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Agreed, if we check in the old code, we should make it > > > clear > > > > > that > > > > > > > it > > > > > > > > will > > > > > > > > > > > be removed as soon as the FLIP-27 based version of the > > > > > connector > > > > > > is > > > > > > > > > > there. > > > > > > > > > > > We should not commit to maintaining the old version, > that > > > > would > > > > > > be > > > > > > > > indeed > > > > > > > > > > > too much overhead. > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin < > > > > > becket....@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > > > > > > > > > > > > > > > > > > > > > > Yes, the overhead would just be review capacity. In > > > fact, I > > > > > am > > > > > > > not > > > > > > > > > > > worrying > > > > > > > > > > > > too much about the review capacity. That is just a > one > > > time > > > > > > cost. > > > > > > > > My > > > > > > > > > > > > concern is mainly about the long term burden. Assume > we > > > > have > > > > > > new > > > > > > > > source > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar > > > connectors > > > > in > > > > > > old > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new > source > > > > > > interface, > > > > > > > > the > > > > > > > > > > old > > > > > > > > > > > > Pulsar connector might be deprecated almost > immediately > > > > after > > > > > > > > checked > > > > > > > > > > in, > > > > > > > > > > > > but we may still have to maintain two code bases. For > > the > > > > > > > existing > > > > > > > > > > > > connectors, we have to do that anyways. But it would > be > > > > good > > > > > to > > > > > > > > avoid > > > > > > > > > > > > introducing a new connector with the same problem. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen < > > > > > se...@apache.org > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > > > Nice to see this lively discussion about the Pulsar > > > > > > connector. > > > > > > > > > > > > > Some thoughts on the open questions: > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a community > > > package > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like the discussion is more going towards > > > > > > contribution. I > > > > > > > > think > > > > > > > > > > > > that > > > > > > > > > > > > > is good, especially if we think that we want to > > build a > > > > > > > similarly > > > > > > > > > > deep > > > > > > > > > > > > > integration with Pulsar as we have for example with > > > > Kafka. > > > > > > The > > > > > > > > > > > connector > > > > > > > > > > > > > already looks like a more thorough connector than > > many > > > > > others > > > > > > > we > > > > > > > > have > > > > > > > > > > > in > > > > > > > > > > > > > the repository. > > > > > > > > > > > > > > > > > > > > > > > > > > With either a repo split, or the new build system, > I > > > hope > > > > > > that > > > > > > > > the > > > > > > > > > > > build > > > > > > > > > > > > > overhead is not a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > ## Committer Support > > > > > > > > > > > > > > > > > > > > > > > > > > Becket offered some help already, I can also help a > > > bit. > > > > I > > > > > > hope > > > > > > > > that > > > > > > > > > > > > > between us, we can cover this. > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > > > > > > > > > > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some > > PoC-ing > > > > > > > > progress, but > > > > > > > > > > > > will > > > > > > > > > > > > > take 2 more months, I would estimate, before it is > > > fully > > > > > > > > available. > > > > > > > > > > > > > > > > > > > > > > > > > > If we want to be on the safe side with the > > > contribution, > > > > we > > > > > > > > should > > > > > > > > > > > > > contribute the source sooner and adjust it later. > > That > > > > > would > > > > > > > also > > > > > > > > > > help > > > > > > > > > > > us > > > > > > > > > > > > > in case things get crazy towards the 1.10 feature > > > freeze > > > > > and > > > > > > it > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > > hard to find time to review the new changes. > > > > > > > > > > > > > What would be the overhead of contributing now? > Given > > > > that > > > > > > the > > > > > > > > code > > > > > > > > > > is > > > > > > > > > > > > > already there, it looks like it would be only > review > > > > > > capacity, > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen < > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your attention and the promotion of > this > > > > work. > > > > > > > > > > > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible for > more > > > > > > specific > > > > > > > > > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not reached a > > > > > consensus. > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > I will explain all the functionalities of the > > > existing > > > > > > > > connector in > > > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to > > > > > continue > > > > > > > our > > > > > > > > > > > > > > discussions in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin < > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar > > > connector > > > > > in > > > > > > > > 1.10, > > > > > > > > > > how > > > > > > > > > > > > > about > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > do the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to > Flink > > > main > > > > > > repo > > > > > > > > as it > > > > > > > > > > > is a > > > > > > > > > > > > > new > > > > > > > > > > > > > > > public interface to Flink main repo. > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away > as > > > > there > > > > > is > > > > > > > no > > > > > > > > > > change > > > > > > > > > > > > to > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > sink interface so far. > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is > > > going > > > > to > > > > > > be > > > > > > > > code > > > > > > > > > > > > freeze > > > > > > > > > > > > > in > > > > > > > > > > > > > > > late Nov and let's say we give a month to the > > > > > development > > > > > > > and > > > > > > > > > > > review > > > > > > > > > > > > of > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by > late > > > > Oct. > > > > > > > There > > > > > > > > are > > > > > > > > > > > > still > > > > > > > > > > > > > 7 > > > > > > > > > > > > > > > weeks. Personally I think it is doable. If > > FLIP-27 > > > is > > > > > not > > > > > > > > ready > > > > > > > > > > by > > > > > > > > > > > > late > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar > connector > > > with > > > > > the > > > > > > > > > > existing > > > > > > > > > > > > > source > > > > > > > > > > > > > > > interface. This means we will have Pulsar > > connector > > > > in > > > > > > > Flink > > > > > > > > > > 1.10, > > > > > > > > > > > > > either > > > > > > > > > > > > > > > with or without FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink and > > source > > > > > > checked > > > > > > > > in > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > it might make sense to have two FLIPs, one for > > > Pulsar > > > > > > sink > > > > > > > > and > > > > > > > > > > > > another > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > Pulsar source. And we can start the work on > > Pulsar > > > > sink > > > > > > > right > > > > > > > > > > away. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo < > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the take from Flink community? Shall > we > > > wait > > > > > for > > > > > > > > FLIP-27 > > > > > > > > > > > or > > > > > > > > > > > > > > shall we > > > > > > > > > > > > > > > > proceed to next steps? And what the next > steps > > > are? > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li < > > > > > > > > bowenl...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in Flink > > can > > > > be a > > > > > > > good > > > > > > > > > > mutual > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > both communities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar > connector > > is > > > > the > > > > > > 1st > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > that integrates with Flink's metadata > > > management > > > > > > system > > > > > > > > and > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > It'll be cool to see how the integration > > turns > > > > out > > > > > > and > > > > > > > > > > whether > > > > > > > > > > > we > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are > > > currently > > > > in > > > > > > > > Beta, to > > > > > > > > > > > > cater > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of > > > > merging > > > > > > > > Pulsar > > > > > > > > > > > > > connector > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > Flink 1.10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, > e.g. > > > > maybe > > > > > > one > > > > > > > > for > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > source/sink functionalities and another for > > > > schema > > > > > > and > > > > > > > > > > catalog > > > > > > > > > > > > > > > > integration, > > > > > > > > > > > > > > > > > just to make them easier to review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for > FLIP-27. > > > But > > > > I > > > > > > > don't > > > > > > > > > > think > > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > > should be a blocker in cases where it > cannot > > > make > > > > > its > > > > > > > way > > > > > > > > > > into > > > > > > > > > > > > 1.10 > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time for > > > > > > committers > > > > > > > to > > > > > > > > > > > review > > > > > > > > > > > > or > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new > > > > interfaces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bowen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin < > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends on when the > > new > > > > > > source > > > > > > > > > > > interface > > > > > > > > > > > > is > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > be ready. Personally I think it would be > > > there > > > > in > > > > > > > > about a > > > > > > > > > > > month > > > > > > > > > > > > > or > > > > > > > > > > > > > > so. > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > I could be too optimistic. It would also > be > > > > good > > > > > to > > > > > > > > hear > > > > > > > > > > what > > > > > > > > > > > > do > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > and Stephan think as they are also > involved > > > in > > > > > > > FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general I think we should have Pulsar > > > > > connector > > > > > > in > > > > > > > > Flink > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > preferably with the new source interface. > > We > > > > can > > > > > > also > > > > > > > > check > > > > > > > > > > > it > > > > > > > > > > > > in > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > now > > > > > > > > > > > > > > > > > > with old source interface, but I suspect > > few > > > > > users > > > > > > > > will use > > > > > > > > > > > it > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > next official release. Therefore, it > seems > > > > > > reasonable > > > > > > > > to > > > > > > > > > > > wait a > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > to see whether we can jump to the new > > source > > > > > > > > interface. As > > > > > > > > > > > long > > > > > > > > > > > > > as > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little > > bit > > > > > > doesn't > > > > > > > > seem > > > > > > > > > > to > > > > > > > > > > > > hurt > > > > > > > > > > > > > > much. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till > > Rohrmann > > > < > > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem would be > > if > > > we > > > > > > > > committed > > > > > > > > > > the > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > connector before the new source > interface > > > is > > > > > > ready. > > > > > > > > If I > > > > > > > > > > > > > > understood > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > correctly, then we need to support the > > old > > > > > source > > > > > > > > > > interface > > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > existing connectors. By checking it in > > > early > > > > I > > > > > > > could > > > > > > > > see > > > > > > > > > > > the > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > our users could start using the > connector > > > > > > earlier. > > > > > > > > > > > Moreover, > > > > > > > > > > > > it > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration is > > > being > > > > > > > delayed > > > > > > > > in > > > > > > > > > > > case > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > source interface should be delayed. The > > > only > > > > > > > > downside I > > > > > > > > > > see > > > > > > > > > > > > is > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > review effort and potential fixes which > > > might > > > > > be > > > > > > > > > > irrelevant > > > > > > > > > > > > for > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > source interface implementation. I > guess > > it > > > > > > mainly > > > > > > > > > > depends > > > > > > > > > > > on > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > certain > > > > > > > > > > > > > > > > > > > we are when the new source interface > will > > > be > > > > > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket > > Qin < > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some update on > > FLIP-27. > > > > > > > Although > > > > > > > > the > > > > > > > > > > > FLIP > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > discussion thread has been quiet for > > some > > > > > > time, a > > > > > > > > few > > > > > > > > > > > > > > committer / > > > > > > > > > > > > > > > > > > > > contributors in Flink community were > > > > actually > > > > > > > > > > prototyping > > > > > > > > > > > > the > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > We have made some good progress there > > but > > > > > want > > > > > > to > > > > > > > > > > update > > > > > > > > > > > > the > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > after the entire thing is verified to > > > work > > > > in > > > > > > > case > > > > > > > > > > there > > > > > > > > > > > > are > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > > > > > minute surprise in the > implementation. > > I > > > > > don't > > > > > > > > have an > > > > > > > > > > > > exact > > > > > > > > > > > > > > ETA > > > > > > > > > > > > > > > > yet, > > > > > > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > > > > > guess it is going to be within a > month > > or > > > > so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the current > Flink > > > > Pulsar > > > > > > > > connector > > > > > > > > > > > and > > > > > > > > > > > > > > see if > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to > > avoid > > > > the > > > > > > > case > > > > > > > > that > > > > > > > > > > > we > > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > Pulsar connector with some review > > efforts > > > > and > > > > > > > > shortly > > > > > > > > > > > after > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > Source interface is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie > > > Shen < > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and > > > > > suggestions! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the > > > connector > > > > > has > > > > > > > > always > > > > > > > > > > > been > > > > > > > > > > > > to > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > users with the latest features of > > both > > > > > > systems > > > > > > > as > > > > > > > > > > soon > > > > > > > > > > > as > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > propose to contribute the connector > > to > > > > > Flink > > > > > > > and > > > > > > > > hope > > > > > > > > > > > to > > > > > > > > > > > > > get > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink > > > > experts > > > > > > to > > > > > > > > ensure > > > > > > > > > > > the > > > > > > > > > > > > > > high > > > > > > > > > > > > > > > > > > quality > > > > > > > > > > > > > > > > > > > > > of the connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its > existence > > > at > > > > > the > > > > > > > > > > beginning > > > > > > > > > > > of > > > > > > > > > > > > > > > > reworking > > > > > > > > > > > > > > > > > > > > > the connector implementation based > on > > > > Flink > > > > > > > 1.9; > > > > > > > > we > > > > > > > > > > > also > > > > > > > > > > > > > > wanted > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > build a connector that supports > both > > > > batch > > > > > > and > > > > > > > > stream > > > > > > > > > > > > > > computing > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > on it. > > > > > > > > > > > > > > > > > > > > > However, it has been inactive for > > some > > > > > time, > > > > > > so > > > > > > > > we > > > > > > > > > > > > decided > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > a connector with most of the new > > > > features, > > > > > > such > > > > > > > > as > > > > > > > > > > the > > > > > > > > > > > > new > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > and the new catalog API first. We > > will > > > > pay > > > > > > > > attention > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > progress > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and > > incorporate > > > it > > > > > > with > > > > > > > > the > > > > > > > > > > > > > connector > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > > > > > > as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of the > > > > connector, > > > > > > we > > > > > > > > are > > > > > > > > > > > > > following > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink > repository > > > and > > > > > > aimed > > > > > > > to > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > throughout > > > > > > > > > > > > > > > > > > > > > tests as we could. We are also > happy > > to > > > > > hear > > > > > > > > > > > suggestions > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > supervision from the Flink > community > > to > > > > > > improve > > > > > > > > the > > > > > > > > > > > > > > stability and > > > > > > > > > > > > > > > > > > > > > performance of the connector > > > > continuously. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM > Sijie > > > Guo > > > > < > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments > > and > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the main > > question > > > > > here > > > > > > is > > > > > > > > > > about - > > > > > > > > > > > > > "how > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > community maintain the > connector?". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from > myself. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to host > > this > > > > > > > > integration > > > > > > > > > > is > > > > > > > > > > > > kind > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > less > > > > > > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > > > > > > > here. I believe there can be many > > > ways > > > > to > > > > > > > > achieve > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > As part of the contribution, what > > we > > > > are > > > > > > > > looking > > > > > > > > > > for > > > > > > > > > > > > here > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > communities can build the > > > collaboration > > > > > > > > > > relationship > > > > > > > > > > > on > > > > > > > > > > > > > > > > > developing > > > > > > > > > > > > > > > > > > > > > > the integration between Pulsar > and > > > > Flink. > > > > > > > Even > > > > > > > > we > > > > > > > > > > can > > > > > > > > > > > > try > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > catch > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink > > > community. > > > > We > > > > > > are > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > facing the fact that we have less > > > > > > experiences > > > > > > > > in > > > > > > > > > > > Flink > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > folks > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > community. In order to make sure > we > > > > > > maintain > > > > > > > > and > > > > > > > > > > > > deliver > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink > > > integration > > > > > to > > > > > > > the > > > > > > > > > > users > > > > > > > > > > > > who > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > technologies, we need some help > > from > > > > the > > > > > > > > experts > > > > > > > > > > from > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 > > > for a > > > > > > > while. > > > > > > > > > > > > Originally > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > > > > > thinking > > > > > > > > > > > > > > > > > > > > > > of contributing the connectors > back > > > > after > > > > > > > > > > integrating > > > > > > > > > > > > > with > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. > But > > we > > > > > > decided > > > > > > > > to > > > > > > > > > > > > initiate > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > conversation > > > > > > > > > > > > > > > > > > > > > > as early as possible. Because we > > > > believe > > > > > > > there > > > > > > > > are > > > > > > > > > > > more > > > > > > > > > > > > > > > > benefits > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > > > it now rather than later. As part > > of > > > > > > > > contribution, > > > > > > > > > > it > > > > > > > > > > > > can > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > community understand more about > > > Pulsar > > > > > and > > > > > > > the > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > points. > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink > > community > > > > > > verify > > > > > > > > the > > > > > > > > > > new > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog > > API). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM > > Becket > > > > > Qin < > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in > > > > contributing > > > > > > the > > > > > > > > > > Pulsar > > > > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think having > Pulsar > > > > > > connector > > > > > > > > with > > > > > > > > > > > > strong > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So > I > > am > > > > > happy > > > > > > > the > > > > > > > > > > > > shepherd > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to > > > > provide > > > > > > > some > > > > > > > > > > > context > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > recent > > > > > > > > > > > > > > > > > > > > > efforts on > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink maintains > > its > > > > > > > > connector has > > > > > > > > > > > hit > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > > > > > > bar. > > > > > > > > > > > > > > > > > > > > > > > With more and more connectors > > > coming > > > > > into > > > > > > > > Flink > > > > > > > > > > > repo, > > > > > > > > > > > > > we > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > facing a > > > > > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > > problems such as long build and > > > > testing > > > > > > > > time. To > > > > > > > > > > > > > address > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > problem, > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the > > following: > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors > into > > a > > > > > > separate > > > > > > > > > > > > repository. > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > temporarily > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential > solution > > > to > > > > > > > shorten > > > > > > > > the > > > > > > > > > > > > build > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to > > stay > > > > as > > > > > > > > ecosystem > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > tries > > > > > > > > > > > > > > > > > > > > > > > to provide good support for > > > > > functionality > > > > > > > and > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > tests. > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink > > > > Ecosystem > > > > > > > > project > > > > > > > > > > > > website > > > > > > > > > > > > > > and it > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > > through some final approval > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it > would > > > be > > > > > > great > > > > > > > to > > > > > > > > > > first > > > > > > > > > > > > see > > > > > > > > > > > > > > if we > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an > ecosystem > > > > > project > > > > > > > with > > > > > > > > > > great > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink > Pulsar > > > > > > connector > > > > > > > > is > > > > > > > > > > > tested > > > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > we can learn something to > > maintain > > > it > > > > > as > > > > > > an > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If > the > > > > > quality > > > > > > > as > > > > > > > > an > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well > > adopt > > > it > > > > > > into > > > > > > > > the > > > > > > > > > > main > > > > > > > > > > > > > repo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is > > > > FLIP-27 > > > > > > > where > > > > > > > > we > > > > > > > > > > are > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > Flink source connector > > architecture > > > > and > > > > > > > > > > interface. > > > > > > > > > > > > This > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing > > > wise, > > > > if > > > > > > we > > > > > > > > are > > > > > > > > > > > going > > > > > > > > > > > > to > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am > > > > wondering > > > > > if > > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > > hold > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt > to > > > the > > > > > new > > > > > > > > > > interface > > > > > > > > > > > > to > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM > > > > Chesnay > > > > > > > > Schepler < > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may > > end > > > > up > > > > > > > > repeating > > > > > > > > > > > > > history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts > > at > > > > > > > > contributing a > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > of which failed because no > > > > committer > > > > > > was > > > > > > > > > > getting > > > > > > > > > > > > > > involved, > > > > > > > > > > > > > > > > > > > despite > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a > dedicated > > > > > > > discussion > > > > > > > > > > thread > > > > > > > > > > > > > > about the > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting > several > > > +1's > > > > > > from > > > > > > > > > > > > committers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make sure > that > > > if > > > > we > > > > > > > > > > > > welcome/approve > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > contribution it will actually > > get > > > > the > > > > > > > > attention > > > > > > > > > > > it > > > > > > > > > > > > > > > > deserves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to > > > recommend > > > > > > > > maintaining > > > > > > > > > > > the > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it > from > > > the > > > > > > > > > > documentation > > > > > > > > > > > > to > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for > > > sharing > > > > > > > > artifacts > > > > > > > > > > > among > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > (what's > > > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?), > > this > > > > may > > > > > > be a > > > > > > > > > > better > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till > > > Rohrmann > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting > > this > > > > > > > discussion > > > > > > > > > > > Yijie. > > > > > > > > > > > > I > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a very > > > > valuable > > > > > > > > addition > > > > > > > > > > > since > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it would > > > further > > > > > > > expand > > > > > > > > > > > Flink's > > > > > > > > > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > Also > > > > > > > > > > > > > > > > > > > > > > > > > from a project perspective > it > > > > makes > > > > > > > > sense for > > > > > > > > > > > me > > > > > > > > > > > > to > > > > > > > > > > > > > > place > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > the downstream project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main concern/question is > > how > > > > can > > > > > > the > > > > > > > > Flink > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in > > the > > > > past > > > > > > > that > > > > > > > > > > > > connectors > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > actively developed > components > > > > > because > > > > > > > > they > > > > > > > > > > need > > > > > > > > > > > > to > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > kept > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > sync > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > external system and with > > Flink. > > > > > Given > > > > > > > > that > > > > > > > > > > the > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > willing > > > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining, > > > > improving > > > > > > and > > > > > > > > > > > evolving > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can > > achieve > > > > > this. > > > > > > > > Hence, > > > > > > > > > > +1 > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 > > AM > > > > > Sijie > > > > > > > Guo > > > > > > > > < > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main > driver > > > > behind > > > > > > > > > > FLINK-9641 > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > FLINK-9168, > > > > > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > > try to > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on this. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 > > was > > > > > > created > > > > > > > > for > > > > > > > > > > > > bringing > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The > > > integration > > > > > was > > > > > > > done > > > > > > > > > > with > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > 1.6.0. > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > sent out > > > > > > > > > > > > > > > > > > > > > > > > pull > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago > > and > > > we > > > > > > ended > > > > > > > > up > > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > > those > > > > > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to > > use > > > > > Flink > > > > > > > to > > > > > > > > > > > process > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > > > > > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty > > > simple > > > > > and > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > no > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have > > > made a > > > > > lot > > > > > > > of > > > > > > > > > > > changes > > > > > > > > > > > > in > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > brought > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the > > > first-class > > > > > > > > citizen in > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > integrated > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > >> other computing engines > for > > > > > > processing > > > > > > > > > > Pulsar > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how > to > > > > > > integrate > > > > > > > > with > > > > > > > > > > > Flink > > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > way. > > > > > > > > > > > > > > > > > > > > > Then > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the > pulsar-flink > > > > > > > connectors > > > > > > > > from > > > > > > > > > > > the > > > > > > > > > > > > > > ground > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and > catalog > > > API > > > > as > > > > > > the > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > citizen > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that > being > > > > said, > > > > > > in > > > > > > > > the > > > > > > > > > > new > > > > > > > > > > > > > > > > pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar > as a > > > > flink > > > > > > > > catalog > > > > > > > > > > and > > > > > > > > > > > > > > query / > > > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example about > how > > > to > > > > > use > > > > > > > > Pulsar > > > > > > > > > > as > > > > > > > > > > > a > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > catalog: > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a > > blog > > > > post > > > > > > > > > > explaining > > > > > > > > > > > > why > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > re-implement > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 > and > > > > what > > > > > > are > > > > > > > > the > > > > > > > > > > > > changes > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > made > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not > > just > > > a > > > > > > simple > > > > > > > > data > > > > > > > > > > > sink > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully > > > > integrated > > > > > > > > streaming > > > > > > > > > > > > data > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > Flink in > > > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source, > > > > > schema/catalog > > > > > > > and > > > > > > > > > > > state). > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > combination > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a > > great > > > > > > > streaming > > > > > > > > > > > > warehouse > > > > > > > > > > > > > > > > > > architecture > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified > > data > > > > > > > > processing. > > > > > > > > > > > Since > > > > > > > > > > > > we > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > talking > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar > > integration > > > to > > > > > > Flink > > > > > > > > here, > > > > > > > > > > > we > > > > > > > > > > > > > are > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > dedicated to > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and > evolve > > > the > > > > > > > > integration > > > > > > > > > > > > with > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit > > more > > > > > > > background > > > > > > > > > > about > > > > > > > > > > > > the > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know > > what > > > > are > > > > > > your > > > > > > > > > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at > 11:54 > > > AM > > > > > Yun > > > > > > > > Tang < > > > > > > > > > > > > > > > > > myas...@live.com> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar > > becomes > > > > > more > > > > > > > and > > > > > > > > more > > > > > > > > > > > > > popular > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people > willing > > to > > > > > > > > contribute to > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further > > > discussion, > > > > > > would > > > > > > > > you > > > > > > > > > > > please > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > explanation > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship between > > this > > > > > > thread > > > > > > > to > > > > > > > > > > > current > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > JIRAs > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2] > > > > > connector? > > > > > > > > Will the > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > contains > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally > > > different > > > > > > > > > > > implementation? > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen < > > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September > 3, > > > > 2019 > > > > > > > 13:57 > > > > > > > > > > > > > > > > > > > > > > > > >>> To: dev@flink.apache.org > < > > > > > > > > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] > > Contribute > > > > > > Pulsar > > > > > > > > Flink > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community! > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the > > > > > discussion > > > > > > > of > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction > to > > > > Apache > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a > > > > > multi-tenant, > > > > > > > > > > > > > > high-performance > > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. > > > > Pulsar > > > > > > > > includes > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for > multiple > > > > > > clusters > > > > > > > > in a > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication > of > > > > > > messages > > > > > > > > across > > > > > > > > > > > > > > clusters, > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > low > > > > > > > > > > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, > > > > seamless > > > > > > > > > > scalability > > > > > > > > > > > to > > > > > > > > > > > > > > over a > > > > > > > > > > > > > > > > > > > million > > > > > > > > > > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message > > > delivery > > > > > > with > > > > > > > > > > > persistent > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache > > > BookKeeper. > > > > > > > > Nowadays, > > > > > > > > > > > Pulsar > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > adopted > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more > companies[2]. > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar > > Flink > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink > connector > > we > > > > are > > > > > > > > planning > > > > > > > > > > to > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > built > > > > > > > > > > > > > > > > > > > > > > > upon > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar > > 2.4.0. > > > > The > > > > > > > main > > > > > > > > > > > features > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming > > > source > > > > > with > > > > > > > > > > > > exactly-once > > > > > > > > > > > > > > > > > guarantee. > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results > to > > > > > Pulsar > > > > > > > with > > > > > > > > > > > > > > at-least-once > > > > > > > > > > > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > > (We > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to > > > > exactly-once > > > > > > as > > > > > > > > well > > > > > > > > > > > when > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features > ready > > in > > > > its > > > > > > > 2.5.0 > > > > > > > > > > > > version) > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new > > Table > > > > API > > > > > > Type > > > > > > > > > > system > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically > (de)serialize > > > > > > messages > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > > > help > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink > new > > > > > Catalog > > > > > > > API > > > > > > > > > > > > > > (FLIP-30[4]), > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as > > > tables > > > > in > > > > > > > > Table API > > > > > > > > > > > as > > > > > > > > > > > > > > well as > > > > > > > > > > > > > > > > > SQL > > > > > > > > > > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > https://pulsar.apache.org/ > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
