Hi Stephan, I have no doubt about the value of adding Pulsar connector to Flink repo. My concern is about how exactly we are going to do it.
As mentioned before, I believe that we can handle connectors more > pragmatically and less strict than the core of Flink, if it helps unlocking > users. I can see the benefit of being less restrict for the initial connector code adoption. However, I don't think we should be less restrict on the maintenance commitment once the code is in Flink repo. It only makes sense to check in something and ask users to use if we plan to maintain it. If I understand correctly, the current plan so far is following: 1. release 1.10 - Check in Pulsar connector on old interface and label it as beta version. - encourage users to try it and report bugs. 2. release 1.11 - Check in Pulsar connector on new interface (a.k.a new Pulsar connector) and label it as beta version - Deprecate the old Pulsar connector - Fix bugs reported on old Pulsar connector from release 1.10 - Ask users to migrate from old Pulsar connector to new Pulsar connector 3. release 1.12 - Announce end of support for old Pulsar connector and remove the code - Fix bugs reported on new Pulsar connector. If this is the plan, it seems neither Flink nor the users trying the old Pulsar connector will benefit from this experimental old Pulsar connector, because whatever feedbacks we got or bugs we fix on the old Pulsar connector are immediately thrown away in one or two releases. If we check in the old Pulsar connector right now, the only option I see is to maintain it for a while (e.g. a year or more). IMO, the immediate deprecation and code removal hurts the users much more than asking them to wait for another release. I personally think that we can avoid this maintenance burden by going directly to the new Pulsar connector, especially given that users can still use the connector even if they are not in Flink repo. That said, I am OK with maintaining both old and new Pulsar connector if we believe that having the Pulsar connector available right now in Flink repo is more important. Thanks, Jiangjie (Becket) Qin On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <se...@apache.org> wrote: > My assumption is as Sijie's, that once the connector is either part of > Flink, or part of the streamnative repo. No double maintenance. > > I feel this discussion is very much caught in problems that are all > solvable if we want to solve them. > Maybe we can think what our goal for users and the communities is? > > - Do we want to help build a relationship between the Pulsar and Flink > open source communities? > - Will users find a connector in the streamnative repository? > - Will users trust a connector that is not part of Flink as much? > > And then decide what is best according to the overall goals there. > As mentioned before, I believe that we can handle connectors more > pragmatically and less strict than the core of Flink, if it helps unlocking > users. > > Best, > Stephan > > > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> wrote: > > > Thanks Becket. > > > > I think it is better for the Flink community to judge the benefits of > doing > > this. I was trying to provide some views from outsiders. > > > > Thanks, > > Sijie > > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <becket....@gmail.com> > wrote: > > > > > Hi Sijie, > > > > > > Yes, we will have to support existing old connectors and new connectors > > in > > > parallel for a while. We have to take that maintenance overhead because > > > existing connectors have been used by the users for a long time. I > guess > > It > > > may take at least a year for us to fully remove the old connectors. > > > > > > Process wise, we can do the same for Pulsar connector. But I am not > sure > > if > > > we want to have the same burden on Pulsar connector, and I would like > to > > > understand the benefit of doing that. > > > > > > For users, the benefit of having the old Pulsar connector checked in > > seems > > > limited because 1) that code base will be immediately deprecated in the > > > next release in 3-4 months; 2) users can always use it even if it is > not > > in > > > the Flink code base. Admittedly it is not as convenient as having it in > > > Flink code base, but doesn't seem super either. And after 3-4 months, > > users > > > can just use the new connector in Flink repo. > > > > > > For Flink developers, the old connector code base is not something that > > we > > > want to evolve later. Instead, these code will be deprecated and > > > removed. So why do we want to get a beta version out to attract people > to > > > use something we don't want to maintain? > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > > Thanks everyone here. Sorry for jumping into the discussion here. > > > > > > > > I am not very familiar about the deprecation process in Flink. If I > > > > misunderstood the process, please fix me. > > > > > > > > As far as I understand, FLIP-27 is introducing a new unified API for > > > > connectors. After it introduces the new API > > > > and before moving all the existing connectors from old API to new > API, > > > both > > > > old ApI and new API will co-exist > > > > for a while until Flink moves all existing connectors to new API. So > > the > > > > Pulsar connector (using old API) can > > > > follow the deprecation process with other connector using old API and > > the > > > > deprecation of old API, no? > > > > > > > > If that's the case, I think contributing the current connector back > to > > > > Flink rather than maintaining it outside Flink > > > > would provide a bit more benefits. We can deprecate the existing > > > > streamnative/pulsar-flink repo and point the users > > > > to use the connector in Flink repo. So all the review processes will > > > happen > > > > within Flink for both old connector and > > > > new connector. It also reduces the confusions for the users as the > > > > documentation and code base happen in one place. > > > > > > > > Thoughts? > > > > > > > > - Sijie > > > > > > > > > > > > > > > > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <becket....@gmail.com> > > > wrote: > > > > > > > > > Thanks for the explanation, Stephan. I have a few questions / > > thoughts. > > > > > > > > > > So that means we will remove the old connector without a major > > version > > > > > bump, is that correct? > > > > > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors > > will > > > > > always work because we saw some dependency class collisions in the > > > past. > > > > To > > > > > make it safe we may have to maintain the old code for one more > > release. > > > > > > > > > > To be honest I am still wondering if we have to put the old > connector > > > in > > > > > Flink repo. if we check in the old connector to Flink. We will end > up > > > in > > > > > the following situation: > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > > 2. Old connector in Flink Repo, which may be different from the one > > in > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12) > > > > > 3. New connector in Flink Repo. > > > > > > > > > > We need to think about how to make the users in each case happy. > > > > > - For users of (1), I assume Sijie and Yijie will have to maintain > > the > > > > code > > > > > a bit longer for its own compatibility even after we have (2). In > > that > > > > > case, bugs found in old connector may or may not need to be fixed > in > > > both > > > > > Flink and the streamnative/pulsar-flink repo. > > > > > - For users of (2), will we provide bug fixes? If we do, it will > be a > > > > > little awkward because those bug fixes will be immediately > deprecated > > > in > > > > > 1.11, and removed in 1.12. So we are essentially asking users to > > > migrate > > > > > away from the bug fix. After Flink 1.12, users may still have to > > switch > > > > to > > > > > use (3) due to the potential dependency class conflicts mentioned > > > above. > > > > > - Users of (3) have a much easier life and don't need to worry too > > > much. > > > > > > > > > > The above story seems a little complicated to tell. I think it will > > be > > > > much > > > > > easier to not have (2) at all. > > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > > 3. New connector in Flink Repo. > > > > > > > > > > - Old connector will only be maintained in > streamnative/pulsar-flink > > > repo > > > > > until it is fully deprecated. Users can always use the existing > > Pulsar > > > > > connector in that repo. > > > > > - New connector will be in Flink repo and maintained like the other > > > > > connectors. > > > > > > > > > > This seems much simpler for users understand and they will not be > > > blocked > > > > > from using the old connector. If the concern is about the quality > of > > > the > > > > > connector in streamnative/pulsar-flink repo, is it enough for us > just > > > to > > > > > review the code in streamnative/pulsar-flink connector to make sure > > it > > > > > looks good from Flink's perspective? > > > > > > > > > > What do you think? > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <se...@apache.org> > > wrote: > > > > > > > > > > > My take would be the following: > > > > > > > > > > > > - If we merge the connector now and replace it with a FLIP-27 > > > version > > > > > > before the 1.10 release, then we need no deprecation process > > > > > > - If we don't manage to replace it with a FLIP-27 version > before > > > the > > > > > 1.10 > > > > > > release, than it is good that we have the other version, so no > > users > > > > get > > > > > > blocked. > > > > > > > > > > > > In the latter case we can see how we want to do it. Immediate > > removal > > > > of > > > > > > the old version or deprecation label and keeping it for one more > > > > release. > > > > > > Given that you should be able to use a Flink 1.10 connector with > > > Flink > > > > > 1.11 > > > > > > as well (stable public APIs) there is also a workaround if you > need > > > an > > > > > old > > > > > > connector in a newer version. So immediate removal might even be > > > > > feasible. > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin < > becket....@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > Thanks for the clarification. I completely agree with you and > > > Thomas > > > > on > > > > > > the > > > > > > > process of adding connectors to Flink repo. However, I am > > wondering > > > > > what > > > > > > is > > > > > > > the deprecation process? Given the main concern here was that > we > > > may > > > > > have > > > > > > > to maintain two Pulsar connector code bases until the old one > is > > > > > removed > > > > > > > from the repo, it would be good to know how long we have to do > > > that. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <se...@apache.org > > > > > > wrote: > > > > > > > > > > > > > > > Some quick thoughts on the connector contribution process. I > > > > > basically > > > > > > > > reiterate here what Thomas mentioned in another thread about > > the > > > > > > Kinesis > > > > > > > > connector. > > > > > > > > > > > > > > > > For connectors, we should favor a low-overhead contribution > > > > process, > > > > > > and > > > > > > > > accept user code and changes more readily than in the core > > > system. > > > > > > > > That is because connectors have both a big variety of > scenarios > > > > they > > > > > > get > > > > > > > > used in (only through use and many small contributions do > they > > > > become > > > > > > > > really useful over time) and at the same time, and committers > > do > > > > not > > > > > > use > > > > > > > > the connector themselves and usually cannot foresee too well > > what > > > > is > > > > > > > > needed. > > > > > > > > > > > > > > > > Further more, a missing connector (or connector feature) is > > > often a > > > > > > > bigger > > > > > > > > show stopper for users than a missing API or system feature. > > > > > > > > > > > > > > > > Along these lines of thougt, the conclusion would be to take > > the > > > > > Pulsar > > > > > > > > connector now, focus the review on legal/dependencies/rough > > code > > > > > style > > > > > > > and > > > > > > > > conventions, label it as "beta" (in the sense of "new code" > > that > > > is > > > > > > "not > > > > > > > > yet tested through longer use") and go ahead. And then evolve > > it > > > > > > quickly > > > > > > > > without putting formal blockers in the way, meaning also > > adding a > > > > new > > > > > > > FLIP > > > > > > > > 27 version when it is there. > > > > > > > > > > > > > > > > Best, > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < > > becket....@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > Could you please follow the FLIP process to start a new > FLIP > > > > > > > [DISCUSSION] > > > > > > > > > thread in the mailing list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > > > > > > > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So > > there > > > > is a > > > > > > > FLIP > > > > > > > > > number collision. Can you change the FLIP number to 72? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > > > walter...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > > > > > > > > Would you mind enabling comments/suggestions on the > google > > > doc > > > > > > link? > > > > > > > > This > > > > > > > > > > way the contributors from the community can comment on > the > > > doc. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > > > I've drafted a FLIP that describes the current design > of > > > the > > > > > > Pulsar > > > > > > > > > > > connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > > > > > > > > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > > > > walter...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > Sorry for joining the discussion late and thanks > Yijie > > & > > > > > Sijie > > > > > > > for > > > > > > > > > > > driving > > > > > > > > > > > > the discussion. > > > > > > > > > > > > I also think the Pulsar connector would be a very > > > valuable > > > > > > > addition > > > > > > > > > to > > > > > > > > > > > > Flink. I can also help out a bit on the review side > :-) > > > > > > > > > > > > > > > > > > > > > > > > Regarding the timeline, I also share concerns with > > Becket > > > > on > > > > > > the > > > > > > > > > > > > relationship between the new Pulsar connector and > > > FLIP-27. > > > > > > > > > > > > There's also another discussion just started by > Stephan > > > on > > > > > > > dropping > > > > > > > > > > Kafka > > > > > > > > > > > > 9/10 support on next Flink release [1]. Although the > > > > > situation > > > > > > > is > > > > > > > > > > > somewhat > > > > > > > > > > > > different, and Kafka 9/10 connector has been in Flink > > for > > > > > > almost > > > > > > > > 3-4 > > > > > > > > > > > years, > > > > > > > > > > > > based on the discussion I am not sure if a major > > version > > > > > > release > > > > > > > > is a > > > > > > > > > > > > requirement for removing old connector supports. > > > > > > > > > > > > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree the > > old > > > > > > > connector > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is there. > > As > > > > > > Stephan > > > > > > > > > > stated, > > > > > > > > > > > it > > > > > > > > > > > > is easier to contribute the source sooner and adjust > it > > > > > later. > > > > > > > > > > > > We should also ensure we clearly communicate the > > message: > > > > for > > > > > > > > > example, > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27 > > connector > > > > page > > > > > > of > > > > > > > > the > > > > > > > > > > > > website, documentations, etc. Any other thoughts? > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Technically speaking, removing the old connector > code > > > is > > > > a > > > > > > > > > backwards > > > > > > > > > > > > > incompatible change which requires a major version > > > bump, > > > > > i.e. > > > > > > > > Flink > > > > > > > > > > > 2.x. > > > > > > > > > > > > > Given that we don't have a clear plan on when to > have > > > the > > > > > > next > > > > > > > > > major > > > > > > > > > > > > > version release, it seems unclear how long the old > > > > > connector > > > > > > > code > > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > > there if we check it in right now. Or will we > remove > > > the > > > > > old > > > > > > > > > > connector > > > > > > > > > > > > > without a major version bump? In any case, it > sounds > > > not > > > > > > quite > > > > > > > > user > > > > > > > > > > > > > friendly to the those who might use the old Pulsar > > > > > > connector. I > > > > > > > > am > > > > > > > > > > not > > > > > > > > > > > sure > > > > > > > > > > > > > if it is worth these potential problems in order to > > > have > > > > > the > > > > > > > > Pulsar > > > > > > > > > > > source > > > > > > > > > > > > > connector checked in one or two months earlier. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen < > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Agreed, if we check in the old code, we should > make > > > it > > > > > > clear > > > > > > > > that > > > > > > > > > > it > > > > > > > > > > > will > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based version > of > > > the > > > > > > > > connector > > > > > > > > > is > > > > > > > > > > > > > there. > > > > > > > > > > > > > > We should not commit to maintaining the old > > version, > > > > that > > > > > > > would > > > > > > > > > be > > > > > > > > > > > indeed > > > > > > > > > > > > > > too much overhead. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin < > > > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, the overhead would just be review > capacity. > > In > > > > > > fact, I > > > > > > > > am > > > > > > > > > > not > > > > > > > > > > > > > > worrying > > > > > > > > > > > > > > > too much about the review capacity. That is > just > > a > > > > one > > > > > > time > > > > > > > > > cost. > > > > > > > > > > > My > > > > > > > > > > > > > > > concern is mainly about the long term burden. > > > Assume > > > > we > > > > > > > have > > > > > > > > > new > > > > > > > > > > > source > > > > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar > > > > > > connectors > > > > > > > in > > > > > > > > > old > > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new > > > > source > > > > > > > > > interface, > > > > > > > > > > > the > > > > > > > > > > > > > old > > > > > > > > > > > > > > > Pulsar connector might be deprecated almost > > > > immediately > > > > > > > after > > > > > > > > > > > checked > > > > > > > > > > > > > in, > > > > > > > > > > > > > > > but we may still have to maintain two code > bases. > > > For > > > > > the > > > > > > > > > > existing > > > > > > > > > > > > > > > connectors, we have to do that anyways. But it > > > would > > > > be > > > > > > > good > > > > > > > > to > > > > > > > > > > > avoid > > > > > > > > > > > > > > > introducing a new connector with the same > > problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen < > > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice to see this lively discussion about the > > > Pulsar > > > > > > > > > connector. > > > > > > > > > > > > > > > > Some thoughts on the open questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a > > community > > > > > > package > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like the discussion is more going > towards > > > > > > > > > contribution. I > > > > > > > > > > > think > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > is good, especially if we think that we want > to > > > > > build a > > > > > > > > > > similarly > > > > > > > > > > > > > deep > > > > > > > > > > > > > > > > integration with Pulsar as we have for > example > > > with > > > > > > > Kafka. > > > > > > > > > The > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > already looks like a more thorough connector > > than > > > > > many > > > > > > > > others > > > > > > > > > > we > > > > > > > > > > > have > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > the repository. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > With either a repo split, or the new build > > > system, > > > > I > > > > > > hope > > > > > > > > > that > > > > > > > > > > > the > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > overhead is not a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Committer Support > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket offered some help already, I can also > > > help a > > > > > > bit. > > > > > > > I > > > > > > > > > hope > > > > > > > > > > > that > > > > > > > > > > > > > > > > between us, we can cover this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making > some > > > > > PoC-ing > > > > > > > > > > > progress, but > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > take 2 more months, I would estimate, before > it > > > is > > > > > > fully > > > > > > > > > > > available. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we want to be on the safe side with the > > > > > > contribution, > > > > > > > we > > > > > > > > > > > should > > > > > > > > > > > > > > > > contribute the source sooner and adjust it > > later. > > > > > That > > > > > > > > would > > > > > > > > > > also > > > > > > > > > > > > > help > > > > > > > > > > > > > > us > > > > > > > > > > > > > > > > in case things get crazy towards the 1.10 > > feature > > > > > > freeze > > > > > > > > and > > > > > > > > > it > > > > > > > > > > > would > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > hard to find time to review the new changes. > > > > > > > > > > > > > > > > What would be the overhead of contributing > now? > > > > Given > > > > > > > that > > > > > > > > > the > > > > > > > > > > > code > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > already there, it looks like it would be only > > > > review > > > > > > > > > capacity, > > > > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen < > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your attention and the promotion > > of > > > > this > > > > > > > work. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible > > for > > > > more > > > > > > > > > specific > > > > > > > > > > > > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not > > reached > > > a > > > > > > > > consensus. > > > > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > > > > I will explain all the functionalities of > the > > > > > > existing > > > > > > > > > > > connector in > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and > > Catalog) > > > to > > > > > > > > continue > > > > > > > > > > our > > > > > > > > > > > > > > > > > discussions in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin > < > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have > Pulsar > > > > > > connector > > > > > > > > in > > > > > > > > > > > 1.10, > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > do the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector > to > > > > Flink > > > > > > main > > > > > > > > > repo > > > > > > > > > > > as it > > > > > > > > > > > > > > is a > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > public interface to Flink main repo. > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right > > away > > > > as > > > > > > > there > > > > > > > > is > > > > > > > > > > no > > > > > > > > > > > > > change > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > sink interface so far. > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink > 1.10 > > > is > > > > > > going > > > > > > > to > > > > > > > > > be > > > > > > > > > > > code > > > > > > > > > > > > > > > freeze > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > late Nov and let's say we give a month to > > the > > > > > > > > development > > > > > > > > > > and > > > > > > > > > > > > > > review > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 > > by > > > > late > > > > > > > Oct. > > > > > > > > > > There > > > > > > > > > > > are > > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > > 7 > > > > > > > > > > > > > > > > > > weeks. Personally I think it is doable. > If > > > > > FLIP-27 > > > > > > is > > > > > > > > not > > > > > > > > > > > ready > > > > > > > > > > > > > by > > > > > > > > > > > > > > > late > > > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar > > > > connector > > > > > > with > > > > > > > > the > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > interface. This means we will have Pulsar > > > > > connector > > > > > > > in > > > > > > > > > > Flink > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > either > > > > > > > > > > > > > > > > > > with or without FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink > > and > > > > > source > > > > > > > > > checked > > > > > > > > > > > in > > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > > it might make sense to have two FLIPs, > one > > > for > > > > > > Pulsar > > > > > > > > > sink > > > > > > > > > > > and > > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > Pulsar source. And we can start the work > on > > > > > Pulsar > > > > > > > sink > > > > > > > > > > right > > > > > > > > > > > > > away. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo > < > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the take from Flink community? > > Shall > > > > we > > > > > > wait > > > > > > > > for > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > shall we > > > > > > > > > > > > > > > > > > > proceed to next steps? And what the > next > > > > steps > > > > > > are? > > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen > Li < > > > > > > > > > > > bowenl...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in > > > Flink > > > > > can > > > > > > > be a > > > > > > > > > > good > > > > > > > > > > > > > mutual > > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > both communities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar > > > > connector > > > > > is > > > > > > > the > > > > > > > > > 1st > > > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > that integrates with Flink's metadata > > > > > > management > > > > > > > > > system > > > > > > > > > > > and > > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the > > integration > > > > > turns > > > > > > > out > > > > > > > > > and > > > > > > > > > > > > > whether > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which > are > > > > > > currently > > > > > > > in > > > > > > > > > > > Beta, to > > > > > > > > > > > > > > > cater > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in > > favor > > > of > > > > > > > merging > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > > Flink 1.10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized > > PRs, > > > > e.g. > > > > > > > maybe > > > > > > > > > one > > > > > > > > > > > for > > > > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > > > > source/sink functionalities and > another > > > for > > > > > > > schema > > > > > > > > > and > > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > > integration, > > > > > > > > > > > > > > > > > > > > just to make them easier to review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for > > > > FLIP-27. > > > > > > But > > > > > > > I > > > > > > > > > > don't > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > > > > > should be a blocker in cases where it > > > > cannot > > > > > > make > > > > > > > > its > > > > > > > > > > way > > > > > > > > > > > > > into > > > > > > > > > > > > > > > 1.10 > > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of > time > > > for > > > > > > > > > committers > > > > > > > > > > to > > > > > > > > > > > > > > review > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to > new > > > > > > > interfaces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bowen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket > > > Qin < > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends on > when > > > the > > > > > new > > > > > > > > > source > > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it > would > > > be > > > > > > there > > > > > > > in > > > > > > > > > > > about a > > > > > > > > > > > > > > month > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > so. > > > > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It would > > > also > > > > be > > > > > > > good > > > > > > > > to > > > > > > > > > > > hear > > > > > > > > > > > > > what > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > > > > and Stephan think as they are also > > > > involved > > > > > > in > > > > > > > > > > FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general I think we should have > > > Pulsar > > > > > > > > connector > > > > > > > > > in > > > > > > > > > > > Flink > > > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > > > > preferably with the new source > > > interface. > > > > > We > > > > > > > can > > > > > > > > > also > > > > > > > > > > > check > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > > > now > > > > > > > > > > > > > > > > > > > > > with old source interface, but I > > > suspect > > > > > few > > > > > > > > users > > > > > > > > > > > will use > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > next official release. Therefore, > it > > > > seems > > > > > > > > > reasonable > > > > > > > > > > > to > > > > > > > > > > > > > > wait a > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > to see whether we can jump to the > new > > > > > source > > > > > > > > > > > interface. As > > > > > > > > > > > > > > long > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a > > > little > > > > > bit > > > > > > > > > doesn't > > > > > > > > > > > seem > > > > > > > > > > > > > to > > > > > > > > > > > > > > > hurt > > > > > > > > > > > > > > > > > much. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till > > > > > Rohrmann > > > > > > < > > > > > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem > > would > > > be > > > > > if > > > > > > we > > > > > > > > > > > committed > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > connector before the new source > > > > interface > > > > > > is > > > > > > > > > ready. > > > > > > > > > > > If I > > > > > > > > > > > > > > > > > understood > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to > support > > > the > > > > > old > > > > > > > > source > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > existing connectors. By checking > it > > > in > > > > > > early > > > > > > > I > > > > > > > > > > could > > > > > > > > > > > see > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > our users could start using the > > > > connector > > > > > > > > > earlier. > > > > > > > > > > > > > > Moreover, > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar > integration > > > is > > > > > > being > > > > > > > > > > delayed > > > > > > > > > > > in > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > > > > source interface should be > delayed. > > > The > > > > > > only > > > > > > > > > > > downside I > > > > > > > > > > > > > see > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > > > > review effort and potential fixes > > > which > > > > > > might > > > > > > > > be > > > > > > > > > > > > > irrelevant > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > source interface implementation. > I > > > > guess > > > > > it > > > > > > > > > mainly > > > > > > > > > > > > > depends > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > certain > > > > > > > > > > > > > > > > > > > > > > we are when the new source > > interface > > > > will > > > > > > be > > > > > > > > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM > > Becket > > > > > Qin < > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some update > on > > > > > FLIP-27. > > > > > > > > > > Although > > > > > > > > > > > the > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > discussion thread has been > quiet > > > for > > > > > some > > > > > > > > > time, a > > > > > > > > > > > few > > > > > > > > > > > > > > > > > committer / > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink community > > > were > > > > > > > actually > > > > > > > > > > > > > prototyping > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > > > > We have made some good progress > > > there > > > > > but > > > > > > > > want > > > > > > > > > to > > > > > > > > > > > > > update > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > > > > after the entire thing is > > verified > > > to > > > > > > work > > > > > > > in > > > > > > > > > > case > > > > > > > > > > > > > there > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the > > > > implementation. > > > > > I > > > > > > > > don't > > > > > > > > > > > have an > > > > > > > > > > > > > > > exact > > > > > > > > > > > > > > > > > ETA > > > > > > > > > > > > > > > > > > > yet, > > > > > > > > > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be within > a > > > > month > > > > > or > > > > > > > so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the > current > > > > Flink > > > > > > > Pulsar > > > > > > > > > > > connector > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > see if > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be > good > > to > > > > > avoid > > > > > > > the > > > > > > > > > > case > > > > > > > > > > > that > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some > review > > > > > efforts > > > > > > > and > > > > > > > > > > > shortly > > > > > > > > > > > > > > after > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > Source interface is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM > > > Yijie > > > > > > Shen < > > > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback > and > > > > > > > > suggestions! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of > the > > > > > > connector > > > > > > > > has > > > > > > > > > > > always > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > > users with the latest > features > > of > > > > > both > > > > > > > > > systems > > > > > > > > > > as > > > > > > > > > > > > > soon > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the > > > connector > > > > > to > > > > > > > > Flink > > > > > > > > > > and > > > > > > > > > > > hope > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from > > > Flink > > > > > > > experts > > > > > > > > > to > > > > > > > > > > > ensure > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > high > > > > > > > > > > > > > > > > > > > > > quality > > > > > > > > > > > > > > > > > > > > > > > > of the connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its > > > > existence > > > > > > at > > > > > > > > the > > > > > > > > > > > > > beginning > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > reworking > > > > > > > > > > > > > > > > > > > > > > > > the connector implementation > > > based > > > > on > > > > > > > Flink > > > > > > > > > > 1.9; > > > > > > > > > > > we > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > wanted > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > build a connector that > supports > > > > both > > > > > > > batch > > > > > > > > > and > > > > > > > > > > > stream > > > > > > > > > > > > > > > > > computing > > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > > > > on it. > > > > > > > > > > > > > > > > > > > > > > > > However, it has been inactive > > for > > > > > some > > > > > > > > time, > > > > > > > > > so > > > > > > > > > > > we > > > > > > > > > > > > > > > decided > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > > a connector with most of the > > new > > > > > > > features, > > > > > > > > > such > > > > > > > > > > > as > > > > > > > > > > > > > the > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API > first. > > We > > > > > will > > > > > > > pay > > > > > > > > > > > attention > > > > > > > > > > > > > to > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > progress > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and > > > > > incorporate > > > > > > it > > > > > > > > > with > > > > > > > > > > > the > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > > > > > > > > > as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of > > the > > > > > > > connector, > > > > > > > > > we > > > > > > > > > > > are > > > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink > > > > repository > > > > > > and > > > > > > > > > aimed > > > > > > > > > > to > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > throughout > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are > also > > > > happy > > > > > to > > > > > > > > hear > > > > > > > > > > > > > > suggestions > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink > > > > community > > > > > to > > > > > > > > > improve > > > > > > > > > > > the > > > > > > > > > > > > > > > > > stability and > > > > > > > > > > > > > > > > > > > > > > > > performance of the connector > > > > > > > continuously. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 > AM > > > > Sijie > > > > > > Guo > > > > > > > < > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the > > > comments > > > > > and > > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the > main > > > > > question > > > > > > > > here > > > > > > > > > is > > > > > > > > > > > > > about - > > > > > > > > > > > > > > > > "how > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > community maintain the > > > > connector?". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from > > > > myself. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to > > > host > > > > > this > > > > > > > > > > > integration > > > > > > > > > > > > > is > > > > > > > > > > > > > > > kind > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > less > > > > > > > > > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can > be > > > many > > > > > > ways > > > > > > > to > > > > > > > > > > > achieve > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > As part of the > contribution, > > > what > > > > > we > > > > > > > are > > > > > > > > > > > looking > > > > > > > > > > > > > for > > > > > > > > > > > > > > > here > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > > > > communities can build the > > > > > > collaboration > > > > > > > > > > > > > relationship > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > developing > > > > > > > > > > > > > > > > > > > > > > > > > the integration between > > Pulsar > > > > and > > > > > > > Flink. > > > > > > > > > > Even > > > > > > > > > > > we > > > > > > > > > > > > > can > > > > > > > > > > > > > > > try > > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > catch > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink > > > > > > community. > > > > > > > We > > > > > > > > > are > > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we > have > > > less > > > > > > > > > experiences > > > > > > > > > > > in > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > > folks > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > community. In order to make > > > sure > > > > we > > > > > > > > > maintain > > > > > > > > > > > and > > > > > > > > > > > > > > > deliver > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink > > > > > > integration > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > > users > > > > > > > > > > > > > > > who > > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need some > > help > > > > > from > > > > > > > the > > > > > > > > > > > experts > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following > > > FLIP-27 > > > > > > for a > > > > > > > > > > while. > > > > > > > > > > > > > > > Originally > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > > > > > > > > thinking > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the > > connectors > > > > back > > > > > > > after > > > > > > > > > > > > > integrating > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in > > FLIP-27. > > > > But > > > > > we > > > > > > > > > decided > > > > > > > > > > > to > > > > > > > > > > > > > > > initiate > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > conversation > > > > > > > > > > > > > > > > > > > > > > > > > as early as possible. > Because > > > we > > > > > > > believe > > > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > benefits > > > > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than later. > As > > > part > > > > > of > > > > > > > > > > > contribution, > > > > > > > > > > > > > it > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > community understand more > > about > > > > > > Pulsar > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > > > > points. > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink > > > > > community > > > > > > > > > verify > > > > > > > > > > > the > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. > > catalog > > > > > API). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 > > AM > > > > > Becket > > > > > > > > Qin < > > > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest > in > > > > > > > contributing > > > > > > > > > the > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think > having > > > > Pulsar > > > > > > > > > connector > > > > > > > > > > > with > > > > > > > > > > > > > > > strong > > > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to > Flink. > > > So > > > > I > > > > > am > > > > > > > > happy > > > > > > > > > > the > > > > > > > > > > > > > > > shepherd > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also > > like > > > to > > > > > > > provide > > > > > > > > > > some > > > > > > > > > > > > > > context > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > recent > > > > > > > > > > > > > > > > > > > > > > > > efforts on > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink > > > maintains > > > > > its > > > > > > > > > > > connector has > > > > > > > > > > > > > > hit > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > > > > > > > > > bar. > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more > > connectors > > > > > > coming > > > > > > > > into > > > > > > > > > > > Flink > > > > > > > > > > > > > > repo, > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > facing a > > > > > > > > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long > build > > > and > > > > > > > testing > > > > > > > > > > > time. To > > > > > > > > > > > > > > > > address > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > > problem, > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the > connectors > > > > into > > > > > a > > > > > > > > > separate > > > > > > > > > > > > > > > repository. > > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > temporarily > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential > > > > solution > > > > > > to > > > > > > > > > > shorten > > > > > > > > > > > the > > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the > connectors > > > to > > > > > stay > > > > > > > as > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > tries > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good support > for > > > > > > > > functionality > > > > > > > > > > and > > > > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > > > > tests. > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a > > Flink > > > > > > > Ecosystem > > > > > > > > > > > project > > > > > > > > > > > > > > > website > > > > > > > > > > > > > > > > > and it > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > > > > > through some final > approval > > > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, > it > > > > would > > > > > > be > > > > > > > > > great > > > > > > > > > > to > > > > > > > > > > > > > first > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > if we > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an > > > > ecosystem > > > > > > > > project > > > > > > > > > > with > > > > > > > > > > > > > great > > > > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the > Flink > > > > Pulsar > > > > > > > > > connector > > > > > > > > > > > is > > > > > > > > > > > > > > tested > > > > > > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn something to > > > > > maintain > > > > > > it > > > > > > > > as > > > > > > > > > an > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test > coverage. > > If > > > > the > > > > > > > > quality > > > > > > > > > > as > > > > > > > > > > > an > > > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as > > well > > > > > adopt > > > > > > it > > > > > > > > > into > > > > > > > > > > > the > > > > > > > > > > > > > main > > > > > > > > > > > > > > > > repo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing > effort > > > is > > > > > > > FLIP-27 > > > > > > > > > > where > > > > > > > > > > > we > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector > > > > > architecture > > > > > > > and > > > > > > > > > > > > > interface. > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore > > > timing > > > > > > wise, > > > > > > > if > > > > > > > > > we > > > > > > > > > > > are > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I > > am > > > > > > > wondering > > > > > > > > if > > > > > > > > > > we > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > hold > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector > > > adapt > > > > to > > > > > > the > > > > > > > > new > > > > > > > > > > > > > interface > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at > 4:32 > > > PM > > > > > > > Chesnay > > > > > > > > > > > Schepler < > > > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that > we > > > may > > > > > end > > > > > > > up > > > > > > > > > > > repeating > > > > > > > > > > > > > > > > history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2 > > > attempts > > > > > at > > > > > > > > > > > contributing a > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed because > > no > > > > > > > committer > > > > > > > > > was > > > > > > > > > > > > > getting > > > > > > > > > > > > > > > > > involved, > > > > > > > > > > > > > > > > > > > > > > despite > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a > > > > dedicated > > > > > > > > > > discussion > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > > about the > > > > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting > > > > several > > > > > > +1's > > > > > > > > > from > > > > > > > > > > > > > > > committers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make > > sure > > > > that > > > > > > if > > > > > > > we > > > > > > > > > > > > > > > welcome/approve > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will > > > actually > > > > > get > > > > > > > the > > > > > > > > > > > attention > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > deserves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined > to > > > > > > recommend > > > > > > > > > > > maintaining > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to > > it > > > > from > > > > > > the > > > > > > > > > > > > > documentation > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page > > for > > > > > > sharing > > > > > > > > > > > artifacts > > > > > > > > > > > > > > among > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > > (what's > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that > > anyway?), > > > > > this > > > > > > > may > > > > > > > > > be a > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, > Till > > > > > > Rohrmann > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for > > starting > > > > > this > > > > > > > > > > discussion > > > > > > > > > > > > > > Yijie. > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a > > very > > > > > > > valuable > > > > > > > > > > > addition > > > > > > > > > > > > > > since > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it > > would > > > > > > further > > > > > > > > > > expand > > > > > > > > > > > > > > Flink's > > > > > > > > > > > > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > > > Also > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project > > > perspective > > > > it > > > > > > > makes > > > > > > > > > > > sense for > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > place > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream > project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main > > concern/question > > > is > > > > > how > > > > > > > can > > > > > > > > > the > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have > seen > > > in > > > > > the > > > > > > > past > > > > > > > > > > that > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed > > > > components > > > > > > > > because > > > > > > > > > > > they > > > > > > > > > > > > > need > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > kept > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > sync > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > external system and > > with > > > > > Flink. > > > > > > > > Given > > > > > > > > > > > that > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > willing > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with > > maintaining, > > > > > > > improving > > > > > > > > > and > > > > > > > > > > > > > > evolving > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we > can > > > > > achieve > > > > > > > > this. > > > > > > > > > > > Hence, > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 > at > > > 2:03 > > > > > AM > > > > > > > > Sijie > > > > > > > > > > Guo > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main > > > > driver > > > > > > > behind > > > > > > > > > > > > > FLINK-9641 > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > FLINK-9168, > > > > > > > > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > > > > > try to > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on > > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and > > > FLINK-9168 > > > > > was > > > > > > > > > created > > > > > > > > > > > for > > > > > > > > > > > > > > > bringing > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The > > > > > > integration > > > > > > > > was > > > > > > > > > > done > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > 1.6.0. > > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > > sent out > > > > > > > > > > > > > > > > > > > > > > > > > > > pull > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a > year > > > ago > > > > > and > > > > > > we > > > > > > > > > ended > > > > > > > > > > > up > > > > > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > > > > > those > > > > > > > > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar > > users > > > to > > > > > use > > > > > > > > Flink > > > > > > > > > > to > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is > > > pretty > > > > > > simple > > > > > > > > and > > > > > > > > > > > there > > > > > > > > > > > > > is > > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we > > > have > > > > > > made a > > > > > > > > lot > > > > > > > > > > of > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > brought > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the > > > > > > first-class > > > > > > > > > > > citizen in > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > > integrated > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing > > engines > > > > for > > > > > > > > > processing > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink > > how > > > > to > > > > > > > > > integrate > > > > > > > > > > > with > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > > > way. > > > > > > > > > > > > > > > > > > > > > > > > Then > > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the > > > > pulsar-flink > > > > > > > > > > connectors > > > > > > > > > > > from > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > ground > > > > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and > > > > catalog > > > > > > API > > > > > > > as > > > > > > > > > the > > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > > > citizen > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With > that > > > > being > > > > > > > said, > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register > > pulsar > > > > as a > > > > > > > flink > > > > > > > > > > > catalog > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > query / > > > > > > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink > > SQL. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example > > about > > > > how > > > > > > to > > > > > > > > use > > > > > > > > > > > Pulsar > > > > > > > > > > > > > as > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > catalog: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also > > written a > > > > > blog > > > > > > > post > > > > > > > > > > > > > explaining > > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > re-implement > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink > > 1.9 > > > > and > > > > > > > what > > > > > > > > > are > > > > > > > > > > > the > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > made > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is > > not > > > > > just > > > > > > a > > > > > > > > > simple > > > > > > > > > > > data > > > > > > > > > > > > > > sink > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a > > fully > > > > > > > integrated > > > > > > > > > > > streaming > > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > Flink in > > > > > > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source, > > > > > > > > schema/catalog > > > > > > > > > > and > > > > > > > > > > > > > > state). > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > combination > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can > create > > a > > > > > great > > > > > > > > > > streaming > > > > > > > > > > > > > > > warehouse > > > > > > > > > > > > > > > > > > > > > architecture > > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, > > unified > > > > > data > > > > > > > > > > > processing. > > > > > > > > > > > > > > Since > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > talking > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar > > > > > integration > > > > > > to > > > > > > > > > Flink > > > > > > > > > > > here, > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > > dedicated to > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve > and > > > > evolve > > > > > > the > > > > > > > > > > > integration > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink > and > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a > > bit > > > > > more > > > > > > > > > > background > > > > > > > > > > > > > about > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me > > know > > > > > what > > > > > > > are > > > > > > > > > your > > > > > > > > > > > > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 > at > > > > 11:54 > > > > > > AM > > > > > > > > Yun > > > > > > > > > > > Tang < > > > > > > > > > > > > > > > > > > > > myas...@live.com> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that > Pulsar > > > > > becomes > > > > > > > > more > > > > > > > > > > and > > > > > > > > > > > more > > > > > > > > > > > > > > > > popular > > > > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people > > > > willing > > > > > to > > > > > > > > > > > contribute to > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further > > > > > > discussion, > > > > > > > > > would > > > > > > > > > > > you > > > > > > > > > > > > > > please > > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > > > explanation > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship > > > between > > > > > this > > > > > > > > > thread > > > > > > > > > > to > > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > > JIRAs > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink > > [2] > > > > > > > > connector? > > > > > > > > > > > Will the > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > contains > > > > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or > totally > > > > > > different > > > > > > > > > > > > > > implementation? > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen < > > > > > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, > > > September > > > > 3, > > > > > > > 2019 > > > > > > > > > > 13:57 > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > > > dev@flink.apache.org > > > > < > > > > > > > > > > > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] > > > > > Contribute > > > > > > > > > Pulsar > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink > Community! > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to > open > > > the > > > > > > > > discussion > > > > > > > > > > of > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back > to > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief > > introduction > > > > to > > > > > > > Apache > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] > is a > > > > > > > > multi-tenant, > > > > > > > > > > > > > > > > > high-performance > > > > > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging > > > system. > > > > > > > Pulsar > > > > > > > > > > > includes > > > > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for > > > > multiple > > > > > > > > > clusters > > > > > > > > > > > in a > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > > > geo-replication > > > > of > > > > > > > > > messages > > > > > > > > > > > across > > > > > > > > > > > > > > > > > clusters, > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > low > > > > > > > > > > > > > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end > > latency, > > > > > > > seamless > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > over a > > > > > > > > > > > > > > > > > > > > > > million > > > > > > > > > > > > > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed > > message > > > > > > delivery > > > > > > > > > with > > > > > > > > > > > > > > persistent > > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache > > > > > > BookKeeper. > > > > > > > > > > > Nowadays, > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > > > adopted > > > > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more > > > > companies[2]. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of > > Pulsar > > > > > Flink > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink > > > > connector > > > > > we > > > > > > > are > > > > > > > > > > > planning > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > built > > > > > > > > > > > > > > > > > > > > > > > > > > upon > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and > > Pulsar > > > > > 2.4.0. > > > > > > > The > > > > > > > > > > main > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a > > streaming > > > > > > source > > > > > > > > with > > > > > > > > > > > > > > > exactly-once > > > > > > > > > > > > > > > > > > > > guarantee. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming > > > results > > > > to > > > > > > > > Pulsar > > > > > > > > > > with > > > > > > > > > > > > > > > > > at-least-once > > > > > > > > > > > > > > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > > > > > (We > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this > to > > > > > > > exactly-once > > > > > > > > > as > > > > > > > > > > > well > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction > features > > > > ready > > > > > in > > > > > > > its > > > > > > > > > > 2.5.0 > > > > > > > > > > > > > > > version) > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink > > new > > > > > Table > > > > > > > API > > > > > > > > > Type > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically > > > > (de)serialize > > > > > > > > > messages > > > > > > > > > > > with > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with > > Flink > > > > new > > > > > > > > Catalog > > > > > > > > > > API > > > > > > > > > > > > > > > > > (FLIP-30[4]), > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar > topics > > as > > > > > > tables > > > > > > > in > > > > > > > > > > > Table API > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > well as > > > > > > > > > > > > > > > > > > > > SQL > > > > > > > > > > > > > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > https://pulsar.apache.org/ > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
