My assumption is as Sijie's, that once the connector is either part of Flink, or part of the streamnative repo. No double maintenance.
I feel this discussion is very much caught in problems that are all solvable if we want to solve them. Maybe we can think what our goal for users and the communities is? - Do we want to help build a relationship between the Pulsar and Flink open source communities? - Will users find a connector in the streamnative repository? - Will users trust a connector that is not part of Flink as much? And then decide what is best according to the overall goals there. As mentioned before, I believe that we can handle connectors more pragmatically and less strict than the core of Flink, if it helps unlocking users. Best, Stephan On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> wrote: > Thanks Becket. > > I think it is better for the Flink community to judge the benefits of doing > this. I was trying to provide some views from outsiders. > > Thanks, > Sijie > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <becket....@gmail.com> wrote: > > > Hi Sijie, > > > > Yes, we will have to support existing old connectors and new connectors > in > > parallel for a while. We have to take that maintenance overhead because > > existing connectors have been used by the users for a long time. I guess > It > > may take at least a year for us to fully remove the old connectors. > > > > Process wise, we can do the same for Pulsar connector. But I am not sure > if > > we want to have the same burden on Pulsar connector, and I would like to > > understand the benefit of doing that. > > > > For users, the benefit of having the old Pulsar connector checked in > seems > > limited because 1) that code base will be immediately deprecated in the > > next release in 3-4 months; 2) users can always use it even if it is not > in > > the Flink code base. Admittedly it is not as convenient as having it in > > Flink code base, but doesn't seem super either. And after 3-4 months, > users > > can just use the new connector in Flink repo. > > > > For Flink developers, the old connector code base is not something that > we > > want to evolve later. Instead, these code will be deprecated and > > removed. So why do we want to get a beta version out to attract people to > > use something we don't want to maintain? > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> wrote: > > > > > Thanks everyone here. Sorry for jumping into the discussion here. > > > > > > I am not very familiar about the deprecation process in Flink. If I > > > misunderstood the process, please fix me. > > > > > > As far as I understand, FLIP-27 is introducing a new unified API for > > > connectors. After it introduces the new API > > > and before moving all the existing connectors from old API to new API, > > both > > > old ApI and new API will co-exist > > > for a while until Flink moves all existing connectors to new API. So > the > > > Pulsar connector (using old API) can > > > follow the deprecation process with other connector using old API and > the > > > deprecation of old API, no? > > > > > > If that's the case, I think contributing the current connector back to > > > Flink rather than maintaining it outside Flink > > > would provide a bit more benefits. We can deprecate the existing > > > streamnative/pulsar-flink repo and point the users > > > to use the connector in Flink repo. So all the review processes will > > happen > > > within Flink for both old connector and > > > new connector. It also reduces the confusions for the users as the > > > documentation and code base happen in one place. > > > > > > Thoughts? > > > > > > - Sijie > > > > > > > > > > > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > Thanks for the explanation, Stephan. I have a few questions / > thoughts. > > > > > > > > So that means we will remove the old connector without a major > version > > > > bump, is that correct? > > > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors > will > > > > always work because we saw some dependency class collisions in the > > past. > > > To > > > > make it safe we may have to maintain the old code for one more > release. > > > > > > > > To be honest I am still wondering if we have to put the old connector > > in > > > > Flink repo. if we check in the old connector to Flink. We will end up > > in > > > > the following situation: > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > 2. Old connector in Flink Repo, which may be different from the one > in > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12) > > > > 3. New connector in Flink Repo. > > > > > > > > We need to think about how to make the users in each case happy. > > > > - For users of (1), I assume Sijie and Yijie will have to maintain > the > > > code > > > > a bit longer for its own compatibility even after we have (2). In > that > > > > case, bugs found in old connector may or may not need to be fixed in > > both > > > > Flink and the streamnative/pulsar-flink repo. > > > > - For users of (2), will we provide bug fixes? If we do, it will be a > > > > little awkward because those bug fixes will be immediately deprecated > > in > > > > 1.11, and removed in 1.12. So we are essentially asking users to > > migrate > > > > away from the bug fix. After Flink 1.12, users may still have to > switch > > > to > > > > use (3) due to the potential dependency class conflicts mentioned > > above. > > > > - Users of (3) have a much easier life and don't need to worry too > > much. > > > > > > > > The above story seems a little complicated to tell. I think it will > be > > > much > > > > easier to not have (2) at all. > > > > 1. Old connector in streamnative/pulsar-flink repo. > > > > 3. New connector in Flink Repo. > > > > > > > > - Old connector will only be maintained in streamnative/pulsar-flink > > repo > > > > until it is fully deprecated. Users can always use the existing > Pulsar > > > > connector in that repo. > > > > - New connector will be in Flink repo and maintained like the other > > > > connectors. > > > > > > > > This seems much simpler for users understand and they will not be > > blocked > > > > from using the old connector. If the concern is about the quality of > > the > > > > connector in streamnative/pulsar-flink repo, is it enough for us just > > to > > > > review the code in streamnative/pulsar-flink connector to make sure > it > > > > looks good from Flink's perspective? > > > > > > > > What do you think? > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <se...@apache.org> > wrote: > > > > > > > > > My take would be the following: > > > > > > > > > > - If we merge the connector now and replace it with a FLIP-27 > > version > > > > > before the 1.10 release, then we need no deprecation process > > > > > - If we don't manage to replace it with a FLIP-27 version before > > the > > > > 1.10 > > > > > release, than it is good that we have the other version, so no > users > > > get > > > > > blocked. > > > > > > > > > > In the latter case we can see how we want to do it. Immediate > removal > > > of > > > > > the old version or deprecation label and keeping it for one more > > > release. > > > > > Given that you should be able to use a Flink 1.10 connector with > > Flink > > > > 1.11 > > > > > as well (stable public APIs) there is also a workaround if you need > > an > > > > old > > > > > connector in a newer version. So immediate removal might even be > > > > feasible. > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <becket....@gmail.com> > > > > wrote: > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > Thanks for the clarification. I completely agree with you and > > Thomas > > > on > > > > > the > > > > > > process of adding connectors to Flink repo. However, I am > wondering > > > > what > > > > > is > > > > > > the deprecation process? Given the main concern here was that we > > may > > > > have > > > > > > to maintain two Pulsar connector code bases until the old one is > > > > removed > > > > > > from the repo, it would be good to know how long we have to do > > that. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <se...@apache.org> > > > wrote: > > > > > > > > > > > > > Some quick thoughts on the connector contribution process. I > > > > basically > > > > > > > reiterate here what Thomas mentioned in another thread about > the > > > > > Kinesis > > > > > > > connector. > > > > > > > > > > > > > > For connectors, we should favor a low-overhead contribution > > > process, > > > > > and > > > > > > > accept user code and changes more readily than in the core > > system. > > > > > > > That is because connectors have both a big variety of scenarios > > > they > > > > > get > > > > > > > used in (only through use and many small contributions do they > > > become > > > > > > > really useful over time) and at the same time, and committers > do > > > not > > > > > use > > > > > > > the connector themselves and usually cannot foresee too well > what > > > is > > > > > > > needed. > > > > > > > > > > > > > > Further more, a missing connector (or connector feature) is > > often a > > > > > > bigger > > > > > > > show stopper for users than a missing API or system feature. > > > > > > > > > > > > > > Along these lines of thougt, the conclusion would be to take > the > > > > Pulsar > > > > > > > connector now, focus the review on legal/dependencies/rough > code > > > > style > > > > > > and > > > > > > > conventions, label it as "beta" (in the sense of "new code" > that > > is > > > > > "not > > > > > > > yet tested through longer use") and go ahead. And then evolve > it > > > > > quickly > > > > > > > without putting formal blockers in the way, meaning also > adding a > > > new > > > > > > FLIP > > > > > > > 27 version when it is there. > > > > > > > > > > > > > > Best, > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < > becket....@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > Could you please follow the FLIP process to start a new FLIP > > > > > > [DISCUSSION] > > > > > > > > thread in the mailing list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > > > > > > > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So > there > > > is a > > > > > > FLIP > > > > > > > > number collision. Can you change the FLIP number to 72? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > > walter...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > > > > > > > > > Would you mind enabling comments/suggestions on the google > > doc > > > > > link? > > > > > > > This > > > > > > > > > way the contributors from the community can comment on the > > doc. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > > > > > > henry.yijies...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > I've drafted a FLIP that describes the current design of > > the > > > > > Pulsar > > > > > > > > > > connector: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > > > > > > > > > > > > > > > > > > > > Please take a look and let me know what you think. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > > > > walter...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > Sorry for joining the discussion late and thanks Yijie > & > > > > Sijie > > > > > > for > > > > > > > > > > driving > > > > > > > > > > > the discussion. > > > > > > > > > > > I also think the Pulsar connector would be a very > > valuable > > > > > > addition > > > > > > > > to > > > > > > > > > > > Flink. I can also help out a bit on the review side :-) > > > > > > > > > > > > > > > > > > > > > > Regarding the timeline, I also share concerns with > Becket > > > on > > > > > the > > > > > > > > > > > relationship between the new Pulsar connector and > > FLIP-27. > > > > > > > > > > > There's also another discussion just started by Stephan > > on > > > > > > dropping > > > > > > > > > Kafka > > > > > > > > > > > 9/10 support on next Flink release [1]. Although the > > > > situation > > > > > > is > > > > > > > > > > somewhat > > > > > > > > > > > different, and Kafka 9/10 connector has been in Flink > for > > > > > almost > > > > > > > 3-4 > > > > > > > > > > years, > > > > > > > > > > > based on the discussion I am not sure if a major > version > > > > > release > > > > > > > is a > > > > > > > > > > > requirement for removing old connector supports. > > > > > > > > > > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree the > old > > > > > > connector > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is there. > As > > > > > Stephan > > > > > > > > > stated, > > > > > > > > > > it > > > > > > > > > > > is easier to contribute the source sooner and adjust it > > > > later. > > > > > > > > > > > We should also ensure we clearly communicate the > message: > > > for > > > > > > > > example, > > > > > > > > > > > putting an experimental flag on the pre-FLIP27 > connector > > > page > > > > > of > > > > > > > the > > > > > > > > > > > website, documentations, etc. Any other thoughts? > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Rong > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Technically speaking, removing the old connector code > > is > > > a > > > > > > > > backwards > > > > > > > > > > > > incompatible change which requires a major version > > bump, > > > > i.e. > > > > > > > Flink > > > > > > > > > > 2.x. > > > > > > > > > > > > Given that we don't have a clear plan on when to have > > the > > > > > next > > > > > > > > major > > > > > > > > > > > > version release, it seems unclear how long the old > > > > connector > > > > > > code > > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > there if we check it in right now. Or will we remove > > the > > > > old > > > > > > > > > connector > > > > > > > > > > > > without a major version bump? In any case, it sounds > > not > > > > > quite > > > > > > > user > > > > > > > > > > > > friendly to the those who might use the old Pulsar > > > > > connector. I > > > > > > > am > > > > > > > > > not > > > > > > > > > > sure > > > > > > > > > > > > if it is worth these potential problems in order to > > have > > > > the > > > > > > > Pulsar > > > > > > > > > > source > > > > > > > > > > > > connector checked in one or two months earlier. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen < > > > > > se...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Agreed, if we check in the old code, we should make > > it > > > > > clear > > > > > > > that > > > > > > > > > it > > > > > > > > > > will > > > > > > > > > > > > > be removed as soon as the FLIP-27 based version of > > the > > > > > > > connector > > > > > > > > is > > > > > > > > > > > > there. > > > > > > > > > > > > > We should not commit to maintaining the old > version, > > > that > > > > > > would > > > > > > > > be > > > > > > > > > > indeed > > > > > > > > > > > > > too much overhead. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin < > > > > > > > becket....@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, the overhead would just be review capacity. > In > > > > > fact, I > > > > > > > am > > > > > > > > > not > > > > > > > > > > > > > worrying > > > > > > > > > > > > > > too much about the review capacity. That is just > a > > > one > > > > > time > > > > > > > > cost. > > > > > > > > > > My > > > > > > > > > > > > > > concern is mainly about the long term burden. > > Assume > > > we > > > > > > have > > > > > > > > new > > > > > > > > > > source > > > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar > > > > > connectors > > > > > > in > > > > > > > > old > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new > > > source > > > > > > > > interface, > > > > > > > > > > the > > > > > > > > > > > > old > > > > > > > > > > > > > > Pulsar connector might be deprecated almost > > > immediately > > > > > > after > > > > > > > > > > checked > > > > > > > > > > > > in, > > > > > > > > > > > > > > but we may still have to maintain two code bases. > > For > > > > the > > > > > > > > > existing > > > > > > > > > > > > > > connectors, we have to do that anyways. But it > > would > > > be > > > > > > good > > > > > > > to > > > > > > > > > > avoid > > > > > > > > > > > > > > introducing a new connector with the same > problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen < > > > > > > > se...@apache.org > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nice to see this lively discussion about the > > Pulsar > > > > > > > > connector. > > > > > > > > > > > > > > > Some thoughts on the open questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a > community > > > > > package > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looks like the discussion is more going towards > > > > > > > > contribution. I > > > > > > > > > > think > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > is good, especially if we think that we want to > > > > build a > > > > > > > > > similarly > > > > > > > > > > > > deep > > > > > > > > > > > > > > > integration with Pulsar as we have for example > > with > > > > > > Kafka. > > > > > > > > The > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > already looks like a more thorough connector > than > > > > many > > > > > > > others > > > > > > > > > we > > > > > > > > > > have > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the repository. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > With either a repo split, or the new build > > system, > > > I > > > > > hope > > > > > > > > that > > > > > > > > > > the > > > > > > > > > > > > > build > > > > > > > > > > > > > > > overhead is not a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Committer Support > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket offered some help already, I can also > > help a > > > > > bit. > > > > > > I > > > > > > > > hope > > > > > > > > > > that > > > > > > > > > > > > > > > between us, we can cover this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some > > > > PoC-ing > > > > > > > > > > progress, but > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > take 2 more months, I would estimate, before it > > is > > > > > fully > > > > > > > > > > available. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we want to be on the safe side with the > > > > > contribution, > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > > > > contribute the source sooner and adjust it > later. > > > > That > > > > > > > would > > > > > > > > > also > > > > > > > > > > > > help > > > > > > > > > > > > > us > > > > > > > > > > > > > > > in case things get crazy towards the 1.10 > feature > > > > > freeze > > > > > > > and > > > > > > > > it > > > > > > > > > > would > > > > > > > > > > > > > be > > > > > > > > > > > > > > > hard to find time to review the new changes. > > > > > > > > > > > > > > > What would be the overhead of contributing now? > > > Given > > > > > > that > > > > > > > > the > > > > > > > > > > code > > > > > > > > > > > > is > > > > > > > > > > > > > > > already there, it looks like it would be only > > > review > > > > > > > > capacity, > > > > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen < > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your attention and the promotion > of > > > this > > > > > > work. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible > for > > > more > > > > > > > > specific > > > > > > > > > > > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not > reached > > a > > > > > > > consensus. > > > > > > > > > > > > > Therefore, > > > > > > > > > > > > > > > > I will explain all the functionalities of the > > > > > existing > > > > > > > > > > connector in > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and > Catalog) > > to > > > > > > > continue > > > > > > > > > our > > > > > > > > > > > > > > > > discussions in FLIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your kind help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin < > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar > > > > > connector > > > > > > > in > > > > > > > > > > 1.10, > > > > > > > > > > > > how > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > do the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to > > > Flink > > > > > main > > > > > > > > repo > > > > > > > > > > as it > > > > > > > > > > > > > is a > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > public interface to Flink main repo. > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right > away > > > as > > > > > > there > > > > > > > is > > > > > > > > > no > > > > > > > > > > > > change > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > sink interface so far. > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 > > is > > > > > going > > > > > > to > > > > > > > > be > > > > > > > > > > code > > > > > > > > > > > > > > freeze > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > late Nov and let's say we give a month to > the > > > > > > > development > > > > > > > > > and > > > > > > > > > > > > > review > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 > by > > > late > > > > > > Oct. > > > > > > > > > There > > > > > > > > > > are > > > > > > > > > > > > > > still > > > > > > > > > > > > > > > 7 > > > > > > > > > > > > > > > > > weeks. Personally I think it is doable. If > > > > FLIP-27 > > > > > is > > > > > > > not > > > > > > > > > > ready > > > > > > > > > > > > by > > > > > > > > > > > > > > late > > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar > > > connector > > > > > with > > > > > > > the > > > > > > > > > > > > existing > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > interface. This means we will have Pulsar > > > > connector > > > > > > in > > > > > > > > > Flink > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > either > > > > > > > > > > > > > > > > > with or without FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink > and > > > > source > > > > > > > > checked > > > > > > > > > > in > > > > > > > > > > > > > > > > separately, > > > > > > > > > > > > > > > > > it might make sense to have two FLIPs, one > > for > > > > > Pulsar > > > > > > > > sink > > > > > > > > > > and > > > > > > > > > > > > > > another > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > Pulsar source. And we can start the work on > > > > Pulsar > > > > > > sink > > > > > > > > > right > > > > > > > > > > > > away. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo < > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What's the take from Flink community? > Shall > > > we > > > > > wait > > > > > > > for > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > shall we > > > > > > > > > > > > > > > > > > proceed to next steps? And what the next > > > steps > > > > > are? > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li < > > > > > > > > > > bowenl...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in > > Flink > > > > can > > > > > > be a > > > > > > > > > good > > > > > > > > > > > > mutual > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > both communities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar > > > connector > > > > is > > > > > > the > > > > > > > > 1st > > > > > > > > > > > > > streaming > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > that integrates with Flink's metadata > > > > > management > > > > > > > > system > > > > > > > > > > and > > > > > > > > > > > > > > Catalog > > > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > > It'll be cool to see how the > integration > > > > turns > > > > > > out > > > > > > > > and > > > > > > > > > > > > whether > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are > > > > > currently > > > > > > in > > > > > > > > > > Beta, to > > > > > > > > > > > > > > cater > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in > favor > > of > > > > > > merging > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > > > > Flink 1.10. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized > PRs, > > > e.g. > > > > > > maybe > > > > > > > > one > > > > > > > > > > for > > > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > > > source/sink functionalities and another > > for > > > > > > schema > > > > > > > > and > > > > > > > > > > > > catalog > > > > > > > > > > > > > > > > > > integration, > > > > > > > > > > > > > > > > > > > just to make them easier to review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for > > > FLIP-27. > > > > > But > > > > > > I > > > > > > > > > don't > > > > > > > > > > > > think > > > > > > > > > > > > > > > > FLIP-27 > > > > > > > > > > > > > > > > > > > should be a blocker in cases where it > > > cannot > > > > > make > > > > > > > its > > > > > > > > > way > > > > > > > > > > > > into > > > > > > > > > > > > > > 1.10 > > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time > > for > > > > > > > > committers > > > > > > > > > to > > > > > > > > > > > > > review > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new > > > > > > interfaces. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bowen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket > > Qin < > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends on when > > the > > > > new > > > > > > > > source > > > > > > > > > > > > > interface > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it would > > be > > > > > there > > > > > > in > > > > > > > > > > about a > > > > > > > > > > > > > month > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > so. > > > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It would > > also > > > be > > > > > > good > > > > > > > to > > > > > > > > > > hear > > > > > > > > > > > > what > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > > > and Stephan think as they are also > > > involved > > > > > in > > > > > > > > > FLIP-27. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general I think we should have > > Pulsar > > > > > > > connector > > > > > > > > in > > > > > > > > > > Flink > > > > > > > > > > > > > > 1.10, > > > > > > > > > > > > > > > > > > > > preferably with the new source > > interface. > > > > We > > > > > > can > > > > > > > > also > > > > > > > > > > check > > > > > > > > > > > > > it > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > right > > > > > > > > > > > > > > > > > > > now > > > > > > > > > > > > > > > > > > > > with old source interface, but I > > suspect > > > > few > > > > > > > users > > > > > > > > > > will use > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > next official release. Therefore, it > > > seems > > > > > > > > reasonable > > > > > > > > > > to > > > > > > > > > > > > > wait a > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > to see whether we can jump to the new > > > > source > > > > > > > > > > interface. As > > > > > > > > > > > > > long > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a > > little > > > > bit > > > > > > > > doesn't > > > > > > > > > > seem > > > > > > > > > > > > to > > > > > > > > > > > > > > hurt > > > > > > > > > > > > > > > > much. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till > > > > Rohrmann > > > > > < > > > > > > > > > > > > > > > trohrm...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem > would > > be > > > > if > > > > > we > > > > > > > > > > committed > > > > > > > > > > > > the > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > connector before the new source > > > interface > > > > > is > > > > > > > > ready. > > > > > > > > > > If I > > > > > > > > > > > > > > > > understood > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > correctly, then we need to support > > the > > > > old > > > > > > > source > > > > > > > > > > > > interface > > > > > > > > > > > > > > > > anyway > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > existing connectors. By checking it > > in > > > > > early > > > > > > I > > > > > > > > > could > > > > > > > > > > see > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > our users could start using the > > > connector > > > > > > > > earlier. > > > > > > > > > > > > > Moreover, > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration > > is > > > > > being > > > > > > > > > delayed > > > > > > > > > > in > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > > > source interface should be delayed. > > The > > > > > only > > > > > > > > > > downside I > > > > > > > > > > > > see > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > extra > > > > > > > > > > > > > > > > > > > > > review effort and potential fixes > > which > > > > > might > > > > > > > be > > > > > > > > > > > > irrelevant > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > source interface implementation. I > > > guess > > > > it > > > > > > > > mainly > > > > > > > > > > > > depends > > > > > > > > > > > > > on > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > certain > > > > > > > > > > > > > > > > > > > > > we are when the new source > interface > > > will > > > > > be > > > > > > > > ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM > Becket > > > > Qin < > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some update on > > > > FLIP-27. > > > > > > > > > Although > > > > > > > > > > the > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > discussion thread has been quiet > > for > > > > some > > > > > > > > time, a > > > > > > > > > > few > > > > > > > > > > > > > > > > committer / > > > > > > > > > > > > > > > > > > > > > > contributors in Flink community > > were > > > > > > actually > > > > > > > > > > > > prototyping > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > entire > > > > > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > > > We have made some good progress > > there > > > > but > > > > > > > want > > > > > > > > to > > > > > > > > > > > > update > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > FLIP > > > > > > > > > > > > > > > > > > > wiki > > > > > > > > > > > > > > > > > > > > > > after the entire thing is > verified > > to > > > > > work > > > > > > in > > > > > > > > > case > > > > > > > > > > > > there > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > > > > > > > minute surprise in the > > > implementation. > > > > I > > > > > > > don't > > > > > > > > > > have an > > > > > > > > > > > > > > exact > > > > > > > > > > > > > > > > ETA > > > > > > > > > > > > > > > > > > yet, > > > > > > > > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > > > > > > > > guess it is going to be within a > > > month > > > > or > > > > > > so. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the current > > > Flink > > > > > > Pulsar > > > > > > > > > > connector > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > see if > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good > to > > > > avoid > > > > > > the > > > > > > > > > case > > > > > > > > > > that > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > checked > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some review > > > > efforts > > > > > > and > > > > > > > > > > shortly > > > > > > > > > > > > > after > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > Source interface is ready. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM > > Yijie > > > > > Shen < > > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and > > > > > > > suggestions! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the > > > > > connector > > > > > > > has > > > > > > > > > > always > > > > > > > > > > > > > been > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > users with the latest features > of > > > > both > > > > > > > > systems > > > > > > > > > as > > > > > > > > > > > > soon > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > possible. > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the > > connector > > > > to > > > > > > > Flink > > > > > > > > > and > > > > > > > > > > hope > > > > > > > > > > > > > to > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from > > Flink > > > > > > experts > > > > > > > > to > > > > > > > > > > ensure > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > high > > > > > > > > > > > > > > > > > > > > quality > > > > > > > > > > > > > > > > > > > > > > > of the connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its > > > existence > > > > > at > > > > > > > the > > > > > > > > > > > > beginning > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > reworking > > > > > > > > > > > > > > > > > > > > > > > the connector implementation > > based > > > on > > > > > > Flink > > > > > > > > > 1.9; > > > > > > > > > > we > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > wanted > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > build a connector that supports > > > both > > > > > > batch > > > > > > > > and > > > > > > > > > > stream > > > > > > > > > > > > > > > > computing > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > > > > > > > > > > > > on it. > > > > > > > > > > > > > > > > > > > > > > > However, it has been inactive > for > > > > some > > > > > > > time, > > > > > > > > so > > > > > > > > > > we > > > > > > > > > > > > > > decided > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > > > > > a connector with most of the > new > > > > > > features, > > > > > > > > such > > > > > > > > > > as > > > > > > > > > > > > the > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > type > > > > > > > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API first. > We > > > > will > > > > > > pay > > > > > > > > > > attention > > > > > > > > > > > > to > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > progress > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and > > > > incorporate > > > > > it > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > soon > > > > > > > > > > > > > > > > > > > > > > > as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of > the > > > > > > connector, > > > > > > > > we > > > > > > > > > > are > > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink > > > repository > > > > > and > > > > > > > > aimed > > > > > > > > > to > > > > > > > > > > > > > provide > > > > > > > > > > > > > > > > > > > throughout > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are also > > > happy > > > > to > > > > > > > hear > > > > > > > > > > > > > suggestions > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink > > > community > > > > to > > > > > > > > improve > > > > > > > > > > the > > > > > > > > > > > > > > > > stability and > > > > > > > > > > > > > > > > > > > > > > > performance of the connector > > > > > > continuously. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM > > > Sijie > > > > > Guo > > > > > > < > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the > > comments > > > > and > > > > > > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the main > > > > question > > > > > > > here > > > > > > > > is > > > > > > > > > > > > about - > > > > > > > > > > > > > > > "how > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > community maintain the > > > connector?". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from > > > myself. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to > > host > > > > this > > > > > > > > > > integration > > > > > > > > > > > > is > > > > > > > > > > > > > > kind > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > less > > > > > > > > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can be > > many > > > > > ways > > > > > > to > > > > > > > > > > achieve > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > As part of the contribution, > > what > > > > we > > > > > > are > > > > > > > > > > looking > > > > > > > > > > > > for > > > > > > > > > > > > > > here > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > > two > > > > > > > > > > > > > > > > > > > > > > > > communities can build the > > > > > collaboration > > > > > > > > > > > > relationship > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > developing > > > > > > > > > > > > > > > > > > > > > > > > the integration between > Pulsar > > > and > > > > > > Flink. > > > > > > > > > Even > > > > > > > > > > we > > > > > > > > > > > > can > > > > > > > > > > > > > > try > > > > > > > > > > > > > > > > our > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > catch > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink > > > > > community. > > > > > > We > > > > > > > > are > > > > > > > > > > still > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we have > > less > > > > > > > > experiences > > > > > > > > > > in > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > > > folks > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > community. In order to make > > sure > > > we > > > > > > > > maintain > > > > > > > > > > and > > > > > > > > > > > > > > deliver > > > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink > > > > > integration > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > users > > > > > > > > > > > > > > who > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need some > help > > > > from > > > > > > the > > > > > > > > > > experts > > > > > > > > > > > > from > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following > > FLIP-27 > > > > > for a > > > > > > > > > while. > > > > > > > > > > > > > > Originally > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > were > > > > > > > > > > > > > > > > > > > > > > > thinking > > > > > > > > > > > > > > > > > > > > > > > > of contributing the > connectors > > > back > > > > > > after > > > > > > > > > > > > integrating > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in > FLIP-27. > > > But > > > > we > > > > > > > > decided > > > > > > > > > > to > > > > > > > > > > > > > > initiate > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > conversation > > > > > > > > > > > > > > > > > > > > > > > > as early as possible. Because > > we > > > > > > believe > > > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > benefits > > > > > > > > > > > > > > > > > > > > > doing > > > > > > > > > > > > > > > > > > > > > > > > it now rather than later. As > > part > > > > of > > > > > > > > > > contribution, > > > > > > > > > > > > it > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > community understand more > about > > > > > Pulsar > > > > > > > and > > > > > > > > > the > > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > > > integration > > > > > > > > > > > > > > > > > > > > > > > points. > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink > > > > community > > > > > > > > verify > > > > > > > > > > the > > > > > > > > > > > > new > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > API > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > well > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. > catalog > > > > API). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 > AM > > > > Becket > > > > > > > Qin < > > > > > > > > > > > > > > > > > > becket....@gmail.com> > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in > > > > > > contributing > > > > > > > > the > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > connector. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think having > > > Pulsar > > > > > > > > connector > > > > > > > > > > with > > > > > > > > > > > > > > strong > > > > > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. > > So > > > I > > > > am > > > > > > > happy > > > > > > > > > the > > > > > > > > > > > > > > shepherd > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also > like > > to > > > > > > provide > > > > > > > > > some > > > > > > > > > > > > > context > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > recent > > > > > > > > > > > > > > > > > > > > > > > efforts on > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink > > maintains > > > > its > > > > > > > > > > connector has > > > > > > > > > > > > > hit > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > scalability > > > > > > > > > > > > > > > > > > > > > > > bar. > > > > > > > > > > > > > > > > > > > > > > > > > With more and more > connectors > > > > > coming > > > > > > > into > > > > > > > > > > Flink > > > > > > > > > > > > > repo, > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > facing a > > > > > > > > > > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long build > > and > > > > > > testing > > > > > > > > > > time. To > > > > > > > > > > > > > > > address > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > > > > > problem, > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors > > > into > > > > a > > > > > > > > separate > > > > > > > > > > > > > > repository. > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > temporarily > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential > > > solution > > > > > to > > > > > > > > > shorten > > > > > > > > > > the > > > > > > > > > > > > > > build > > > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors > > to > > > > stay > > > > > > as > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > while > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > tries > > > > > > > > > > > > > > > > > > > > > > > > > to provide good support for > > > > > > > functionality > > > > > > > > > and > > > > > > > > > > > > > > > > compatibility > > > > > > > > > > > > > > > > > > > > tests. > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a > Flink > > > > > > Ecosystem > > > > > > > > > > project > > > > > > > > > > > > > > website > > > > > > > > > > > > > > > > and it > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > going > > > > > > > > > > > > > > > > > > > > > > > > > through some final approval > > > > > process. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it > > > would > > > > > be > > > > > > > > great > > > > > > > > > to > > > > > > > > > > > > first > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > if we > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an > > > ecosystem > > > > > > > project > > > > > > > > > with > > > > > > > > > > > > great > > > > > > > > > > > > > > > > support. > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink > > > Pulsar > > > > > > > > connector > > > > > > > > > > is > > > > > > > > > > > > > tested > > > > > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > > > > > > > > > if > > > > > > > > > > > > > > > > > > > > > > > > > we can learn something to > > > > maintain > > > > > it > > > > > > > as > > > > > > > > an > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. > If > > > the > > > > > > > quality > > > > > > > > > as > > > > > > > > > > an > > > > > > > > > > > > > > > ecosystem > > > > > > > > > > > > > > > > > > > project > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > hard > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as > well > > > > adopt > > > > > it > > > > > > > > into > > > > > > > > > > the > > > > > > > > > > > > main > > > > > > > > > > > > > > > repo. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort > > is > > > > > > FLIP-27 > > > > > > > > > where > > > > > > > > > > we > > > > > > > > > > > > are > > > > > > > > > > > > > > > > making > > > > > > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector > > > > architecture > > > > > > and > > > > > > > > > > > > interface. > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > > > > > > likely > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore > > timing > > > > > wise, > > > > > > if > > > > > > > > we > > > > > > > > > > are > > > > > > > > > > > > > going > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I > am > > > > > > wondering > > > > > > > if > > > > > > > > > we > > > > > > > > > > > > should > > > > > > > > > > > > > > > hold > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > little > > > > > > > > > > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector > > adapt > > > to > > > > > the > > > > > > > new > > > > > > > > > > > > interface > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > > > > > > shortly > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 > > PM > > > > > > Chesnay > > > > > > > > > > Schepler < > > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we > > may > > > > end > > > > > > up > > > > > > > > > > repeating > > > > > > > > > > > > > > > history. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2 > > attempts > > > > at > > > > > > > > > > contributing a > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > > both > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed because > no > > > > > > committer > > > > > > > > was > > > > > > > > > > > > getting > > > > > > > > > > > > > > > > involved, > > > > > > > > > > > > > > > > > > > > > despite > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a > > > dedicated > > > > > > > > > discussion > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > about the > > > > > > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting > > > several > > > > > +1's > > > > > > > > from > > > > > > > > > > > > > > committers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make > sure > > > that > > > > > if > > > > > > we > > > > > > > > > > > > > > welcome/approve > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will > > actually > > > > get > > > > > > the > > > > > > > > > > attention > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > deserves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to > > > > > recommend > > > > > > > > > > maintaining > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to > it > > > from > > > > > the > > > > > > > > > > > > documentation > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > exposure. > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page > for > > > > > sharing > > > > > > > > > > artifacts > > > > > > > > > > > > > among > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > > (what's > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that > anyway?), > > > > this > > > > > > may > > > > > > > > be a > > > > > > > > > > > > better > > > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till > > > > > Rohrmann > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for > starting > > > > this > > > > > > > > > discussion > > > > > > > > > > > > > Yijie. > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a > very > > > > > > valuable > > > > > > > > > > addition > > > > > > > > > > > > > since > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > becomes > > > > > > > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it > would > > > > > further > > > > > > > > > expand > > > > > > > > > > > > > Flink's > > > > > > > > > > > > > > > > > > > > > > interoperability. > > > > > > > > > > > > > > > > > > > > > > > Also > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project > > perspective > > > it > > > > > > makes > > > > > > > > > > sense for > > > > > > > > > > > > > me > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > place > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main > concern/question > > is > > > > how > > > > > > can > > > > > > > > the > > > > > > > > > > Flink > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > maintain > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen > > in > > > > the > > > > > > past > > > > > > > > > that > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > most > > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed > > > components > > > > > > > because > > > > > > > > > > they > > > > > > > > > > > > need > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > kept > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > sync > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > external system and > with > > > > Flink. > > > > > > > Given > > > > > > > > > > that > > > > > > > > > > > > the > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > community > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > willing > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with > maintaining, > > > > > > improving > > > > > > > > and > > > > > > > > > > > > > evolving > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > connector, > > > > > > > > > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can > > > > achieve > > > > > > > this. > > > > > > > > > > Hence, > > > > > > > > > > > > +1 > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at > > 2:03 > > > > AM > > > > > > > Sijie > > > > > > > > > Guo > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > guosi...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main > > > driver > > > > > > behind > > > > > > > > > > > > FLINK-9641 > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > FLINK-9168, > > > > > > > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > > > > > > > > > > try to > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on > > this. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and > > FLINK-9168 > > > > was > > > > > > > > created > > > > > > > > > > for > > > > > > > > > > > > > > bringing > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The > > > > > integration > > > > > > > was > > > > > > > > > done > > > > > > > > > > > > with > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > 1.6.0. > > > > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > > > > sent out > > > > > > > > > > > > > > > > > > > > > > > > > > pull > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year > > ago > > > > and > > > > > we > > > > > > > > ended > > > > > > > > > > up > > > > > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > > > > > > those > > > > > > > > > > > > > > > > > > > > > > > connectors > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar > users > > to > > > > use > > > > > > > Flink > > > > > > > > > to > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > ). > > > > > > > > > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is > > pretty > > > > > simple > > > > > > > and > > > > > > > > > > there > > > > > > > > > > > > is > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > considerations. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we > > have > > > > > made a > > > > > > > lot > > > > > > > > > of > > > > > > > > > > > > > changes > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > brought > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the > > > > > first-class > > > > > > > > > > citizen in > > > > > > > > > > > > > > > Pulsar. > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > integrated > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing > engines > > > for > > > > > > > > processing > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > streams > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink > how > > > to > > > > > > > > integrate > > > > > > > > > > with > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > > way. > > > > > > > > > > > > > > > > > > > > > > > Then > > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the > > > pulsar-flink > > > > > > > > > connectors > > > > > > > > > > from > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > ground > > > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and > > > catalog > > > > > API > > > > > > as > > > > > > > > the > > > > > > > > > > > > > > first-class > > > > > > > > > > > > > > > > > > citizen > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that > > > being > > > > > > said, > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register > pulsar > > > as a > > > > > > flink > > > > > > > > > > catalog > > > > > > > > > > > > and > > > > > > > > > > > > > > > > query / > > > > > > > > > > > > > > > > > > > > process > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > event > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink > SQL. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example > about > > > how > > > > > to > > > > > > > use > > > > > > > > > > Pulsar > > > > > > > > > > > > as > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > catalog: > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also > written a > > > > blog > > > > > > post > > > > > > > > > > > > explaining > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > re-implement > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink > 1.9 > > > and > > > > > > what > > > > > > > > are > > > > > > > > > > the > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > made > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is > not > > > > just > > > > > a > > > > > > > > simple > > > > > > > > > > data > > > > > > > > > > > > > sink > > > > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a > fully > > > > > > integrated > > > > > > > > > > streaming > > > > > > > > > > > > > > data > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > Flink in > > > > > > > > > > > > > > > > > > > > > > > > > > many > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source, > > > > > > > schema/catalog > > > > > > > > > and > > > > > > > > > > > > > state). > > > > > > > > > > > > > > > The > > > > > > > > > > > > > > > > > > > > > combination > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create > a > > > > great > > > > > > > > > streaming > > > > > > > > > > > > > > warehouse > > > > > > > > > > > > > > > > > > > > architecture > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, > unified > > > > data > > > > > > > > > > processing. > > > > > > > > > > > > > Since > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > talking > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar > > > > integration > > > > > to > > > > > > > > Flink > > > > > > > > > > here, > > > > > > > > > > > > > we > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > > > > > > dedicated to > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and > > > evolve > > > > > the > > > > > > > > > > integration > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and > > > > Pulsar. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a > bit > > > > more > > > > > > > > > background > > > > > > > > > > > > about > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > flink > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me > know > > > > what > > > > > > are > > > > > > > > your > > > > > > > > > > > > > thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at > > > 11:54 > > > > > AM > > > > > > > Yun > > > > > > > > > > Tang < > > > > > > > > > > > > > > > > > > > myas...@live.com> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar > > > > becomes > > > > > > > more > > > > > > > > > and > > > > > > > > > > more > > > > > > > > > > > > > > > popular > > > > > > > > > > > > > > > > > > > > recently > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people > > > willing > > > > to > > > > > > > > > > contribute to > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > ecosystem. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further > > > > > discussion, > > > > > > > > would > > > > > > > > > > you > > > > > > > > > > > > > please > > > > > > > > > > > > > > > > give > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > > > > > > explanation > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship > > between > > > > this > > > > > > > > thread > > > > > > > > > to > > > > > > > > > > > > > current > > > > > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > > > > JIRAs > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink > [2] > > > > > > > connector? > > > > > > > > > > Will the > > > > > > > > > > > > > > > > > > contribution > > > > > > > > > > > > > > > > > > > > > > contains > > > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally > > > > > different > > > > > > > > > > > > > implementation? > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641 > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168 > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > ________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen < > > > > > > > > > > > > henry.yijies...@gmail.com> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, > > September > > > 3, > > > > > > 2019 > > > > > > > > > 13:57 > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > > dev@flink.apache.org > > > < > > > > > > > > > > > > > dev@flink.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] > > > > Contribute > > > > > > > > Pulsar > > > > > > > > > > Flink > > > > > > > > > > > > > > > > connector > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community! > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open > > the > > > > > > > discussion > > > > > > > > > of > > > > > > > > > > > > > > > contributing > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > Flink > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to > > > > Flink. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief > introduction > > > to > > > > > > Apache > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a > > > > > > > multi-tenant, > > > > > > > > > > > > > > > > high-performance > > > > > > > > > > > > > > > > > > > > > > distributed > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging > > system. > > > > > > Pulsar > > > > > > > > > > includes > > > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for > > > multiple > > > > > > > > clusters > > > > > > > > > > in a > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > instance, > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > > geo-replication > > > of > > > > > > > > messages > > > > > > > > > > across > > > > > > > > > > > > > > > > clusters, > > > > > > > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > > > > > > low > > > > > > > > > > > > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end > latency, > > > > > > seamless > > > > > > > > > > > > scalability > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > over a > > > > > > > > > > > > > > > > > > > > > million > > > > > > > > > > > > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed > message > > > > > delivery > > > > > > > > with > > > > > > > > > > > > > persistent > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > > > > > > storage > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache > > > > > BookKeeper. > > > > > > > > > > Nowadays, > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > > adopted > > > > > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more > > > companies[2]. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of > Pulsar > > > > Flink > > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink > > > connector > > > > we > > > > > > are > > > > > > > > > > planning > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > contribute > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > built > > > > > > > > > > > > > > > > > > > > > > > > > upon > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and > Pulsar > > > > 2.4.0. > > > > > > The > > > > > > > > > main > > > > > > > > > > > > > features > > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a > streaming > > > > > source > > > > > > > with > > > > > > > > > > > > > > exactly-once > > > > > > > > > > > > > > > > > > > guarantee. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming > > results > > > to > > > > > > > Pulsar > > > > > > > > > with > > > > > > > > > > > > > > > > at-least-once > > > > > > > > > > > > > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > > > > (We > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to > > > > > > exactly-once > > > > > > > > as > > > > > > > > > > well > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > gets > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features > > > ready > > > > in > > > > > > its > > > > > > > > > 2.5.0 > > > > > > > > > > > > > > version) > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink > new > > > > Table > > > > > > API > > > > > > > > Type > > > > > > > > > > > > system > > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically > > > (de)serialize > > > > > > > > messages > > > > > > > > > > with > > > > > > > > > > > > the > > > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > Pulsar > > > > > > > > > > > > > > > > > > > > > > > schema. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with > Flink > > > new > > > > > > > Catalog > > > > > > > > > API > > > > > > > > > > > > > > > > (FLIP-30[4]), > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics > as > > > > > tables > > > > > > in > > > > > > > > > > Table API > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > well as > > > > > > > > > > > > > > > > > > > SQL > > > > > > > > > > > > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > > > > > https://pulsar.apache.org/ > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
