My take would be the following:

  - If we merge the connector now and replace it with a FLIP-27 version
before the 1.10 release, then we need no deprecation process
  - If we don't manage to replace it with a FLIP-27 version before the 1.10
release, than it is good that we have the other version, so no users get
blocked.

In the latter case we can see how we want to do it. Immediate removal of
the old version or deprecation label and keeping it for one more release.
Given that you should be able to use a Flink 1.10 connector with Flink 1.11
as well (stable public APIs) there is also a workaround if you need an old
connector in a newer version. So immediate removal might even be feasible.


On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <becket....@gmail.com> wrote:

> Hi Stephan,
>
> Thanks for the clarification. I completely agree with you and Thomas on the
> process of adding connectors to Flink repo. However, I am wondering what is
> the deprecation process? Given the main concern here was that we may have
> to maintain two Pulsar connector code bases until the old one is removed
> from the repo, it would be good to know how long we have to do that.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <se...@apache.org> wrote:
>
> > Some quick thoughts on the connector contribution process. I basically
> > reiterate here what Thomas mentioned in another thread about the Kinesis
> > connector.
> >
> > For connectors, we should favor a low-overhead contribution process, and
> > accept user code and changes more readily than in the core system.
> > That is because connectors have both a big variety of scenarios they get
> > used in (only through use and many small contributions do they become
> > really useful over time) and at the same time, and committers do not use
> > the connector themselves and usually cannot foresee too well what is
> > needed.
> >
> > Further more, a missing connector (or connector feature) is often a
> bigger
> > show stopper for users than a missing API or system feature.
> >
> > Along these lines of thougt, the conclusion would be to take the Pulsar
> > connector now, focus the review on legal/dependencies/rough code style
> and
> > conventions, label it as "beta" (in the sense of "new code" that is "not
> > yet tested through longer use") and go ahead. And then evolve it quickly
> > without putting formal blockers in the way, meaning also adding a new
> FLIP
> > 27 version when it is there.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <becket....@gmail.com> wrote:
> >
> > > Hi Yijie,
> > >
> > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > thread in the mailing list?
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > >
> > > I see two FLIP-69 discussion in the mailing list now. So there is a
> FLIP
> > > number collision. Can you change the FLIP number to 72?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <walter...@gmail.com>
> wrote:
> > >
> > > > Hi Yijie,
> > > >
> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc link?
> > This
> > > > way the contributors from the community can comment on the doc.
> > > >
> > > > Best,
> > > > Rong
> > > >
> > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> henry.yijies...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I've drafted a FLIP that describes the current design of the Pulsar
> > > > > connector:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > >
> > > > > Please take a look and let me know what you think.
> > > > >
> > > > > Thanks,
> > > > > Yijie
> > > > >
> > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <walter...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie
> for
> > > > > driving
> > > > > > the discussion.
> > > > > > I also think the Pulsar connector would be a very valuable
> addition
> > > to
> > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > >
> > > > > > Regarding the timeline, I also share concerns with Becket on the
> > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > There's also another discussion just started by Stephan on
> dropping
> > > > Kafka
> > > > > > 9/10 support on next Flink release [1].  Although the situation
> is
> > > > > somewhat
> > > > > > different, and Kafka 9/10 connector has been in Flink for almost
> > 3-4
> > > > > years,
> > > > > > based on the discussion I am not sure if a major version release
> > is a
> > > > > > requirement for removing old connector supports.
> > > > > >
> > > > > > I think there shouldn't be a blocker if we agree the old
> connector
> > > will
> > > > > be
> > > > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > > > stated,
> > > > > it
> > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > We should also ensure we clearly communicate the message: for
> > > example,
> > > > > > putting an experimental flag on the pre-FLIP27 connector page of
> > the
> > > > > > website, documentations, etc. Any other thoughts?
> > > > > >
> > > > > > --
> > > > > > Rong
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <becket....@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Technically speaking, removing the old connector code is a
> > > backwards
> > > > > > > incompatible change which requires a major version bump, i.e.
> > Flink
> > > > > 2.x.
> > > > > > > Given that we don't have a clear plan on when to have the next
> > > major
> > > > > > > version release, it seems unclear how long the old connector
> code
> > > > will
> > > > > be
> > > > > > > there if we check it in right now. Or will we remove the old
> > > > connector
> > > > > > > without a major version bump? In any case, it sounds not quite
> > user
> > > > > > > friendly to the those who might use the old Pulsar connector. I
> > am
> > > > not
> > > > > sure
> > > > > > > if it is worth these potential problems in order to have the
> > Pulsar
> > > > > source
> > > > > > > connector checked in one or two months earlier.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <se...@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Agreed, if we check in the old code, we should make it clear
> > that
> > > > it
> > > > > will
> > > > > > > > be removed as soon as the FLIP-27 based version of the
> > connector
> > > is
> > > > > > > there.
> > > > > > > > We should not commit to maintaining the old version, that
> would
> > > be
> > > > > indeed
> > > > > > > > too much overhead.
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > becket....@gmail.com
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stephan,
> > > > > > > > >
> > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > >
> > > > > > > > > Yes, the overhead would just be review capacity. In fact, I
> > am
> > > > not
> > > > > > > > worrying
> > > > > > > > > too much about the review capacity. That is just a one time
> > > cost.
> > > > > My
> > > > > > > > > concern is mainly about the long term burden. Assume we
> have
> > > new
> > > > > source
> > > > > > > > > interface ready in 1.10 with newly added Pulsar connectors
> in
> > > old
> > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > interface,
> > > > > the
> > > > > > > old
> > > > > > > > > Pulsar connector might be deprecated almost immediately
> after
> > > > > checked
> > > > > > > in,
> > > > > > > > > but we may still have to maintain two code bases. For the
> > > > existing
> > > > > > > > > connectors, we have to do that anyways. But it would be
> good
> > to
> > > > > avoid
> > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > se...@apache.org
> > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all!
> > > > > > > > > >
> > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > connector.
> > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > >
> > > > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > > > >
> > > > > > > > > > Looks like the discussion is more going towards
> > > contribution. I
> > > > > think
> > > > > > > > > that
> > > > > > > > > > is good, especially if we think that we want to build a
> > > > similarly
> > > > > > > deep
> > > > > > > > > > integration with Pulsar as we have for example with
> Kafka.
> > > The
> > > > > > > > connector
> > > > > > > > > > already looks like a more thorough connector than many
> > others
> > > > we
> > > > > have
> > > > > > > > in
> > > > > > > > > > the repository.
> > > > > > > > > >
> > > > > > > > > > With either a repo split, or the new build system, I hope
> > > that
> > > > > the
> > > > > > > > build
> > > > > > > > > > overhead is not a problem.
> > > > > > > > > >
> > > > > > > > > > ## Committer Support
> > > > > > > > > >
> > > > > > > > > > Becket offered some help already, I can also help a bit.
> I
> > > hope
> > > > > that
> > > > > > > > > > between us, we can cover this.
> > > > > > > > > >
> > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > >
> > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > progress, but
> > > > > > > > > will
> > > > > > > > > > take 2 more months, I would estimate, before it is fully
> > > > > available.
> > > > > > > > > >
> > > > > > > > > > If we want to be on the safe side with the contribution,
> we
> > > > > should
> > > > > > > > > > contribute the source sooner and adjust it later. That
> > would
> > > > also
> > > > > > > help
> > > > > > > > us
> > > > > > > > > > in case things get crazy towards the 1.10 feature freeze
> > and
> > > it
> > > > > would
> > > > > > > > be
> > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > What would be the overhead of contributing now? Given
> that
> > > the
> > > > > code
> > > > > > > is
> > > > > > > > > > already there, it looks like it would be only review
> > > capacity,
> > > > > right?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > henry.yijies...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone!
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your attention and the promotion of this
> work.
> > > > > > > > > > >
> > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > specific
> > > > > > > > > discussions.
> > > > > > > > > > >
> > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > consensus.
> > > > > > > > Therefore,
> > > > > > > > > > > I will explain all the functionalities of the existing
> > > > > connector in
> > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > continue
> > > > our
> > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yijie
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > becket....@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > >
> > > > > > > > > > > > If we agree that the goal is to have Pulsar connector
> > in
> > > > > 1.10,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > we
> > > > > > > > > > > > do the following:
> > > > > > > > > > > >
> > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> > > repo
> > > > > as it
> > > > > > > > is a
> > > > > > > > > > new
> > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> there
> > is
> > > > no
> > > > > > > change
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going
> to
> > > be
> > > > > code
> > > > > > > > > freeze
> > > > > > > > > > in
> > > > > > > > > > > > late Nov and let's say we give a month to the
> > development
> > > > and
> > > > > > > > review
> > > > > > > > > of
> > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> Oct.
> > > > There
> > > > > are
> > > > > > > > > still
> > > > > > > > > > 7
> > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is
> > not
> > > > > ready
> > > > > > > by
> > > > > > > > > late
> > > > > > > > > > > > Oct, we can review and check in Pulsar connector with
> > the
> > > > > > > existing
> > > > > > > > > > source
> > > > > > > > > > > > interface. This means we will have Pulsar connector
> in
> > > > Flink
> > > > > > > 1.10,
> > > > > > > > > > either
> > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > >
> > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > checked
> > > > > in
> > > > > > > > > > > separately,
> > > > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> > > sink
> > > > > and
> > > > > > > > > another
> > > > > > > > > > > for
> > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar
> sink
> > > > right
> > > > > > > away.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > guosi...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What's the take from Flink community? Shall we wait
> > for
> > > > > FLIP-27
> > > > > > > > or
> > > > > > > > > > > shall we
> > > > > > > > > > > > > proceed to next steps? And what the next steps are?
> > :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Sijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > bowenl...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think having a Pulsar connector in Flink can
> be a
> > > > good
> > > > > > > mutual
> > > > > > > > > > > benefit
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Another perspective is that Pulsar connector is
> the
> > > 1st
> > > > > > > > streaming
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > that integrates with Flink's metadata management
> > > system
> > > > > and
> > > > > > > > > Catalog
> > > > > > > > > > > APIs.
> > > > > > > > > > > > > > It'll be cool to see how the integration turns
> out
> > > and
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > > need to
> > > > > > > > > > > > > > improve Flink Catalog stack, which are currently
> in
> > > > > Beta, to
> > > > > > > > > cater
> > > > > > > > > > to
> > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> merging
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > into
> > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> maybe
> > > one
> > > > > for
> > > > > > > > basic
> > > > > > > > > > > > > > source/sink functionalities and another for
> schema
> > > and
> > > > > > > catalog
> > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But
> I
> > > > don't
> > > > > > > think
> > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > should be a blocker in cases where it cannot make
> > its
> > > > way
> > > > > > > into
> > > > > > > > > 1.10
> > > > > > > > > > > or
> > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > committers
> > > > to
> > > > > > > > review
> > > > > > > > > or
> > > > > > > > > > > for
> > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> interfaces.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > becket....@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > source
> > > > > > > > interface
> > > > > > > > > is
> > > > > > > > > > > going
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be ready. Personally I think it would be there
> in
> > > > > about a
> > > > > > > > month
> > > > > > > > > > or
> > > > > > > > > > > so.
> > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > I could be too optimistic. It would also be
> good
> > to
> > > > > hear
> > > > > > > what
> > > > > > > > > do
> > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > and Stephan think as they are also involved in
> > > > FLIP-27.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In general I think we should have Pulsar
> > connector
> > > in
> > > > > Flink
> > > > > > > > > 1.10,
> > > > > > > > > > > > > > > preferably with the new source interface. We
> can
> > > also
> > > > > check
> > > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > right
> > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > with old source interface, but I suspect few
> > users
> > > > > will use
> > > > > > > > it
> > > > > > > > > > > before
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > reasonable
> > > > > to
> > > > > > > > wait a
> > > > > > > > > > > little
> > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > interface. As
> > > > > > > > long
> > > > > > > > > > as
> > > > > > > > > > > we
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > doesn't
> > > > > seem
> > > > > > > to
> > > > > > > > > hurt
> > > > > > > > > > > much.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > > > trohrm...@apache.org
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > > > committed
> > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector before the new source interface is
> > > ready.
> > > > > If I
> > > > > > > > > > > understood
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > correctly, then we need to support the old
> > source
> > > > > > > interface
> > > > > > > > > > > anyway
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > existing connectors. By checking it in early
> I
> > > > could
> > > > > see
> > > > > > > > the
> > > > > > > > > > > benefit
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > our users could start using the connector
> > > earlier.
> > > > > > > > Moreover,
> > > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > > > delayed
> > > > > in
> > > > > > > > case
> > > > > > > > > > > that the
> > > > > > > > > > > > > > > > source interface should be delayed. The only
> > > > > downside I
> > > > > > > see
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > review effort and potential fixes which might
> > be
> > > > > > > irrelevant
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > mainly
> > > > > > > depends
> > > > > > > > on
> > > > > > > > > > how
> > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > we are when the new source interface will be
> > > ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > Although
> > > > > the
> > > > > > > > FLIP
> > > > > > > > > > > wiki
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > time, a
> > > > > few
> > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > contributors in Flink community were
> actually
> > > > > > > prototyping
> > > > > > > > > the
> > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > We have made some good progress there but
> > want
> > > to
> > > > > > > update
> > > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > after the entire thing is verified to work
> in
> > > > case
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > some
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > don't
> > > > > have an
> > > > > > > > > exact
> > > > > > > > > > > ETA
> > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > guess it is going to be within a month or
> so.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I am happy to review the current Flink
> Pulsar
> > > > > connector
> > > > > > > > and
> > > > > > > > > > > see if
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid
> the
> > > > case
> > > > > that
> > > > > > > > we
> > > > > > > > > > > checked
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar connector with some review efforts
> and
> > > > > shortly
> > > > > > > > after
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > > > henry.yijies...@gmail.com
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > suggestions!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As Sijie said, the goal of the connector
> > has
> > > > > always
> > > > > > > > been
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > users with the latest features of both
> > > systems
> > > > as
> > > > > > > soon
> > > > > > > > as
> > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > Flink
> > > > and
> > > > > hope
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> experts
> > > to
> > > > > ensure
> > > > > > > > the
> > > > > > > > > > > high
> > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at
> > the
> > > > > > > beginning
> > > > > > > > of
> > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > the connector implementation based on
> Flink
> > > > 1.9;
> > > > > we
> > > > > > > > also
> > > > > > > > > > > wanted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > build a connector that supports both
> batch
> > > and
> > > > > stream
> > > > > > > > > > > computing
> > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > time,
> > > so
> > > > > we
> > > > > > > > > decided
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > a connector with most of the new
> features,
> > > such
> > > > > as
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > type
> > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > and the new catalog API first. We will
> pay
> > > > > attention
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> > > with
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > as
> > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regarding the test status of the
> connector,
> > > we
> > > > > are
> > > > > > > > > > following
> > > > > > > > > > > the
> > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > connectors' test in Flink repository and
> > > aimed
> > > > to
> > > > > > > > provide
> > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > hear
> > > > > > > > suggestions
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > improve
> > > > > the
> > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > performance of the connector
> continuously.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo
> <
> > > > > > > > > > guosi...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > feedback.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > here
> > > is
> > > > > > > about -
> > > > > > > > > > "how
> > > > > > > > > > > can
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > integration
> > > > > > > is
> > > > > > > > > kind
> > > > > > > > > > > of
> > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > here. I believe there can be many ways
> to
> > > > > achieve
> > > > > > > it.
> > > > > > > > > > > > > > > > > > > As part of the contribution, what we
> are
> > > > > looking
> > > > > > > for
> > > > > > > > > here
> > > > > > > > > > > is
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > > > relationship
> > > > > > > > on
> > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> Flink.
> > > > Even
> > > > > we
> > > > > > > can
> > > > > > > > > try
> > > > > > > > > > > our
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > up all the updates in Flink community.
> We
> > > are
> > > > > still
> > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > experiences
> > > > > in
> > > > > > > > Flink
> > > > > > > > > > than
> > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > maintain
> > > > > and
> > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration
> > to
> > > > the
> > > > > > > users
> > > > > > > > > who
> > > > > > > > > > > use
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > technologies, we need some help from
> the
> > > > > experts
> > > > > > > from
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > > > while.
> > > > > > > > > Originally
> > > > > > > > > > > we
> > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > of contributing the connectors back
> after
> > > > > > > integrating
> > > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > decided
> > > > > to
> > > > > > > > > initiate
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > as early as possible. Because we
> believe
> > > > there
> > > > > are
> > > > > > > > more
> > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > contribution,
> > > > > > > it
> > > > > > > > > can
> > > > > > > > > > > help
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community understand more about Pulsar
> > and
> > > > the
> > > > > > > > > potential
> > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > verify
> > > > > the
> > > > > > > new
> > > > > > > > > > > connector
> > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > Qin <
> > > > > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> contributing
> > > the
> > > > > > > Pulsar
> > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > connector
> > > > > with
> > > > > > > > > strong
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > happy
> > > > the
> > > > > > > > > shepherd
> > > > > > > > > > > this
> > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> provide
> > > > some
> > > > > > > > context
> > > > > > > > > > and
> > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > connector has
> > > > > > > > hit
> > > > > > > > > > the
> > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > With more and more connectors coming
> > into
> > > > > Flink
> > > > > > > > repo,
> > > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > problems such as long build and
> testing
> > > > > time. To
> > > > > > > > > > address
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > separate
> > > > > > > > > repository.
> > > > > > > > > > > This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > > > shorten
> > > > > the
> > > > > > > > > build
> > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay
> as
> > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > to provide good support for
> > functionality
> > > > and
> > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> Ecosystem
> > > > > project
> > > > > > > > > website
> > > > > > > > > > > and it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> > > great
> > > > to
> > > > > > > first
> > > > > > > > > see
> > > > > > > > > > > if we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > project
> > > > with
> > > > > > > great
> > > > > > > > > > > support.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > connector
> > > > > is
> > > > > > > > tested
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > we can learn something to maintain it
> > as
> > > an
> > > > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > quality
> > > > as
> > > > > an
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> > > into
> > > > > the
> > > > > > > main
> > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> FLIP-27
> > > > where
> > > > > we
> > > > > > > are
> > > > > > > > > > > making
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Flink source connector architecture
> and
> > > > > > > interface.
> > > > > > > > > This
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise,
> if
> > > we
> > > > > are
> > > > > > > > going
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> wondering
> > if
> > > > we
> > > > > > > should
> > > > > > > > > > hold
> > > > > > > > > > > a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the
> > new
> > > > > > > interface
> > > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> Chesnay
> > > > > Schepler <
> > > > > > > > > > > > > > > > ches...@apache.org>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end
> up
> > > > > repeating
> > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > contributing a
> > > > > > > > > > pulsar
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of which failed because no
> committer
> > > was
> > > > > > > getting
> > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > discussion
> > > > > > > thread
> > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> > > from
> > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > We should really make sure that if
> we
> > > > > > > > > welcome/approve
> > > > > > > > > > > such
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > contribution it will actually get
> the
> > > > > attention
> > > > > > > > it
> > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > > > maintaining
> > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > > > documentation
> > > > > > > > > to
> > > > > > > > > > > give
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > > > artifacts
> > > > > > > > among
> > > > > > > > > > the
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this
> may
> > > be a
> > > > > > > better
> > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > discussion
> > > > > > > > Yijie.
> > > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector would be a very
> valuable
> > > > > addition
> > > > > > > > since
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > more popular and it would further
> > > > expand
> > > > > > > > Flink's
> > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> makes
> > > > > sense for
> > > > > > > > me
> > > > > > > > > to
> > > > > > > > > > > place
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > My main concern/question is how
> can
> > > the
> > > > > Flink
> > > > > > > > > > > community
> > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the
> past
> > > > that
> > > > > > > > > connectors
> > > > > > > > > > > are
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > actively developed components
> > because
> > > > > they
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > kept
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > Given
> > > > > that
> > > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> improving
> > > and
> > > > > > > > evolving
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > this.
> > > > > Hence,
> > > > > > > +1
> > > > > > > > > for
> > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > Sijie
> > > > Guo
> > > > > <
> > > > > > > > > > > > > > guosi...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> behind
> > > > > > > FLINK-9641
> > > > > > > > > and
> > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > created
> > > > > for
> > > > > > > > > bringing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration
> > was
> > > > done
> > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> > > ended
> > > > > up
> > > > > > > > > > > maintaining
> > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > Flink
> > > > to
> > > > > > > > process
> > > > > > > > > > > event
> > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > >
> > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple
> > and
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> In the past year, we have made a
> > lot
> > > > of
> > > > > > > > changes
> > > > > > > > > in
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > > > citizen in
> > > > > > > > > > Pulsar.
> > > > > > > > > > > We
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > processing
> > > > > > > Pulsar
> > > > > > > > > > event
> > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > integrate
> > > > > with
> > > > > > > > Flink
> > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > connectors
> > > > > from
> > > > > > > > the
> > > > > > > > > > > ground
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog API
> as
> > > the
> > > > > > > > > first-class
> > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> said,
> > > in
> > > > > the
> > > > > > > new
> > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> flink
> > > > > catalog
> > > > > > > and
> > > > > > > > > > > query /
> > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> This is an example about how to
> > use
> > > > > Pulsar
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog
> post
> > > > > > > explaining
> > > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> what
> > > are
> > > > > the
> > > > > > > > > changes
> > > > > > > > > > we
> > > > > > > > > > > > > made
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> > > simple
> > > > > data
> > > > > > > > sink
> > > > > > > > > > or
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> integrated
> > > > > streaming
> > > > > > > > > data
> > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > schema/catalog
> > > > and
> > > > > > > > state).
> > > > > > > > > > The
> > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > streaming
> > > > > > > > > warehouse
> > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > processing.
> > > > > > > > Since
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> > > Flink
> > > > > here,
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > > > integration
> > > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > background
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what
> are
> > > your
> > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM
> > Yun
> > > > > Tang <
> > > > > > > > > > > > > > myas...@live.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > more
> > > > and
> > > > > more
> > > > > > > > > > popular
> > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > contribute to
> > > > > > > > > Flink
> > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> > > would
> > > > > you
> > > > > > > > please
> > > > > > > > > > > give
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > thread
> > > > to
> > > > > > > > current
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > connector?
> > > > > Will the
> > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > >>>
> ________________________________
> > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > henry.yijies...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> 2019
> > > > 13:57
> > > > > > > > > > > > > > > > > > > > > >>> To: dev@flink.apache.org <
> > > > > > > > dev@flink.apache.org
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > Pulsar
> > > > > Flink
> > > > > > > > > > > connector
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > discussion
> > > > of
> > > > > > > > > > contributing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> Apache
> > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > multi-tenant,
> > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> Pulsar
> > > > > includes
> > > > > > > > > > multiple
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > clusters
> > > > > in a
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > messages
> > > > > across
> > > > > > > > > > > clusters,
> > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> seamless
> > > > > > > scalability
> > > > > > > > to
> > > > > > > > > > > over a
> > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> > > with
> > > > > > > > persistent
> > > > > > > > > > > message
> > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > > > Nowadays,
> > > > > > > > Pulsar
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > connector
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we
> are
> > > > > planning
> > > > > > > to
> > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0.
> The
> > > > main
> > > > > > > > features
> > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source
> > with
> > > > > > > > > exactly-once
> > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > Pulsar
> > > > with
> > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > >>> would update this to
> exactly-once
> > > as
> > > > > well
> > > > > > > > when
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > >>> transaction features ready in
> its
> > > > 2.5.0
> > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table
> API
> > > Type
> > > > > > > system
> > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > messages
> > > > > with
> > > > > > > the
> > > > > > > > > > help
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > Catalog
> > > > API
> > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables
> in
> > > > > Table API
> > > > > > > > as
> > > > > > > > > > > well as
> > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to