Re: [DISCUSS] Streaming connector contributions

[Robert Metzger]

Okay, I agree that this is something we should make very clear.

I'll start a DISCUSS thread.

On Wed, Aug 17, 2016 at 10:07 AM, Stephan Ewen  wrote:

> Cool to see that the Bahir people are so open and helpful!
>
> Concerning moving the connectors: I think we should set up a vote thread,
> or at least a discussion with the possibility to object.
> Removing someones code from the repository is a bit of a sensitive thing in
> my experience, so let's make sure everybody is behind that decision before
> we proceed.
>
> On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger 
> wrote:
>
> > Bahir is now creating a second repository "bahir-flink" for our
> connectors:
> > https://issues.apache.org/jira/browse/INFRA-12440
> >
> > If there are no objections here on the dev list, I would like to move the
> > "flume" and "redis" streaming connector to Bahir.
> > Once they are in, and the file structure at Bahir exists, I'll update our
> > contribution guidelines to point people to Bahir.
> >
> > On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger 
> > wrote:
> >
> > > Hi,
> > >
> > > I just wanted to let you know that I've started a discussion at the
> Bahir
> > > dev list [1]. They seem to be very open to give some of our streaming
> > > connectors a new home.
> > > We are currently discussing some specifics there (whether to put the
> code
> > > into the same repository or into separate ones). Also, they asked for
> > some
> > > help by Flink committers / contributors to help with the reviewing of
> > > incoming contributions.
> > > I'm definitively willing to help out to give the community there a good
> > > start.
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
> > >
> > >
> > > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels 
> > > wrote:
> > >
> > >> Hi Robert,
> > >>
> > >> We had this discussion before when I suggested to use an external
> > >> repository to manage connectors. Ever since I have come to the
> > >> conclusion that the overhead of maintaining two source repositories
> > >> along with maintaining code and integration, documentation, and CI, is
> > >> not worth the effort. Thanks for bringing up the discussion again.
> > >>
> > >> I think the most important issues with moving connectors out of the
> > >> core repository are
> > >>
> > >> 1) Maintenance
> > >> Connectors need to be maintained. Otherwise, they are worthless.
> > >>
> > >> 2) Plugability
> > >> Connectors need to be easy to use for the user. Otherwise, nobody will
> > >> use them.
> > >>
> > >> What is the difference between #1, #3 and #4? I don't see a difference
> > >> except that #3 and #4 are central repositories. Still, maintenance and
> > >> plugability are very unclear.
> > >>
> > >> Only #2 is left :) Apache Bahir looks like a promising project. Let's
> > >> see how the community reacts.
> > >>
> > >> Cheers,
> > >> Max
> > >>
> > >>
> > >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger  >
> > >> wrote:
> > >> > Thank you for your responses. I will get in touch with the Bahir
> > >> community
> > >> > to see what they are thinking about this.
> > >> > Once we know a bit more about the details of such a collaboration,
> we
> > >> can
> > >> > make a final decision here.
> > >> >
> > >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann  >
> > >> wrote:
> > >> >
> > >> >> I agree with Stephan that the main problem is maintenance overhead
> > for
> > >> the
> > >> >> Flink community. If we could maintain all connectors ourselves then
> > >> there
> > >> >> would not be an immediate need to out source the connectors. Thus,
> > the
> > >> >> solution should reduce the workload for the core project.
> > >> >>
> > >> >> Personally, I would favour option #2 if Apache Bahir is willing to
> > add
> > >> >> Flink as another supported streaming platform. This would give the
> > >> >> streaming connectors a more prominent home than the personal
> > >> repository of
> > >> >> a contributor.
> > >> >>
> > >> >> Furthermore, contributing the connectors to Apache Bahir entails
> that
> > >> the
> > >> >> connector artifacts are deployed to a central repository (I
> assume).
> > >> That
> > >> >> way one can more easily add them to one's project instead of having
> > to
> > >> >> build and install the code yourself.
> > >> >>
> > >> >> I'm also not sure what happens to a public github repository which
> > has
> > >> not
> > >> >> been forked and is then deleted. I would assume that the content is
> > >> then
> > >> >> lost.
> > >> >>
> > >> >> To create a "flink-connectors" repository independent of Flink
> sounds
> > >> quite
> > >> >> similar to creating another Apache Bahir. I think it would be
> better
> > to
> > >> >> join forces with the existing Bahir community if possible.
> > >> >>
> > >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen 
> > wrote:
> > >> >>
> > >> 

Re: [DISCUSS] Streaming connector contributions

[Stephan Ewen]

Cool to see that the Bahir people are so open and helpful!

Concerning moving the connectors: I think we should set up a vote thread,
or at least a discussion with the possibility to object.
Removing someones code from the repository is a bit of a sensitive thing in
my experience, so let's make sure everybody is behind that decision before
we proceed.

On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger  wrote:

> Bahir is now creating a second repository "bahir-flink" for our connectors:
> https://issues.apache.org/jira/browse/INFRA-12440
>
> If there are no objections here on the dev list, I would like to move the
> "flume" and "redis" streaming connector to Bahir.
> Once they are in, and the file structure at Bahir exists, I'll update our
> contribution guidelines to point people to Bahir.
>
> On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger 
> wrote:
>
> > Hi,
> >
> > I just wanted to let you know that I've started a discussion at the Bahir
> > dev list [1]. They seem to be very open to give some of our streaming
> > connectors a new home.
> > We are currently discussing some specifics there (whether to put the code
> > into the same repository or into separate ones). Also, they asked for
> some
> > help by Flink committers / contributors to help with the reviewing of
> > incoming contributions.
> > I'm definitively willing to help out to give the community there a good
> > start.
> >
> >
> > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
> >
> >
> > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels 
> > wrote:
> >
> >> Hi Robert,
> >>
> >> We had this discussion before when I suggested to use an external
> >> repository to manage connectors. Ever since I have come to the
> >> conclusion that the overhead of maintaining two source repositories
> >> along with maintaining code and integration, documentation, and CI, is
> >> not worth the effort. Thanks for bringing up the discussion again.
> >>
> >> I think the most important issues with moving connectors out of the
> >> core repository are
> >>
> >> 1) Maintenance
> >> Connectors need to be maintained. Otherwise, they are worthless.
> >>
> >> 2) Plugability
> >> Connectors need to be easy to use for the user. Otherwise, nobody will
> >> use them.
> >>
> >> What is the difference between #1, #3 and #4? I don't see a difference
> >> except that #3 and #4 are central repositories. Still, maintenance and
> >> plugability are very unclear.
> >>
> >> Only #2 is left :) Apache Bahir looks like a promising project. Let's
> >> see how the community reacts.
> >>
> >> Cheers,
> >> Max
> >>
> >>
> >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger 
> >> wrote:
> >> > Thank you for your responses. I will get in touch with the Bahir
> >> community
> >> > to see what they are thinking about this.
> >> > Once we know a bit more about the details of such a collaboration, we
> >> can
> >> > make a final decision here.
> >> >
> >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann 
> >> wrote:
> >> >
> >> >> I agree with Stephan that the main problem is maintenance overhead
> for
> >> the
> >> >> Flink community. If we could maintain all connectors ourselves then
> >> there
> >> >> would not be an immediate need to out source the connectors. Thus,
> the
> >> >> solution should reduce the workload for the core project.
> >> >>
> >> >> Personally, I would favour option #2 if Apache Bahir is willing to
> add
> >> >> Flink as another supported streaming platform. This would give the
> >> >> streaming connectors a more prominent home than the personal
> >> repository of
> >> >> a contributor.
> >> >>
> >> >> Furthermore, contributing the connectors to Apache Bahir entails that
> >> the
> >> >> connector artifacts are deployed to a central repository (I assume).
> >> That
> >> >> way one can more easily add them to one's project instead of having
> to
> >> >> build and install the code yourself.
> >> >>
> >> >> I'm also not sure what happens to a public github repository which
> has
> >> not
> >> >> been forked and is then deleted. I would assume that the content is
> >> then
> >> >> lost.
> >> >>
> >> >> To create a "flink-connectors" repository independent of Flink sounds
> >> quite
> >> >> similar to creating another Apache Bahir. I think it would be better
> to
> >> >> join forces with the existing Bahir community if possible.
> >> >>
> >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen 
> wrote:
> >> >>
> >> >> > Thanks Robert for bringing this up.
> >> >> >
> >> >> > I think that "flink-contrib" will not really solve the problem,
> >> because
> >> >> the
> >> >> > connectors would still have to be maintained by the core community,
> >> we
> >> >> > would need to guarantee test stability. It will be to a large
> extend
> >> >> > similar to adding them to "flink-streaming-connectors", except that
> >> we
> >> >> may
> >> >> > declare via 

Re: [DISCUSS] Streaming connector contributions

[Robert Metzger]

Bahir is now creating a second repository "bahir-flink" for our connectors:
https://issues.apache.org/jira/browse/INFRA-12440

If there are no objections here on the dev list, I would like to move the
"flume" and "redis" streaming connector to Bahir.
Once they are in, and the file structure at Bahir exists, I'll update our
contribution guidelines to point people to Bahir.

On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger  wrote:

> Hi,
>
> I just wanted to let you know that I've started a discussion at the Bahir
> dev list [1]. They seem to be very open to give some of our streaming
> connectors a new home.
> We are currently discussing some specifics there (whether to put the code
> into the same repository or into separate ones). Also, they asked for some
> help by Flink committers / contributors to help with the reviewing of
> incoming contributions.
> I'm definitively willing to help out to give the community there a good
> start.
>
>
> [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
>
>
> On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels 
> wrote:
>
>> Hi Robert,
>>
>> We had this discussion before when I suggested to use an external
>> repository to manage connectors. Ever since I have come to the
>> conclusion that the overhead of maintaining two source repositories
>> along with maintaining code and integration, documentation, and CI, is
>> not worth the effort. Thanks for bringing up the discussion again.
>>
>> I think the most important issues with moving connectors out of the
>> core repository are
>>
>> 1) Maintenance
>> Connectors need to be maintained. Otherwise, they are worthless.
>>
>> 2) Plugability
>> Connectors need to be easy to use for the user. Otherwise, nobody will
>> use them.
>>
>> What is the difference between #1, #3 and #4? I don't see a difference
>> except that #3 and #4 are central repositories. Still, maintenance and
>> plugability are very unclear.
>>
>> Only #2 is left :) Apache Bahir looks like a promising project. Let's
>> see how the community reacts.
>>
>> Cheers,
>> Max
>>
>>
>> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger 
>> wrote:
>> > Thank you for your responses. I will get in touch with the Bahir
>> community
>> > to see what they are thinking about this.
>> > Once we know a bit more about the details of such a collaboration, we
>> can
>> > make a final decision here.
>> >
>> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann 
>> wrote:
>> >
>> >> I agree with Stephan that the main problem is maintenance overhead for
>> the
>> >> Flink community. If we could maintain all connectors ourselves then
>> there
>> >> would not be an immediate need to out source the connectors. Thus, the
>> >> solution should reduce the workload for the core project.
>> >>
>> >> Personally, I would favour option #2 if Apache Bahir is willing to add
>> >> Flink as another supported streaming platform. This would give the
>> >> streaming connectors a more prominent home than the personal
>> repository of
>> >> a contributor.
>> >>
>> >> Furthermore, contributing the connectors to Apache Bahir entails that
>> the
>> >> connector artifacts are deployed to a central repository (I assume).
>> That
>> >> way one can more easily add them to one's project instead of having to
>> >> build and install the code yourself.
>> >>
>> >> I'm also not sure what happens to a public github repository which has
>> not
>> >> been forked and is then deleted. I would assume that the content is
>> then
>> >> lost.
>> >>
>> >> To create a "flink-connectors" repository independent of Flink sounds
>> quite
>> >> similar to creating another Apache Bahir. I think it would be better to
>> >> join forces with the existing Bahir community if possible.
>> >>
>> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen  wrote:
>> >>
>> >> > Thanks Robert for bringing this up.
>> >> >
>> >> > I think that "flink-contrib" will not really solve the problem,
>> because
>> >> the
>> >> > connectors would still have to be maintained by the core community,
>> we
>> >> > would need to guarantee test stability. It will be to a large extend
>> >> > similar to adding them to "flink-streaming-connectors", except that
>> we
>> >> may
>> >> > declare via the artifact name that the quality is not guaranteed to
>> be
>> >> too
>> >> > high.
>> >> >
>> >> > Regarding moving the code to a separate repository: I think that also
>> >> only
>> >> > solves the problem, if that repository is organizationally different
>> from
>> >> > the core clink repository. Different release cycles, different
>> >> maintainers.
>> >> >
>> >> >
>> >> > The quintessence is for me that the connectors need to be handled by
>> a
>> >> > different organization than the core flink community.
>> >> > That would leave us with
>> >> >   - The contributors' own repository
>> >> >   - Apache Bahir
>> >> >   - An organizationally different 

Re: [DISCUSS] Streaming connector contributions

[Robert Metzger]

Hi,

I just wanted to let you know that I've started a discussion at the Bahir
dev list [1]. They seem to be very open to give some of our streaming
connectors a new home.
We are currently discussing some specifics there (whether to put the code
into the same repository or into separate ones). Also, they asked for some
help by Flink committers / contributors to help with the reviewing of
incoming contributions.
I'm definitively willing to help out to give the community there a good
start.


[1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html


On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels  wrote:

> Hi Robert,
>
> We had this discussion before when I suggested to use an external
> repository to manage connectors. Ever since I have come to the
> conclusion that the overhead of maintaining two source repositories
> along with maintaining code and integration, documentation, and CI, is
> not worth the effort. Thanks for bringing up the discussion again.
>
> I think the most important issues with moving connectors out of the
> core repository are
>
> 1) Maintenance
> Connectors need to be maintained. Otherwise, they are worthless.
>
> 2) Plugability
> Connectors need to be easy to use for the user. Otherwise, nobody will use
> them.
>
> What is the difference between #1, #3 and #4? I don't see a difference
> except that #3 and #4 are central repositories. Still, maintenance and
> plugability are very unclear.
>
> Only #2 is left :) Apache Bahir looks like a promising project. Let's
> see how the community reacts.
>
> Cheers,
> Max
>
>
> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger 
> wrote:
> > Thank you for your responses. I will get in touch with the Bahir
> community
> > to see what they are thinking about this.
> > Once we know a bit more about the details of such a collaboration, we can
> > make a final decision here.
> >
> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann 
> wrote:
> >
> >> I agree with Stephan that the main problem is maintenance overhead for
> the
> >> Flink community. If we could maintain all connectors ourselves then
> there
> >> would not be an immediate need to out source the connectors. Thus, the
> >> solution should reduce the workload for the core project.
> >>
> >> Personally, I would favour option #2 if Apache Bahir is willing to add
> >> Flink as another supported streaming platform. This would give the
> >> streaming connectors a more prominent home than the personal repository
> of
> >> a contributor.
> >>
> >> Furthermore, contributing the connectors to Apache Bahir entails that
> the
> >> connector artifacts are deployed to a central repository (I assume).
> That
> >> way one can more easily add them to one's project instead of having to
> >> build and install the code yourself.
> >>
> >> I'm also not sure what happens to a public github repository which has
> not
> >> been forked and is then deleted. I would assume that the content is then
> >> lost.
> >>
> >> To create a "flink-connectors" repository independent of Flink sounds
> quite
> >> similar to creating another Apache Bahir. I think it would be better to
> >> join forces with the existing Bahir community if possible.
> >>
> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen  wrote:
> >>
> >> > Thanks Robert for bringing this up.
> >> >
> >> > I think that "flink-contrib" will not really solve the problem,
> because
> >> the
> >> > connectors would still have to be maintained by the core community, we
> >> > would need to guarantee test stability. It will be to a large extend
> >> > similar to adding them to "flink-streaming-connectors", except that we
> >> may
> >> > declare via the artifact name that the quality is not guaranteed to be
> >> too
> >> > high.
> >> >
> >> > Regarding moving the code to a separate repository: I think that also
> >> only
> >> > solves the problem, if that repository is organizationally different
> from
> >> > the core clink repository. Different release cycles, different
> >> maintainers.
> >> >
> >> >
> >> > The quintessence is for me that the connectors need to be handled by a
> >> > different organization than the core flink community.
> >> > That would leave us with
> >> >   - The contributors' own repository
> >> >   - Apache Bahir
> >> >   - An organizationally different "flink-connectors" repository,
> possibly
> >> > with different committers / PMC
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai <
> tzuli...@gmail.com>
> >> > wrote:
> >> >
> >> > > Good point. Discussion for each new connector is also an overhead we
> >> > should
> >> > > avoid.
> >> > >
> >> > > IMHO, option #2 doesn’t seem like a reasonable staging area. Say we
> >> > decide
> >> > > we’d like to move a connector from Bahir to Flink in the future,
> >> there’ll
> >> > > be two of the connector in separate Apache projects. Personally I
> think
> >> > > option #2 

Re: [DISCUSS] Streaming connector contributions

[Robert Metzger]

Thank you for your responses. I will get in touch with the Bahir community
to see what they are thinking about this.
Once we know a bit more about the details of such a collaboration, we can
make a final decision here.

On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann  wrote:

> I agree with Stephan that the main problem is maintenance overhead for the
> Flink community. If we could maintain all connectors ourselves then there
> would not be an immediate need to out source the connectors. Thus, the
> solution should reduce the workload for the core project.
>
> Personally, I would favour option #2 if Apache Bahir is willing to add
> Flink as another supported streaming platform. This would give the
> streaming connectors a more prominent home than the personal repository of
> a contributor.
>
> Furthermore, contributing the connectors to Apache Bahir entails that the
> connector artifacts are deployed to a central repository (I assume). That
> way one can more easily add them to one's project instead of having to
> build and install the code yourself.
>
> I'm also not sure what happens to a public github repository which has not
> been forked and is then deleted. I would assume that the content is then
> lost.
>
> To create a "flink-connectors" repository independent of Flink sounds quite
> similar to creating another Apache Bahir. I think it would be better to
> join forces with the existing Bahir community if possible.
>
> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen  wrote:
>
> > Thanks Robert for bringing this up.
> >
> > I think that "flink-contrib" will not really solve the problem, because
> the
> > connectors would still have to be maintained by the core community, we
> > would need to guarantee test stability. It will be to a large extend
> > similar to adding them to "flink-streaming-connectors", except that we
> may
> > declare via the artifact name that the quality is not guaranteed to be
> too
> > high.
> >
> > Regarding moving the code to a separate repository: I think that also
> only
> > solves the problem, if that repository is organizationally different from
> > the core clink repository. Different release cycles, different
> maintainers.
> >
> >
> > The quintessence is for me that the connectors need to be handled by a
> > different organization than the core flink community.
> > That would leave us with
> >   - The contributors' own repository
> >   - Apache Bahir
> >   - An organizationally different "flink-connectors" repository, possibly
> > with different committers / PMC
> >
> >
> >
> >
> >
> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai 
> > wrote:
> >
> > > Good point. Discussion for each new connector is also an overhead we
> > should
> > > avoid.
> > >
> > > IMHO, option #2 doesn’t seem like a reasonable staging area. Say we
> > decide
> > > we’d like to move a connector from Bahir to Flink in the future,
> there’ll
> > > be two of the connector in separate Apache projects. Personally I think
> > > option #2 is more like a way to go if some day we’d like to migrate
> some
> > of
> > > our currently supported connectors in Flink to Bahir (perhaps an
> > > alternative to the discussion in "Externalizing the Flink connectors”).
> > >
> > > Defining option #1 as a staging area seems nice; the contributor
> notifies
> > > the Flink community of the work by adding it to the 3rd party packages
> > > page, and in the future we can contact the contributor to ask whether
> > > they’d like to migrate the connector to Flink when we see more user
> > demand
> > > and committer support.
> > >
> > > With this approach, do you think we should assume that future new
> > > connectors always go to option #1 first?   If not, I would assume that
> in
> > > the future, Flink JIRAs for new connectors are only opened if we’d like
> > to
> > > add them directly to Flink (perhaps the contribution guideline can link
> > to
> > > Flink development Roadmaps like [1] as a reference to connectors that
> the
> > > community has already considered to support?).
> > >
> > > [1]
> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
> > > 778DAD7eqw5GANwE/edit#
> > >  > > 778DAD7eqw5GANwE/edit>
> > >
> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger (rmetz...@apache.org)
> > > wrote:
> > >
> > > Hi Gordon,
> > > thank you for your response.
> > >
> > > I agree with your observation that some "staging" area is helpful to
> test
> > > how many contributors / users are interested in a connector. But I
> wonder
> > > if #1 or #2 can also serve as a staging area: As soon as we see that
> > there
> > > is a lot of interest in a connector, we add it to the main
> > > "flink-*-connectors" directory.
> > > Having too many ways to contribute connectors might make the
> discussions
> > > for each new connector quite complicated.
> > >
> > > You are right, the intention of 

Re: [DISCUSS] Streaming connector contributions

[Till Rohrmann]

I agree with Stephan that the main problem is maintenance overhead for the
Flink community. If we could maintain all connectors ourselves then there
would not be an immediate need to out source the connectors. Thus, the
solution should reduce the workload for the core project.

Personally, I would favour option #2 if Apache Bahir is willing to add
Flink as another supported streaming platform. This would give the
streaming connectors a more prominent home than the personal repository of
a contributor.

Furthermore, contributing the connectors to Apache Bahir entails that the
connector artifacts are deployed to a central repository (I assume). That
way one can more easily add them to one's project instead of having to
build and install the code yourself.

I'm also not sure what happens to a public github repository which has not
been forked and is then deleted. I would assume that the content is then
lost.

To create a "flink-connectors" repository independent of Flink sounds quite
similar to creating another Apache Bahir. I think it would be better to
join forces with the existing Bahir community if possible.

On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen  wrote:

> Thanks Robert for bringing this up.
>
> I think that "flink-contrib" will not really solve the problem, because the
> connectors would still have to be maintained by the core community, we
> would need to guarantee test stability. It will be to a large extend
> similar to adding them to "flink-streaming-connectors", except that we may
> declare via the artifact name that the quality is not guaranteed to be too
> high.
>
> Regarding moving the code to a separate repository: I think that also only
> solves the problem, if that repository is organizationally different from
> the core clink repository. Different release cycles, different maintainers.
>
>
> The quintessence is for me that the connectors need to be handled by a
> different organization than the core flink community.
> That would leave us with
>   - The contributors' own repository
>   - Apache Bahir
>   - An organizationally different "flink-connectors" repository, possibly
> with different committers / PMC
>
>
>
>
>
> On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai 
> wrote:
>
> > Good point. Discussion for each new connector is also an overhead we
> should
> > avoid.
> >
> > IMHO, option #2 doesn’t seem like a reasonable staging area. Say we
> decide
> > we’d like to move a connector from Bahir to Flink in the future, there’ll
> > be two of the connector in separate Apache projects. Personally I think
> > option #2 is more like a way to go if some day we’d like to migrate some
> of
> > our currently supported connectors in Flink to Bahir (perhaps an
> > alternative to the discussion in "Externalizing the Flink connectors”).
> >
> > Defining option #1 as a staging area seems nice; the contributor notifies
> > the Flink community of the work by adding it to the 3rd party packages
> > page, and in the future we can contact the contributor to ask whether
> > they’d like to migrate the connector to Flink when we see more user
> demand
> > and committer support.
> >
> > With this approach, do you think we should assume that future new
> > connectors always go to option #1 first?   If not, I would assume that in
> > the future, Flink JIRAs for new connectors are only opened if we’d like
> to
> > add them directly to Flink (perhaps the contribution guideline can link
> to
> > Flink development Roadmaps like [1] as a reference to connectors that the
> > community has already considered to support?).
> >
> > [1]
> > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
> > 778DAD7eqw5GANwE/edit#
> >  > 778DAD7eqw5GANwE/edit>
> >
> > On August 9, 2016 at 6:10:21 PM, Robert Metzger (rmetz...@apache.org)
> > wrote:
> >
> > Hi Gordon,
> > thank you for your response.
> >
> > I agree with your observation that some "staging" area is helpful to test
> > how many contributors / users are interested in a connector. But I wonder
> > if #1 or #2 can also serve as a staging area: As soon as we see that
> there
> > is a lot of interest in a connector, we add it to the main
> > "flink-*-connectors" directory.
> > Having too many ways to contribute connectors might make the discussions
> > for each new connector quite complicated.
> >
> > You are right, the intention of the "Externalizing the Flink connectors”
> > discussion was more about the release intervals, test time etc.
> >
> > Regards,
> > Robert
> >
>

Re: [DISCUSS] Streaming connector contributions

[Stephan Ewen]

Thanks Robert for bringing this up.

I think that "flink-contrib" will not really solve the problem, because the
connectors would still have to be maintained by the core community, we
would need to guarantee test stability. It will be to a large extend
similar to adding them to "flink-streaming-connectors", except that we may
declare via the artifact name that the quality is not guaranteed to be too
high.

Regarding moving the code to a separate repository: I think that also only
solves the problem, if that repository is organizationally different from
the core clink repository. Different release cycles, different maintainers.


The quintessence is for me that the connectors need to be handled by a
different organization than the core flink community.
That would leave us with
  - The contributors' own repository
  - Apache Bahir
  - An organizationally different "flink-connectors" repository, possibly
with different committers / PMC





On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai 
wrote:

> Good point. Discussion for each new connector is also an overhead we should
> avoid.
>
> IMHO, option #2 doesn’t seem like a reasonable staging area. Say we decide
> we’d like to move a connector from Bahir to Flink in the future, there’ll
> be two of the connector in separate Apache projects. Personally I think
> option #2 is more like a way to go if some day we’d like to migrate some of
> our currently supported connectors in Flink to Bahir (perhaps an
> alternative to the discussion in "Externalizing the Flink connectors”).
>
> Defining option #1 as a staging area seems nice; the contributor notifies
> the Flink community of the work by adding it to the 3rd party packages
> page, and in the future we can contact the contributor to ask whether
> they’d like to migrate the connector to Flink when we see more user demand
> and committer support.
>
> With this approach, do you think we should assume that future new
> connectors always go to option #1 first?   If not, I would assume that in
> the future, Flink JIRAs for new connectors are only opened if we’d like to
> add them directly to Flink (perhaps the contribution guideline can link to
> Flink development Roadmaps like [1] as a reference to connectors that the
> community has already considered to support?).
>
> [1]
> https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
> 778DAD7eqw5GANwE/edit#
>  778DAD7eqw5GANwE/edit>
>
> On August 9, 2016 at 6:10:21 PM, Robert Metzger (rmetz...@apache.org)
> wrote:
>
> Hi Gordon,
> thank you for your response.
>
> I agree with your observation that some "staging" area is helpful to test
> how many contributors / users are interested in a connector. But I wonder
> if #1 or #2 can also serve as a staging area: As soon as we see that there
> is a lot of interest in a connector, we add it to the main
> "flink-*-connectors" directory.
> Having too many ways to contribute connectors might make the discussions
> for each new connector quite complicated.
>
> You are right, the intention of the "Externalizing the Flink connectors”
> discussion was more about the release intervals, test time etc.
>
> Regards,
> Robert
>

Re: [DISCUSS] Streaming connector contributions

[Tzu-Li (Gordon) Tai]

Good point. Discussion for each new connector is also an overhead we should
avoid.

IMHO, option #2 doesn’t seem like a reasonable staging area. Say we decide
we’d like to move a connector from Bahir to Flink in the future, there’ll
be two of the connector in separate Apache projects. Personally I think
option #2 is more like a way to go if some day we’d like to migrate some of
our currently supported connectors in Flink to Bahir (perhaps an
alternative to the discussion in "Externalizing the Flink connectors”).

Defining option #1 as a staging area seems nice; the contributor notifies
the Flink community of the work by adding it to the 3rd party packages
page, and in the future we can contact the contributor to ask whether
they’d like to migrate the connector to Flink when we see more user demand
and committer support.

With this approach, do you think we should assume that future new
connectors always go to option #1 first?   If not, I would assume that in
the future, Flink JIRAs for new connectors are only opened if we’d like to
add them directly to Flink (perhaps the contribution guideline can link to
Flink development Roadmaps like [1] as a reference to connectors that the
community has already considered to support?).

[1]
https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-778DAD7eqw5GANwE/edit#


On August 9, 2016 at 6:10:21 PM, Robert Metzger (rmetz...@apache.org) wrote:

Hi Gordon,
thank you for your response.

I agree with your observation that some "staging" area is helpful to test
how many contributors / users are interested in a connector. But I wonder
if #1 or #2 can also serve as a staging area: As soon as we see that there
is a lot of interest in a connector, we add it to the main
"flink-*-connectors" directory.
Having too many ways to contribute connectors might make the discussions
for each new connector quite complicated.

You are right, the intention of the "Externalizing the Flink connectors”
discussion was more about the release intervals, test time etc.

Regards,
Robert

Re: [DISCUSS] Streaming connector contributions

[Robert Metzger]

Hi Gordon,
thank you for your response.

I agree with your observation that some "staging" area is helpful to test
how many contributors / users are interested in a connector. But I wonder
if #1 or #2 can also serve as a staging area: As soon as we see that there
is a lot of interest in a connector, we add it to the main
"flink-*-connectors" directory.
Having too many ways to contribute connectors might make the discussions
for each new connector quite complicated.

You are right, the intention of the "Externalizing the Flink connectors”
discussion was more about the release intervals, test time etc.

Regards,
Robert


On Tue, Aug 9, 2016 at 11:47 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi Robert,
>
> Thank you for bringing the discussion to the mailing lists.
>
> #2 seems like a good option, if we can reach consensus with the Bahir
> community.
>
> However, should we also be considering “staging” (perhaps
> under “flink-contrib”?) a connector contribution until more committers can
> maintain it, and then move it into Flink? I feel like that we shouldn’t be
> redirecting all connector contributions away from Flink simply based on
> committer capacity. Some new connectors may have a wide user pool and can
> be staged first.
>
> So, I feel like redirection of connector contributions should go in 3 ways:
> 1. Redirect to option #1 or option #2
> 2. Stage first. We can officially move them
> to “flink-streaming/batch-connectors” once more committers can maintain
> it.
> 3. Directly contribute it to “flink-streaming/batch-connectors” as an
> officially supported connector, if initially there’s already committers
> willing to maintain it.
>
> As for option #3, from the discussion thread in "Externalizing the
> Flink connectors”,
> it seems like that the solution is mainly to provide
> more maintenance flexibility for our supported connectors, instead of
> solving the issue of committer capacity for new connectors, correct?
>
> Regards,
> Gordon
>
> On August 9, 2016 at 4:43:56 PM, Robert Metzger (rmetz...@apache.org)
> wrote:
>
> #2 Redirect the contribution to Apache Bahir. A recently created Apache
> project out of Apache Spark, to host some of their streaming connectors.
>

Re: [DISCUSS] Streaming connector contributions

[Tzu-Li (Gordon) Tai]

Hi Robert,

Thank you for bringing the discussion to the mailing lists.

#2 seems like a good option, if we can reach consensus with the Bahir
community.

However, should we also be considering “staging” (perhaps
under “flink-contrib”?) a connector contribution until more committers can
maintain it, and then move it into Flink? I feel like that we shouldn’t be
redirecting all connector contributions away from Flink simply based on
committer capacity. Some new connectors may have a wide user pool and can
be staged first.

So, I feel like redirection of connector contributions should go in 3 ways:
1. Redirect to option #1 or option #2
2. Stage first. We can officially move them
to “flink-streaming/batch-connectors” once more committers can maintain it.
3. Directly contribute it to “flink-streaming/batch-connectors” as an
officially supported connector, if initially there’s already committers
willing to maintain it.

As for option #3, from the discussion thread in "Externalizing the
Flink connectors”,
it seems like that the solution is mainly to provide
more maintenance flexibility for our supported connectors, instead of
solving the issue of committer capacity for new connectors, correct?

Regards,
Gordon

On August 9, 2016 at 4:43:56 PM, Robert Metzger (rmetz...@apache.org) wrote:

#2 Redirect the contribution to Apache Bahir. A recently created Apache
project out of Apache Spark, to host some of their streaming connectors.

[DISCUSS] Streaming connector contributions

[Robert Metzger]

Hi,

I would like to start a discussion regarding the streaming connectors in
Flink.
Currently, there are 12 connectors in the main repository, 4 more are
pending as pull requests (rethinkdb, kafka 0.10, ActiveMQ, HBase), and
there were some more discussed on the mailing list / JIRA / pull requests.
Recently, I started rejecting some new connector contributions because I
didn't see enough demand by our users and not enough capacity among the
committers to maintain all connectors.
Instead of just rejecting contributions, I would like to come up with a
solution that works for contributors and the Flink community. Ideally, the
solution should be documented in the contribution guidelines so that it is
transparent for new contributors.

Some ideas what we could do with new connector contributions:

#1 (current approach) Let the contributor host the connector in their own
GitHub repository, and link to the repo from the Flink 3rd party packages
page [1].

#2 Redirect the contribution to Apache Bahir. A recently created Apache
project out of Apache Spark, to host some of their streaming connectors.

#3 Use the "flink-connectors" git repository at apache [2]. For more
details, also check out the thread titled "Externalizing the Flink
connectors".

#4 Use the "project-flink" [3] GitHub account.

I would like to hear your opinion on the problem in general and the
approaches I've suggested.

I personally like #1 and #2, because they should have pretty low overhead
on the infrastructure-side for us (maven files, releases, CI / tests ...).
For #2, we would probably first start a discussion with the Bahir community
to see whether such contributions are in the scope of their project.
#3 is too much overhead for us, in my opinion, and #4 doesn't seem to have
an obvious advantage over #1.


Regards,
Robert


[1] http://flink.apache.org/community.html#third-party-packages
[2] https://git-wip-us.apache.org/repos/asf?p=flink-connectors.git
[3] https://github.com/project-flink