Hi Robert, We had this discussion before when I suggested to use an external repository to manage connectors. Ever since I have come to the conclusion that the overhead of maintaining two source repositories along with maintaining code and integration, documentation, and CI, is not worth the effort. Thanks for bringing up the discussion again.
I think the most important issues with moving connectors out of the core repository are 1) Maintenance Connectors need to be maintained. Otherwise, they are worthless. 2) Plugability Connectors need to be easy to use for the user. Otherwise, nobody will use them. What is the difference between #1, #3 and #4? I don't see a difference except that #3 and #4 are central repositories. Still, maintenance and plugability are very unclear. Only #2 is left :) Apache Bahir looks like a promising project. Let's see how the community reacts. Cheers, Max On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetz...@apache.org> wrote: > Thank you for your responses. I will get in touch with the Bahir community > to see what they are thinking about this. > Once we know a bit more about the details of such a collaboration, we can > make a final decision here. > > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrm...@apache.org> wrote: > >> I agree with Stephan that the main problem is maintenance overhead for the >> Flink community. If we could maintain all connectors ourselves then there >> would not be an immediate need to out source the connectors. Thus, the >> solution should reduce the workload for the core project. >> >> Personally, I would favour option #2 if Apache Bahir is willing to add >> Flink as another supported streaming platform. This would give the >> streaming connectors a more prominent home than the personal repository of >> a contributor. >> >> Furthermore, contributing the connectors to Apache Bahir entails that the >> connector artifacts are deployed to a central repository (I assume). That >> way one can more easily add them to one's project instead of having to >> build and install the code yourself. >> >> I'm also not sure what happens to a public github repository which has not >> been forked and is then deleted. I would assume that the content is then >> lost. >> >> To create a "flink-connectors" repository independent of Flink sounds quite >> similar to creating another Apache Bahir. I think it would be better to >> join forces with the existing Bahir community if possible. >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <se...@apache.org> wrote: >> >> > Thanks Robert for bringing this up. >> > >> > I think that "flink-contrib" will not really solve the problem, because >> the >> > connectors would still have to be maintained by the core community, we >> > would need to guarantee test stability. It will be to a large extend >> > similar to adding them to "flink-streaming-connectors", except that we >> may >> > declare via the artifact name that the quality is not guaranteed to be >> too >> > high. >> > >> > Regarding moving the code to a separate repository: I think that also >> only >> > solves the problem, if that repository is organizationally different from >> > the core clink repository. Different release cycles, different >> maintainers. >> > >> > >> > The quintessence is for me that the connectors need to be handled by a >> > different organization than the core flink community. >> > That would leave us with >> > - The contributors' own repository >> > - Apache Bahir >> > - An organizationally different "flink-connectors" repository, possibly >> > with different committers / PMC >> > >> > >> > >> > >> > >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai <tzuli...@gmail.com> >> > wrote: >> > >> > > Good point. Discussion for each new connector is also an overhead we >> > should >> > > avoid. >> > > >> > > IMHO, option #2 doesn’t seem like a reasonable staging area. Say we >> > decide >> > > we’d like to move a connector from Bahir to Flink in the future, >> there’ll >> > > be two of the connector in separate Apache projects. Personally I think >> > > option #2 is more like a way to go if some day we’d like to migrate >> some >> > of >> > > our currently supported connectors in Flink to Bahir (perhaps an >> > > alternative to the discussion in "Externalizing the Flink connectors”). >> > > >> > > Defining option #1 as a staging area seems nice; the contributor >> notifies >> > > the Flink community of the work by adding it to the 3rd party packages >> > > page, and in the future we can contact the contributor to ask whether >> > > they’d like to migrate the connector to Flink when we see more user >> > demand >> > > and committer support. >> > > >> > > With this approach, do you think we should assume that future new >> > > connectors always go to option #1 first? If not, I would assume that >> in >> > > the future, Flink JIRAs for new connectors are only opened if we’d like >> > to >> > > add them directly to Flink (perhaps the contribution guideline can link >> > to >> > > Flink development Roadmaps like [1] as a reference to connectors that >> the >> > > community has already considered to support?). >> > > >> > > [1] >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm- >> > > 778DAD7eqw5GANwE/edit# >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm- >> > > 778DAD7eqw5GANwE/edit> >> > > >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger (rmetz...@apache.org) >> > > wrote: >> > > >> > > Hi Gordon, >> > > thank you for your response. >> > > >> > > I agree with your observation that some "staging" area is helpful to >> test >> > > how many contributors / users are interested in a connector. But I >> wonder >> > > if #1 or #2 can also serve as a staging area: As soon as we see that >> > there >> > > is a lot of interest in a connector, we add it to the main >> > > "flink-*-connectors" directory. >> > > Having too many ways to contribute connectors might make the >> discussions >> > > for each new connector quite complicated. >> > > >> > > You are right, the intention of the "Externalizing the Flink >> connectors” >> > > discussion was more about the release intervals, test time etc. >> > > >> > > Regards, >> > > Robert >> > > >> > >>
