This plan LGTM. Thanks, Thomas
On Wed, Feb 23, 2022 at 4:28 AM Chesnay Schepler <ches...@apache.org> wrote: > > That sounds fine to me. > > On 23/02/2022 10:49, Konstantin Knauf wrote: > > Hi Chesnay, Hi everyone, > > > > I think the idea for the migration is the following (with the example of > > ElasticSearch). I talked to Martijn offline. > > > > 1. ElasticSearch Connector is released from the core repository with the > > Flink 1.15.0 release. No changes. > > > > 2. At the beginning of the Flink 1.16 release cycle the connector is > > removed from `master` of the core repository. It remains on the > > `release-1.15` branch and earlier release branches. > > > > 3. The connector code is "copied" over to the `master` branch of a > > `flink-connector-elastic-search` repository. Bugfixes to the connector need > > to go to both `release-1.15` and before in the core repository and `master` > > of the external repository. > > > > 4. Once all the processes required to do a release in the > > `flink-connector-elastic-search` are in place (docs integration, release > > automation,...), we release flink-connector-elastic-search:3.0.0, which > > will be compatible with Flink 1.15. At this point, users can choose whether > > they use flink-connector-elastic-search:1.15.x (released from the core > > repository) or flink-connector-elastic-search:3.0.0 already released from > > the external repository with Flink 1.15. The documentation will already > > advertise the one released from the external repository. This is the > > "overlap" that Martijn mentioned. > > > > 5. From here onwards, the release cycle of the ElasticSearch Connector is > > independent. There could be 3.1.0 and 3.0.1 etc. The compatibility matrix > > will be part of the connector documentation. > > > > 6. If there is a patch release for Flink 1.15-, this will of course also > > include flink-connector-elastic-search release from the core repository. > > > > 7. For Flink 1.16, there might or might not be a release of the > > elastic-search-connector from the external repository. Depends on > > compatibility. > > > > I hope this clarifies it a bit and it makes sense to me. > > > > It also makes sense to me to do this as soon as possible (probably once the > > release-1.15 branch is cut) with the example of ElasticSearch. Afterwards > > (hopefully still in the Flink 1.16 release cycle) we do the same for other > > connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or > > helpful to make it a condition that this happens for all connectors at the > > same time. > > > > Cheers and thanks, > > > > Konstantin > > > > > > On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ches...@apache.org> wrote: > > > >> If we don't make a release, I think it would appear as partially > >> externalized (since the binaries are still only created with Flink core, > >> not from the external repository). > >> > >> I'm wondering you are referring to when you say "it appear[s]". Users > >> don't know about it, and contributors can be easily informed about the > >> current state. Who's opinion are you worried about? > >> > >> doing a release [means] that we have then completely externalized the > >> connector > >> > >> In my mind the connector is completely externalized once the connector > >> project is complete and can act independently from the core repo. That > >> includes having all the code, working CI and the documentation being > >> integrated into the Flink website. And /then/ we can do a release. I > >> don't see how this could work any other way; how could we possibly argue > >> that the connector is externalized when development on the connector > >> isn't even possible in that repository? > >> > >> There are also other connectors (like Opensearch and I believe RabbitMQ) > >> that will end up straight in their own repositories > >> > >> > >> Which is a bit of a different situation because here the only source of > >> this connector will have been that repository. > >> > >> Would you prefer to remove a connector in a Flink patch release? > >> > >> > >> No. I think I misread your statement; when you said that there "is 1 > >> release cycle where the connector both exists in Flink core and the > >> external repo", you are referring to 1.15, correct? (although this > >> should also apply to 1.14 so isn't it 2 releases...?) > >> How I understood it was that we'd keep the connector around until 1.16, > >> which would obviously be terrible. > >> > >> On 19/02/2022 13:30, Martijn Visser wrote: > >>> Hi Chesnay, > >>> > >>> I think the advantage of also doing a release is that we have then > >>> completely externalized the connector. If we don't make a release, I > >> think > >>> it would appear as partially externalized (since the binaries are still > >>> only created with Flink core, not from the external repository). It would > >>> also mean that in our documentation we would still point to the binary > >>> created with the Flink core release. > >>> > >>> There are also other connectors (like Opensearch and I believe RabbitMQ) > >>> that will end up straight in their own repositories. Since we also would > >>> like to document those, I don't think the situation will be messy. We can > >>> also solve it with an information hint in the documentation. > >>> > >>> With regards to point 6, do you have an alternative? Would you prefer to > >>> remove a connector in a Flink patch release? > >>> > >>> Best regards, > >>> > >>> Martijn > >>> > >>> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ches...@apache.org> > >> wrote: > >>>> Why do you want to immediately do a release for the elasticsearch > >>>> connector? What does that provide us? > >>>> > >>>> I'd rather first have a fully working setup and integrated documentation > >>>> before thinking about releasing anything. > >>>> Once we have that we may be able to externalize all connectors within 1 > >>>> release cycle and do a clean switch; otherwise we end up with a bit of a > >>>> messy situation for users where some connectors use version scheme A and > >>>> others B. > >>>> > >>>> I also doubt the value of 6). They'll have to update the version anyway > >>>> (and discover at some point that the version scheme has changed), so I > >>>> don't see what this makes easier. > >>>> > >>>> On 18/02/2022 14:54, Martijn Visser wrote: > >>>>> Hi everyone, > >>>>> > >>>>> As a follow-up to earlier discussions [1] [2] to externalize the > >>>> connectors > >>>>> from the Flink repository, I would like to propose a plan to > >> externalize > >>>>> these connectors. The goal of this plan is to start with moving > >>>> connectors > >>>>> to its own repositories without introducing regressions for connector > >>>>> developers. > >>>>> > >>>>> The plan is as follows: > >>>>> > >>>>> 1. A new repository is requested for a connector. > >>>>> 2. The code for that connector is moved to its individual repository, > >>>>> including the commit history > >>>>> 3. Any first release made for a connector in an external connector > >>>>> repository starts with major version 3, so 3.0.0. The reason for that > >> is > >>>>> that we want to decouple the Flink releases from a connector release. > >> If > >>>> we > >>>>> would start with major version 2, it could cause some confusion because > >>>>> people could think a Flink 2.0 has been released. This does mean that > >>>> each > >>>>> connector needs to have a compatibility matrix generated, stating which > >>>>> version number of the connector is compatible with the correct Flink > >>>>> version. > >>>>> 4. The group id and artifact id for the connector will remain the same, > >>>>> only the version is different. > >>>>> 5. The connector dependencies on the Flink website are updated to point > >>>> to > >>>>> the newly released connector artifact. > >>>>> 6. If a connector is moved, there is one release cycle where there will > >>>> be > >>>>> binary releases for that connector in both Flink core and from the > >>>>> connector repository. This is to make Flink users who are upgrading > >>>>> slightly easier. We will have to make a note in the release notes that > >> a > >>>>> connector has been moved and that a user should update any references > >>>> from > >>>>> the original connector artifact (from the Flink connector) to the new > >>>>> connector artifact (from the external conenctor version) > >>>>> > >>>>> We propose to first try to execute this plan for the Elasticsearch > >>>>> connector as follows: > >>>>> > >>>>> 1. We wait until the Flink 1.15 release branch is cut > >>>>> 2. When that's done, the Elasticsearch code (including commit history) > >>>> from > >>>>> Flink's 1.15 release branch will be moved to the > >>>>> flink-connector-elasticsearch main branch. > >>>>> 3. When Flink 1.15 is released, we will also release an Elasticsearch > >>>>> connector for the external connector repository with version 3.0.0. > >>>>> 4. Bugfixes or improvements will be made first pointing to the external > >>>>> connector repository and will be cherry-picked back to the release-1.15 > >>>>> branch in the Flink core repository. > >>>>> 5. The Elasticsearch code, test etc will be removed from the master > >>>> branch > >>>>> in the Flink core repository and dropped with Flink 1.16 > >>>>> > >>>>> Looking forward to your thoughts on this! > >>>>> > >>>>> Best regards, > >>>>> > >>>>> Martijn Visser > >>>>> https://twitter.com/MartijnVisser82 > >>>>> > >>>>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm > >>>>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > >>>>> > > >
