Going back to main subject of this thread, I just wanted to make things clear for all.
Seems like that everybody is agree that we will just deprecate AWS SDK V1 connectors once the alternative will be available, don’t remove them and still distribute artifacts [1] with new releases along with artifacts with IOs based on AWS SDK V2 [2]. Do we need to vote for this or we can accept it without voting if there are no objections? [1] https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services <https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services> [2] https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2 <https://search.maven.org/search?q=a:beam-sdks-java-io-amazon-web-services2> > On 27 Nov 2019, at 18:44, Kenneth Knowles <k...@google.com> wrote: > > > On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath <chamik...@google.com > <mailto:chamik...@google.com>> wrote: > > > On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni <r...@google.com > <mailto:r...@google.com>> wrote: > With regards to @Experimental there are a couple of discussions around its > usage ( or rather over usage! ) on dev@. It is something that we need to > clean up ( some of those IO are now being used on production env for years!). > > I agree that we should move some IO connectors out of experimental state and > probably this should be a separate discussion. Also, this issue is probably > more than for IO connectors since there are other parts of the code that is > marked as experimental as well, sometimes for a good reason (for example, > SDF). > > Yes, let's have a separate thread on @Experimental. There are a ton of > threads that start talking about it, and they all seem to agree it isn't > working. Only one direct thread* that was about something a bit more specific > https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E > > <https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E> > > > > On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko <aromanenko....@gmail.com > <mailto:aromanenko....@gmail.com>> wrote: > AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are marked > as "Experimental". So, it should not be a problem to gracefully deprecate and > finally remove them. We already did the similar procedure for > “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since it > started to support HadoopOutputFormatI as well). Old “HadoopInputFormatIO” > was deprecated and removed after 3 consecutive Beam releases (as we agreed on > mailing list). > > In the same time, some users for some reasons would not be able or to want to > move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 IOs and > accept new features/fixes only for V2 IOs. > > +1 for deprecating AWS V1 IO connectors as opposed to removing as well unless > we can confirm that usage is extremely limited. > > +1 to deprecate as soon as there is an alternative. > > Trying to measure usage is a good idea, but hard. If the maven coordinates > were different we could look at download numbers and dependencies. > > > Talking about “Experimental” annotation. Sorry in advance If I missed that > and switch a subject a bit, but do we have clear rules or an agreement when > IO becomes stable and should not be marked as experimental anymore? Most of > our Java IOs are marked as Experimental but many of them were using in > production by real users under real load. Does it mean that they are ready to > be stable in terms of API? Perhaps, this topic deserves a new discussion if > there are several opinions on that. > > Probably, decision to move component APIs (for example, an IO connector) out > of experimental state should be done on a case-by-case basis. > > Let's repeat these good points on a dedicated thread. > > Kenn > > > > Thanks, > Cham > > >> On 26 Nov 2019, at 00:39, Luke Cwik <lc...@google.com >> <mailto:lc...@google.com>> wrote: >> >> Phase I sounds fine. >> >> Apache Beam follows semantic versioning and I believe removing the IOs will >> be a backwards incompatible change unless they were marked experimental >> which will be a problem for Phase 2. >> >> What is the feasibility of making the V1 transforms wrappers around V2? >> >> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach <cammac...@gmail.com >> <mailto:cammac...@gmail.com>> wrote: >> Hello Beam Devs, >> >> I have been working on the migration of Amazon Web Services IO connectors >> into the new AWS SDK for Java V2. The goal is to have an updated >> implementation aligned with the most recent AWS improvements. So far we have >> already migrated the connectors for AWS SNS, SQS and DynamoDB. >> >> In the meantime some contributions are still going on V1 IOs. So far we have >> dealt with those by porting (or asking contributors) to port the changes >> into V2 IOs too because we don’t want features of both versions to be >> unaligned but this may quickly become a maintenance issue, so we want to >> discuss a plan to stop supporting (deprecate) V1 IOs and encourage users to >> move to V2. >> >> Phase I (ASAP): >> Mark migrated AWS V1 IOs as deprecated >> Document migration path to V2 >> Phase II (end of 2020): >> Decide a date or Beam release to remove the V1 IOs >> Send a notification to the community 3 months before we remove them >> Completely get rid of V1 IOs >> >> Please let me know what you think or if you see any potential issues? >> >> Thanks, >> Cam Mach >> > > > > -- > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone to > the wrong person. > The above terms reflect a potential business arrangement, are provided solely > as a basis for further discussion, and are not intended to be and do not > constitute a legally binding obligation. No legally binding obligations will > be created, implied, or inferred until an agreement in final form is executed > in writing by all parties involved.
