On Tue, Nov 26, 2019 at 7:00 PM Chamikara Jayalath <chamik...@google.com> wrote:
> > > On Tue, Nov 26, 2019 at 6:17 PM Reza Rokni <r...@google.com> wrote: > >> With regards to @Experimental there are a couple of discussions around >> its usage ( or rather over usage! ) on dev@. It is something that we >> need to clean up ( some of those IO are now being used on production env >> for years!). >> > > I agree that we should move some IO connectors out of experimental state > and probably this should be a separate discussion. Also, this issue is > probably more than for IO connectors since there are other parts of the > code that is marked as experimental as well, sometimes for a good reason > (for example, SDF). > Yes, let's have a separate thread on @Experimental. There are a ton of threads that start talking about it, and they all seem to agree it isn't working. Only one direct thread* that was about something a bit more specific https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E > On Tue, Nov 26, 2019 at 8:21 AM Alexey Romanenko <aromanenko....@gmail.com> >>>> wrote: >>>> >>>>> AFAICT, all AWS SDK V1 IOs (SnsIO, SqsIO, DynamoDBIO, KinesisIO) are >>>>> marked as "Experimental". So, it should not be a problem to gracefully >>>>> deprecate and finally remove them. We already did the similar procedure >>>>> for >>>>> “HadoopInputFormatIO”, which was renamed to just “HadoopFormatIO” (since >>>>> it >>>>> started to support HadoopOutputFormatI as well). Old “HadoopInputFormatIO” >>>>> was deprecated and removed after *3 consecutive* Beam releases (as we >>>>> agreed on mailing list). >>>>> >>>>> In the same time, some users for some reasons would not be able or to >>>>> want to move on AWS SDK V2. So, I’d prefer to just deprecate AWS SDK V1 >>>>> IOs >>>>> and accept new features/fixes *only* for V2 IOs. >>>>> >>>> > +1 for deprecating AWS V1 IO connectors as opposed to removing as well > unless we can confirm that usage is extremely limited. > +1 to deprecate as soon as there is an alternative. Trying to measure usage is a good idea, but hard. If the maven coordinates were different we could look at download numbers and dependencies. Talking about “Experimental” annotation. Sorry in advance If I missed that >>>>> and switch a subject a bit, but do we have clear rules or an agreement >>>>> when >>>>> IO becomes stable and should not be marked as experimental anymore? >>>>> *Most* of our Java IOs are marked as Experimental but many of them >>>>> were using in production by real users under real load. Does it mean that >>>>> they are ready to be stable in terms of API? Perhaps, this topic deserves >>>>> a >>>>> new discussion if there are several opinions on that. >>>>> >>>> > Probably, decision to move component APIs (for example, an IO connector) > out of experimental state should be done on a case-by-case basis. > Let's repeat these good points on a dedicated thread. Kenn > > Thanks, > Cham > > >> >>>>> On 26 Nov 2019, at 00:39, Luke Cwik <lc...@google.com> wrote: >>>>> >>>>> Phase I sounds fine. >>>>> >>>>> Apache Beam follows semantic versioning and I believe removing the IOs >>>>> will be a backwards incompatible change unless they were marked >>>>> experimental which will be a problem for Phase 2. >>>>> >>>>> What is the feasibility of making the V1 transforms wrappers around V2? >>>>> >>>>> On Mon, Nov 25, 2019 at 1:46 PM Cam Mach <cammac...@gmail.com> wrote: >>>>> >>>>>> Hello Beam Devs, >>>>>> >>>>>> I have been working on the migration of Amazon Web Services IO >>>>>> connectors into the new AWS SDK for Java V2. The goal is to have an >>>>>> updated >>>>>> implementation aligned with the most recent AWS improvements. So far we >>>>>> have already migrated the connectors for AWS SNS, SQS and DynamoDB. >>>>>> >>>>>> In the meantime some contributions are still going on V1 IOs. So far >>>>>> we have dealt with those by porting (or asking contributors) to port the >>>>>> changes into V2 IOs too because we don’t want features of both versions >>>>>> to >>>>>> be unaligned but this may quickly become a maintenance issue, so we want >>>>>> to >>>>>> discuss a plan to stop supporting (deprecate) V1 IOs and encourage users >>>>>> to >>>>>> move to V2. >>>>>> >>>>>> Phase I (ASAP): >>>>>> >>>>>> - Mark migrated AWS V1 IOs as deprecated >>>>>> - Document migration path to V2 >>>>>> >>>>>> Phase II (end of 2020): >>>>>> >>>>>> - Decide a date or Beam release to remove the V1 IOs >>>>>> - Send a notification to the community 3 months before we remove >>>>>> them >>>>>> - Completely get rid of V1 IOs >>>>>> >>>>>> >>>>>> Please let me know what you think or if you see any potential issues? >>>>>> >>>>>> Thanks, >>>>>> Cam Mach >>>>>> >>>>>> >>>>> >> >> -- >> >> This email may be confidential and privileged. If you received this >> communication by mistake, please don't forward it to anyone else, please >> erase all copies and attachments, and please let me know that it has gone >> to the wrong person. >> >> The above terms reflect a potential business arrangement, are provided >> solely as a basis for further discussion, and are not intended to be and do >> not constitute a legally binding obligation. No legally binding obligations >> will be created, implied, or inferred until an agreement in final form is >> executed in writing by all parties involved. >> >
