Achieving good abstractions will prove elusive since the APIs differ and we will end up with a ton of extra maintenance work that should be not Beam's responsibility. I know that similar code (almost copy pasteable) is not nice to have but we should consider this as a temporary measure and probably try to make deprecation (and possible removal) of older versions as soon as possible. Please remember that this was already discussed in the past [1] and the soft consensus was around rapid deprecation AWS SDK v1 IOs already available in their v2 version and only do improvements on v1 IOs for security related issues and dependency updates. This has not happened yet but the only missing thing is someone to take action.
+cammac...@gmail.com would you have to me to start rolling this plan? otherwise someone else can jump and do it too. [1] https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E On Thu, May 7, 2020 at 11:05 PM Luke Cwik <lc...@google.com> wrote: > > I think you should try and share as much as is reasonable. Using what is > shared between AWS V1 and V2 SDKs would be a good signal as to what should be > shared in Beam. There might be some places where a trivial wrapper could help > but I wouldn't try to create a bunch of grand abstractions that fit both > versions of the AWS SDKs. > > On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <aromanenko....@gmail.com> > wrote: >> >> I’d like to get back to this question, fairly raised by Jonothan a while >> ago, since actually it affects not only KinesisIO but also all other AWS IOs >> in general, that uses AWS SDK of two different versions. >> >> My personal opinion - I’d strongly like to avoid a copy-pasted code for two >> different versions of the same IO (and double support of this), that is >> SDK-independent. In this case, if we wish too provide IO for two AWS SDK >> versions, we have to extract this core logic into one common and >> SDK-independent framework that could used by any of SDK versions then. >> >> In the same time, if it will require a lot of efforts to achieve that, then >> probably we need to decide to leave only one IO based on single AWS SDK >> version (I guess it will be V2 as more modern) and deprecate old version >> (and then remove it to avoid a confusion for users). >> >> What others think about that? >> >> On 18 Apr 2020, at 01:48, Jonothan Farr <jonothan.f...@gmail.com> wrote: >> >> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was >> working on, so I ended up submitting that as >> https://github.com/apache/beam/pull/9899. >> >> But since that still needs work to be fully functional (right now it does >> not handle resharding), I figured I could at least contribute the updates to >> make KinesisIO compatible with the AWS SDK v2, so I split that out and >> submitted it as: https://github.com/apache/beam/pull/11318. >> >> Now some totally reasonable concerns have been raised about maintaining two >> separate codebases, and having to dual-commit every bugfix or feature >> enhancement. That seems to already be the case for all of the other AWS IOs >> in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue >> that trend? What do members of the community think? >> >> One solution is obviously to rewrite the AWS IOs with an added abstraction >> layer so that the SDK-specific code can be factored out. However, the >> KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not >> aware of a way that this could be done so that it's backwards-compatible. >> This means that everyone currently using KinesisIO would most likely have to >> update their code to use the new interface whether they wanted to use the >> new v2 SDK or continue to use v1. I think a rewrite would also be a >> non-trivial task with more than a few details to be worked out first. >> >> From my perspective, the options are: >> >> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO >> and I can continue working on enhanced fanout while a rewrite is being >> worked on, but the community now has to dual-commit any changes to KinesisIO. >> - Close #11318 and continue working on #9899 and I alone have to dual-commit >> the changes until it can be merged, at which point everyone can use enhanced >> fanout while the rewrite is being worked on. Then the community will also >> have to dual-commit but only after enhanced fanout is available. >> - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS >> SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply >> that I wouldn't be interested in contributing to that effort, just for the >> sake of these 2 PRs.) >> - Don't do the rewrite at all and just live with dual-committing until the >> KinesisIO based on the v1 SDK reaches its end of life. No one has to update >> their code to continue using the v1 SDK, only if they want to move to v2. >> >> There may be other options or considerations I'm missing here. What do >> others think? >> >> Thanks, >> Jonothan >> >>
