I think you should try and share as much as is reasonable. Using what is shared between AWS V1 and V2 SDKs would be a good signal as to what should be shared in Beam. There might be some places where a trivial wrapper could help but I wouldn't try to create a bunch of grand abstractions that fit both versions of the AWS SDKs.
On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <aromanenko....@gmail.com> wrote: > I’d like to get back to this question, fairly raised by Jonothan a while > ago, since actually it affects not only KinesisIO but also all other AWS > IOs in general, that uses AWS SDK of two different versions. > > My personal opinion - I’d strongly like to avoid a copy-pasted code for > two different versions of the same IO (and double support of this), that is > SDK-independent. In this case, if we wish too provide IO for two AWS SDK > versions, we have to extract this core logic into one common and > SDK-independent framework that could used by any of SDK versions then. > > In the same time, if it will require a lot of efforts to achieve that, > then probably we need to decide to leave only one IO based on single AWS > SDK version (I guess it will be V2 as more modern) and deprecate old > version (and then remove it to avoid a confusion for users). > > What others think about that? > > On 18 Apr 2020, at 01:48, Jonothan Farr <jonothan.f...@gmail.com> wrote: > > Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was > working on, so I ended up submitting that as > https://github.com/apache/beam/pull/9899. > > But since that still needs work to be fully functional (right now it does > not handle resharding), I figured I could at least contribute the updates > to make KinesisIO compatible with the AWS SDK v2, so I split that out and > submitted it as: https://github.com/apache/beam/pull/11318. > > Now some totally reasonable concerns have been raised about maintaining > two separate codebases, and having to dual-commit every bugfix or feature > enhancement. That seems to already be the case for all of the other AWS IOs > in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue > that trend? What do members of the community think? > > One solution is obviously to rewrite the AWS IOs with an added abstraction > layer so that the SDK-specific code can be factored out. However, the > KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm > not aware of a way that this could be done so that it's > backwards-compatible. This means that everyone currently using KinesisIO > would most likely have to update their code to use the new interface > whether they wanted to use the new v2 SDK or continue to use v1. I think a > rewrite would also be a non-trivial task with more than a few details to be > worked out first. > > From my perspective, the options are: > > - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO > and I can continue working on enhanced fanout while a rewrite is being > worked on, but the community now has to dual-commit any changes to > KinesisIO. > - Close #11318 and continue working on #9899 and I alone have to > dual-commit the changes until it can be merged, at which point everyone can > use enhanced fanout while the rewrite is being worked on. Then the > community will also have to dual-commit but only after enhanced fanout is > available. > - Close both PRs and just wait for a rewrite, and no one can use the v2 > AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to > imply that I wouldn't be interested in contributing to that effort, just > for the sake of these 2 PRs.) > - Don't do the rewrite at all and just live with dual-committing until the > KinesisIO based on the v1 SDK reaches its end of life. No one has to update > their code to continue using the v1 SDK, only if they want to move to v2. > > There may be other options or considerations I'm missing here. What do > others think? > > Thanks, > Jonothan > > >
