I think that we should get back to this question since that time we have more and more AWS-related IO connectors (mostly in Java SDK afaik).
It would be great to have a dedicated Beam’s credentials to run all our AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run such tests against 3rd party implementations, for example, “localstack”, as most comprehensive one. Of course we can observe some potential discrepancy in behaviour between real AWS and other implementations, but if it’s not principal things then it should not be a stopper. I believe that regular running ITs, especially for IO connectors, is a very important. > On 9 Jul 2020, at 22:18, Luke Cwik <lc...@google.com> wrote: > > It has come up a few times[1, 2, 3, 4] and there have also been a few > comments over time about whether someone could donate AWS resources to the > project. > > 1: https://issues.apache.org/jira/browse/BEAM-601 > <https://issues.apache.org/jira/browse/BEAM-601> > 2: https://issues.apache.org/jira/browse/BEAM-3373 > <https://issues.apache.org/jira/browse/BEAM-3373> > 3: https://issues.apache.org/jira/browse/BEAM-3550 > <https://issues.apache.org/jira/browse/BEAM-3550> > 4: https://issues.apache.org/jira/browse/BEAM-3032 > <https://issues.apache.org/jira/browse/BEAM-3032> > On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <maniko...@gmail.com > <mailto:maniko...@gmail.com>> wrote: > Have you guys considered using localstack to run AWS service based > integration tests? > > https://github.com/localstack/localstack > <https://github.com/localstack/localstack> > > On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <piotr.szuber...@polidea.com > <mailto:piotr.szuber...@polidea.com>> wrote: > Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it > tests then. Thanks for your reply :) > > On 2020/07/08 17:13:11, Alexey Romanenko <aromanenko....@gmail.com > <mailto:aromanenko....@gmail.com>> wrote: > > If you mean Java KinesisIO tests, then unit tests are running on Jenkins > > [1] and ITs are not running since it requires AWS credentials that we don’t > > have dedicated to Beam for the moment. > > > > In the same time, you can run KinesisIOIT with your own credentials, like > > we do in Talend (a company that I work for). > > > > [1] > > https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ > > > > <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/> > > > > <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ > > > > <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/>> > > > > > On 8 Jul 2020, at 13:11, Piotr Szuberski <piotr.szuber...@polidea.com > > > <mailto:piotr.szuber...@polidea.com>> wrote: > > > > > > I'm writing KinesisIO external transform with python wrapper and I found > > > that the tests aren't executed anywhere in Jenkins. Am I wrong or there > > > is a reason for that? > > > >
