Just for reference we already run localstack for DynamoDB tests via
testcontainers [1] so everything should be ready (at least module wise) to run
other simulated ITs. Being able to run all our AWS related ITs with localstack
is a really simple and nice contribution to do if anyone is interested.
Localstack looks really cool. We have to consider resource consumption and
latency for where it fits in the testing plan. It definitely seems useful
for having ITs easily. For individual IOs, an in-process implementation can
be much more performant without losing much accuracy (since it is also
I think that we should get back to this question since that time we have more
and more AWS-related IO connectors (mostly in Java SDK afaik).
It would be great to have a dedicated Beam’s credentials to run all our
AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run such
It has come up a few times[1, 2, 3, 4] and there have also been a few
comments over time about whether someone could donate AWS resources to the
project.
1: https://issues.apache.org/jira/browse/BEAM-601
2: https://issues.apache.org/jira/browse/BEAM-3373
3:
Have you guys considered using localstack to run AWS service based
integration tests?
https://github.com/localstack/localstack
On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski,
wrote:
> Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language
> it tests then. Thanks for your reply
Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it
tests then. Thanks for your reply :)
On 2020/07/08 17:13:11, Alexey Romanenko wrote:
> If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1]
> and ITs are not running since it requires AWS
If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1]
and ITs are not running since it requires AWS credentials that we don’t have
dedicated to Beam for the moment.
In the same time, you can run KinesisIOIT with your own credentials, like we do
in Talend (a company that
I'm writing KinesisIO external transform with python wrapper and I found that
the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason
for that?
