[
https://issues.apache.org/jira/browse/BEAM-601?focusedWorklogId=466058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-466058
]
ASF GitHub Bot logged work on BEAM-601:
---------------------------------------
Author: ASF GitHub Bot
Created on: 04/Aug/20 07:01
Start Date: 04/Aug/20 07:01
Worklog Time Spent: 10m
Work Description: piotr-szuberski commented on a change in pull request
#12422:
URL: https://github.com/apache/beam/pull/12422#discussion_r464840886
##########
File path:
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java
##########
@@ -35,33 +38,45 @@
import org.apache.beam.sdk.values.PCollection;
import org.joda.time.Duration;
import org.joda.time.Instant;
+import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.testcontainers.containers.localstack.LocalStackContainer;
/**
* Integration test, that writes and reads data to and from real Kinesis. You
need to provide {@link
- * KinesisTestOptions} in order to run this.
+ * KinesisTestOptions} in order to run this if you want to test it with
production setup. By default
+ * when no options are provided an instance of localstack is used.
*/
@RunWith(JUnit4.class)
public class KinesisIOIT implements Serializable {
- private static int numberOfShards;
- private static int numberOfRows;
-
@Rule public TestPipeline pipelineWrite = TestPipeline.create();
@Rule public TestPipeline pipelineRead = TestPipeline.create();
+ private static LocalStackContainer localstackContainer;
+
private static KinesisTestOptions options;
- private static final Instant now = Instant.now();
Review comment:
It's for the testing with real aws purpose. When a user wants to test
Beam's KinesisIO with an existing stream then it would be best to read only the
records that were written just in the time the test is running. If not, then it
would be possible that write transform is broken and read works well (read some
data from the past) and we have false-positive. It's still possible if a user
uses a stream that receives records from elsewhere, but it cannot be helped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 466058)
Time Spent: 2.5h (was: 2h 20m)
> Enable Kinesis integration tests
> --------------------------------
>
> Key: BEAM-601
> URL: https://issues.apache.org/jira/browse/BEAM-601
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Affects Versions: 0.3.0-incubating
> Reporter: Przemyslaw Pastuszka
> Assignee: Piotr Szuberski
> Priority: P3
> Labels: starter
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> There's an integration test for KinesisIO called KinesisReaderIT, but it is
> currently ignored, because it needs real Kinesis instance setup.
> As part of this task please:
> * setup real Kinesis environment on AWS for testing purposes
> * enable KinesisReaderIT test
> * setup jenkins, so that it passes all KinesisTestOptions when running
> integration tests
> This is a follow up to BEAM-461 requested by [[email protected]] in
> https://github.com/apache/incubator-beam/pull/687/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
