Hi Stefan, Thanks for the info. So if I understand correctly, the pipeline you had in mind is:
Consumer -> Map -> Producer What do you expect as outcome of the mapper failure? That no records are lost but some possibly duplicated in the sink? Regarding the abstraction, I will see what I can do in that regard. From where I start it may make more sense to do some of that as follow-up when the Kafka test is ported. Thanks, Thomas On Thu, Nov 8, 2018 at 10:20 AM Stefan Richter <s.rich...@data-artisans.com> wrote: > Hi, > > I was also just planning to work on it before Stephan contacted Thomas to > ask about this test. > > Thomas, you are right about the structure, the test should also go into > the `run-nightly-tests.sh`. What I was planning to do is a simple job that > consists of a Kinesis consumer, a mapper that fails once after n records, > and a kinesis producer. I was hoping that creation, filling, and validation > of the Kinesis topics can be done with the Java API, not by invoking > commands in a bash script. In general I would try to minimise the amount of > scripting and do as much in Java as possible. It would also be nice if the > test was generalised, e.g. that abstract Producer/Consumer are created from > a Supplier and also the validation is done over some abstraction that lets > us iterate over the produced output. Ideally, this would be a test that we > can reuse for all Consumer/Producer cases and we could also port the tests > for Kafka to that. What do you think? > > Best, > Stefan > > > On 8. Nov 2018, at 07:22, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > > > > Hi Thomas, > > > > I think Stefan Richter is also working on the Kinesis end-to-end test, > and > > seems to be planning to implement it against a real Kinesis service > instead > > of Kinesalite. > > Perhaps efforts should be synced here. > > > > Cheers, > > Gordon > > > > > > On Thu, Nov 8, 2018 at 1:38 PM Thomas Weise <t...@apache.org> wrote: > > > >> Hi, > >> > >> I'm planning to add an end-to-end test for the Kinesis consumer. We have > >> done something similar at Lyft, using Kinesalite, which can be run as > >> Docker container. > >> > >> I see that some tests already make use of Docker, so we can assume it > to be > >> present in the target environment(s)? > >> > >> I also found the following ticket: > >> https://issues.apache.org/jira/browse/FLINK-9007 > >> > >> It suggest to also cover the producer, which may be a good way to create > >> the input data as well. The stream itself can be created with the > Kinesis > >> Java SDK. > >> > >> Following the existing layout, there would be a new module > >> flink-end-to-end-tests/flink-kinesis-test > >> > >> Are there any suggestions or comments regarding this? > >> > >> Thanks, > >> Thomas > >> > >
