Re: How to block tests of Kafka Streams until messages processed?

2016-10-20 Thread Michael Noll
> Would there be any advantage to using the kafka connect method? The advantage is to decouple the data processing (which you do in your app) from the responsibility of making the processing results available to one or more downstream systems, like Cassandra. For example, what will your applicati

Re: How to block tests of Kafka Streams until messages processed?

2016-10-20 Thread Ali Akhtar
Michael, Would there be any advantage to using the kafka connect method? Seems like it'd just add an extra step of overhead? On Thu, Oct 20, 2016 at 12:35 PM, Michael Noll wrote: > Ali, > > my main feedback is similar to what Eno and Dave have already said. In > your situation, options like th

Re: How to block tests of Kafka Streams until messages processed?

2016-10-20 Thread Michael Noll
Ali, my main feedback is similar to what Eno and Dave have already said. In your situation, options like these are what you'd currently need to do since you are writing directly from your Kafka Stream app to Cassandra, rather than writing from your app to Kafka and then using Kafka Connect to ing

Re: How to block tests of Kafka Streams until messages processed?

2016-10-19 Thread Ali Akhtar
Yeah, I did think to use that method, but as you said, it writes to a dummy output topic, which means I'd have to put in magic code just for the tests to pass (the actual code writes to cassandra and not to a dummy topic). On Thu, Oct 20, 2016 at 2:00 AM, Tauzell, Dave wrote: > For similar queu

RE: How to block tests of Kafka Streams until messages processed?

2016-10-19 Thread Tauzell, Dave
For similar queue related tests we put the check in a loop. Check every second until either the result is found or a timeout happens. -Dave -Original Message- From: Ali Akhtar [mailto:ali.rac...@gmail.com] Sent: Wednesday, October 19, 2016 3:38 PM To: users@kafka.apache.org Subject: Ho

Re: How to block tests of Kafka Streams until messages processed?

2016-10-19 Thread Eno Thereska
Wanted to add that there is nothing too special about these utility functions, they are built using a normal consumer. Eno > On 19 Oct 2016, at 21:59, Eno Thereska wrote: > > Hi Ali, > > Any chance you could recycle some of the code we have in > streams/src/test/java/.../streams/integration/

Re: How to block tests of Kafka Streams until messages processed?

2016-10-19 Thread Eno Thereska
Hi Ali, Any chance you could recycle some of the code we have in streams/src/test/java/.../streams/integration/utils? (I know we don't have it easily accessible in Maven, for now perhaps you could copy to your directory?) For example there is a method there "IntegrationTestUtils.waitUntilMinVa