Hi Henry, Could you share the streams configuration for your apps? I.e., the part where you assign application id and all the rest of the configs (just configs, not code).
Thanks Eno > On May 2, 2017, at 8:53 AM, Henry Thacker <he...@henrythacker.com> wrote: > > Thanks all for your replies - I have checked out the docs which were very > helpful. > > I have now moved the separate topic streams to different processes each > with their own app.id and I'm getting the following pattern, with no data > consumed: > > "Starting stream thread [StreamThread-1] > Discovered coordinator .... for group .. > Marking the coordinator .... dead for group .. > Discovered coordinator .... for group .. > Marking the coordinator .... dead for group .." > > The discover and dead states repeat every few minutes. > > During this time, the broker logs look happy. > > One other, hopefully unrelated point, is this cluster is all SSL encrypted. > > Thanks, > Henry > > -- > Henry Thacker > > On 29 April 2017 at 05:31:30, Matthias J. Sax (matth...@confluent.io) wrote: > >> Henry, >> >> you might want to check out the docs, that give an overview of the >> architecture: >> http://docs.confluent.io/current/streams/architecture.html#example >> >> Also, I am wondering why your application did not crash: I would expect >> an exception like >> >> java.lang.IllegalArgumentException: Assigned partition foo-2 for >> non-subscribed topic regex pattern; subscription pattern is bar >> >> Maybe you just don't hit it, because both topics have a single partition >> and not multiple. >> >> Out of interest though, had I subscribed for both topics in one subscriber >> - I would have expected records for both topics interleaved >> >> >> Yes. That should happen. >> >> why when >> >> running this in two separate processes do I not observe the same? >> >> >> Not sure what you mean by this? >> >> If I fix this by changing the application ID for each streaming process - >> does this mean I lose the ability to share state stores between the >> applications? >> >> >> Yes. >> >> >> If both your topics are single partitioned, and you want to share state, >> you will not be able to run with more then one thread in your Streams app. >> >> The only way to work around this, would be to copy the data into another >> topic with more partitions before you process them -- of course, this >> would mean data duplication. >> >> >> -Matthias >> >> >> On 4/28/17 12:45 PM, Henry Thacker wrote: >> >> Thanks Michael and Eno for your help - I always thought the unit of >> parallelism was a combination of topic & partition rather than just >> partition. >> >> Out of interest though, had I subscribed for both topics in one subscriber >> - I would have expected records for both topics interleaved, why when >> running this in two separate processes do I not observe the same? Just >> wanting to try and form a mental model of how this is all working - I will >> try and look through some code over the weekend. >> >> If I fix this by changing the application ID for each streaming process - >> does this mean I lose the ability to share state stores between the >> applications? >> >> Unfortunately the data on the input topics are provided by a third party >> component which sends these keyless messages on a single partition per >> topic, so I have little ability to fix this at source :-( >> >> Thanks, >> Henry >> >> >> ------------------------------ >>
