Cody,
Yes - I was able to verify that I am not seeing duplicate calls to
createDirectStream. If the spark-streaming-kafka-0-10 will work on a 2.3
cluster I can go ahead and give that a shot.
Regards,
Bryan Jeffrey
On Fri, Aug 31, 2018 at 11:56 AM Cody Koeninger wrote:
> Just to be 100% sure,
Just to be 100% sure, when you're logging the group id in
createDirectStream, you no longer see any duplicates?
Regarding testing master, is the blocker that your spark cluster is on
2.3? There's at least a reasonable chance that building an
application assembly jar that uses the master version j
Cody,
We are connecting to multiple clusters for each topic. I did experiment
this morning with both adding a cluster identifier to the group id, as well
as simply moving to use only a single one of our clusters. Neither of
these were successful. I am not able to run a test against master now.
I doubt that fix will get backported to 2.3.x
Are you able to test against master? 2.4 with the fix you linked to
is likely to hit code freeze soon.
>From a quick look at your code, I'm not sure why you're mapping over
an array of brokers. It seems like that would result in different
streams wi
