I will. Thanks anyway Mohammad
On Feb 14, 2017 7:24 PM, "Cody Koeninger" <c...@koeninger.org> wrote: > Not sure what to tell you at that point - maybe compare what is > present in ZK to a known working group. > > On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar <mkar...@phemi.com> > wrote: > > Yes offset nodes are in zk and I can get the values. > > > > On Feb 14, 2017 6:54 PM, "Cody Koeninger" <c...@koeninger.org> wrote: > >> > >> Are there offset nodes in ZK? > >> > >> for a consumer group named mygroup, a topic named test, and partition > >> 0 you should be able to connect to ZK and do e.g. > >> > >> get /consumers/mygroup/offsets/test/0 > >> > >> If those don't exist, those are the ZK nodes you would need to make > >> sure get created / updated from your spark job. > >> > >> > >> > >> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar <mkar...@phemi.com> > >> wrote: > >> > I'm running 0.10 version and > >> > > >> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list > >> > > >> > lists the group. > >> > > >> > On Feb 14, 2017 6:34 PM, "Cody Koeninger" <c...@koeninger.org> wrote: > >> >> > >> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8 > >> >> versions of the kafka server. Which version of the server are you > >> >> running? > >> >> > >> >> At any rate, the low-level consumer used by the > >> >> spark-streaming-kafka-0-8 integration isn't going to create a > consumer > >> >> group or otherwise interact with them. If > >> >> > >> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list > >> >> > >> >> doesn't show the group you're expecting, have you used a consumer > that > >> >> would have created it? Or did you explicitly try to create the > >> >> related ZK nodes yourself? > >> >> > >> >> > >> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar <mkar...@phemi.com> > >> >> wrote: > >> >> > Tried > >> >> > > >> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp > >> >> > --describe > >> >> > --group mygroup > >> >> > > >> >> > and got > >> >> > > >> >> > No topic available for consumer group provided! > >> >> > > >> >> > Let me know if you still need the code (have to make changes before > >> >> > sharing). > >> >> > > >> >> > Thanks > >> >> > > >> >> > > >> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger < > c...@koeninger.org> > >> >> > wrote: > >> >> >> > >> >> >> Can you explain what wasn't successful and/or show code? > >> >> >> > >> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar < > mkar...@phemi.com> > >> >> >> wrote: > >> >> >> > As explained here, direct approach of integration between spark > >> >> >> > streaming > >> >> >> > and kafka does not update offsets in Zookeeper, hence > >> >> >> > Zookeeper-based > >> >> >> > Kafka > >> >> >> > monitoring tools will not show progress (details). We followed > >> >> >> > the > >> >> >> > recommended workaround to update the zookeeper with the latest > >> >> >> > offset > >> >> >> > after > >> >> >> > each batch, but no success. Wondering if there's any end-to-end > >> >> >> > example > >> >> >> > we > >> >> >> > can use? > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Mohammad > >> >> >> > > >> >> > > >> >> > >