Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[Mohammad Kargar]

I will.

Thanks anyway
Mohammad

On Feb 14, 2017 7:24 PM, "Cody Koeninger"  wrote:

> Not sure what to tell you at that point - maybe compare what is
> present in ZK to a known working group.
>
> On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar 
> wrote:
> > Yes offset nodes are in zk and I can get the values.
> >
> > On Feb 14, 2017 6:54 PM, "Cody Koeninger"  wrote:
> >>
> >> Are there offset nodes in ZK?
> >>
> >> for a consumer group named mygroup, a topic named test, and partition
> >> 0 you should be able to connect to ZK and do e.g.
> >>
> >> get /consumers/mygroup/offsets/test/0
> >>
> >> If those don't exist, those are the ZK nodes you would need to make
> >> sure get created / updated from your spark job.
> >>
> >>
> >>
> >> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar 
> >> wrote:
> >> > I'm running 0.10 version and
> >> >
> >> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >> >
> >> >  lists the group.
> >> >
> >> > On Feb 14, 2017 6:34 PM, "Cody Koeninger"  wrote:
> >> >>
> >> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
> >> >> versions of the kafka server.  Which version of the server are you
> >> >> running?
> >> >>
> >> >> At any rate, the low-level consumer used by the
> >> >> spark-streaming-kafka-0-8 integration isn't going to create a
> consumer
> >> >> group or otherwise interact with them.  If
> >> >>
> >> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >> >>
> >> >> doesn't show the group you're expecting, have you used a consumer
> that
> >> >> would have created it?  Or did you explicitly try to create the
> >> >> related ZK nodes yourself?
> >> >>
> >> >>
> >> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar 
> >> >> wrote:
> >> >> > Tried
> >> >> >
> >> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
> >> >> > --describe
> >> >> > --group mygroup
> >> >> >
> >> >> > and got
> >> >> >
> >> >> > No topic available for consumer group provided!
> >> >> >
> >> >> > Let me know if you still need the code (have to make changes before
> >> >> > sharing).
> >> >> >
> >> >> > Thanks
> >> >> >
> >> >> >
> >> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger <
> c...@koeninger.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> Can you explain what wasn't successful and/or show code?
> >> >> >>
> >> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar <
> mkar...@phemi.com>
> >> >> >> wrote:
> >> >> >> > As explained here, direct approach of integration between spark
> >> >> >> > streaming
> >> >> >> > and kafka does not update offsets in Zookeeper, hence
> >> >> >> > Zookeeper-based
> >> >> >> > Kafka
> >> >> >> > monitoring tools will not show progress (details).  We followed
> >> >> >> > the
> >> >> >> > recommended workaround to update the zookeeper with the latest
> >> >> >> > offset
> >> >> >> > after
> >> >> >> > each batch, but no success. Wondering if there's any end-to-end
> >> >> >> > example
> >> >> >> > we
> >> >> >> > can use?
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Mohammad
> >> >> >> >
> >> >> >
> >> >> >
>

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[Cody Koeninger]

Not sure what to tell you at that point - maybe compare what is
present in ZK to a known working group.

On Tue, Feb 14, 2017 at 9:06 PM, Mohammad Kargar  wrote:
> Yes offset nodes are in zk and I can get the values.
>
> On Feb 14, 2017 6:54 PM, "Cody Koeninger"  wrote:
>>
>> Are there offset nodes in ZK?
>>
>> for a consumer group named mygroup, a topic named test, and partition
>> 0 you should be able to connect to ZK and do e.g.
>>
>> get /consumers/mygroup/offsets/test/0
>>
>> If those don't exist, those are the ZK nodes you would need to make
>> sure get created / updated from your spark job.
>>
>>
>>
>> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar 
>> wrote:
>> > I'm running 0.10 version and
>> >
>> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
>> >
>> >  lists the group.
>> >
>> > On Feb 14, 2017 6:34 PM, "Cody Koeninger"  wrote:
>> >>
>> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
>> >> versions of the kafka server.  Which version of the server are you
>> >> running?
>> >>
>> >> At any rate, the low-level consumer used by the
>> >> spark-streaming-kafka-0-8 integration isn't going to create a consumer
>> >> group or otherwise interact with them.  If
>> >>
>> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
>> >>
>> >> doesn't show the group you're expecting, have you used a consumer that
>> >> would have created it?  Or did you explicitly try to create the
>> >> related ZK nodes yourself?
>> >>
>> >>
>> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar 
>> >> wrote:
>> >> > Tried
>> >> >
>> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
>> >> > --describe
>> >> > --group mygroup
>> >> >
>> >> > and got
>> >> >
>> >> > No topic available for consumer group provided!
>> >> >
>> >> > Let me know if you still need the code (have to make changes before
>> >> > sharing).
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger 
>> >> > wrote:
>> >> >>
>> >> >> Can you explain what wasn't successful and/or show code?
>> >> >>
>> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar 
>> >> >> wrote:
>> >> >> > As explained here, direct approach of integration between spark
>> >> >> > streaming
>> >> >> > and kafka does not update offsets in Zookeeper, hence
>> >> >> > Zookeeper-based
>> >> >> > Kafka
>> >> >> > monitoring tools will not show progress (details).  We followed
>> >> >> > the
>> >> >> > recommended workaround to update the zookeeper with the latest
>> >> >> > offset
>> >> >> > after
>> >> >> > each batch, but no success. Wondering if there's any end-to-end
>> >> >> > example
>> >> >> > we
>> >> >> > can use?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Mohammad
>> >> >> >
>> >> >
>> >> >

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[Mohammad Kargar]

Yes offset nodes are in zk and I can get the values.

On Feb 14, 2017 6:54 PM, "Cody Koeninger"  wrote:

> Are there offset nodes in ZK?
>
> for a consumer group named mygroup, a topic named test, and partition
> 0 you should be able to connect to ZK and do e.g.
>
> get /consumers/mygroup/offsets/test/0
>
> If those don't exist, those are the ZK nodes you would need to make
> sure get created / updated from your spark job.
>
>
>
> On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar 
> wrote:
> > I'm running 0.10 version and
> >
> > ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >
> >  lists the group.
> >
> > On Feb 14, 2017 6:34 PM, "Cody Koeninger"  wrote:
> >>
> >> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
> >> versions of the kafka server.  Which version of the server are you
> >> running?
> >>
> >> At any rate, the low-level consumer used by the
> >> spark-streaming-kafka-0-8 integration isn't going to create a consumer
> >> group or otherwise interact with them.  If
> >>
> >> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
> >>
> >> doesn't show the group you're expecting, have you used a consumer that
> >> would have created it?  Or did you explicitly try to create the
> >> related ZK nodes yourself?
> >>
> >>
> >> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar 
> >> wrote:
> >> > Tried
> >> >
> >> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
> >> > --describe
> >> > --group mygroup
> >> >
> >> > and got
> >> >
> >> > No topic available for consumer group provided!
> >> >
> >> > Let me know if you still need the code (have to make changes before
> >> > sharing).
> >> >
> >> > Thanks
> >> >
> >> >
> >> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger 
> >> > wrote:
> >> >>
> >> >> Can you explain what wasn't successful and/or show code?
> >> >>
> >> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar 
> >> >> wrote:
> >> >> > As explained here, direct approach of integration between spark
> >> >> > streaming
> >> >> > and kafka does not update offsets in Zookeeper, hence
> Zookeeper-based
> >> >> > Kafka
> >> >> > monitoring tools will not show progress (details).  We followed the
> >> >> > recommended workaround to update the zookeeper with the latest
> offset
> >> >> > after
> >> >> > each batch, but no success. Wondering if there's any end-to-end
> >> >> > example
> >> >> > we
> >> >> > can use?
> >> >> >
> >> >> > Thanks,
> >> >> > Mohammad
> >> >> >
> >> >
> >> >
>

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[Cody Koeninger]

Are there offset nodes in ZK?

for a consumer group named mygroup, a topic named test, and partition
0 you should be able to connect to ZK and do e.g.

get /consumers/mygroup/offsets/test/0

If those don't exist, those are the ZK nodes you would need to make
sure get created / updated from your spark job.



On Tue, Feb 14, 2017 at 8:40 PM, Mohammad Kargar  wrote:
> I'm running 0.10 version and
>
> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
>
>  lists the group.
>
> On Feb 14, 2017 6:34 PM, "Cody Koeninger"  wrote:
>>
>> As far as I know, kafka-consumer-groups.sh wasn't available with 0.8
>> versions of the kafka server.  Which version of the server are you
>> running?
>>
>> At any rate, the low-level consumer used by the
>> spark-streaming-kafka-0-8 integration isn't going to create a consumer
>> group or otherwise interact with them.  If
>>
>> ./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list
>>
>> doesn't show the group you're expecting, have you used a consumer that
>> would have created it?  Or did you explicitly try to create the
>> related ZK nodes yourself?
>>
>>
>> On Tue, Feb 14, 2017 at 8:07 PM, Mohammad Kargar 
>> wrote:
>> > Tried
>> >
>> > ${kafka-home}/bin/kafka-consumer-groups.sh --zookeeper zhost:zp
>> > --describe
>> > --group mygroup
>> >
>> > and got
>> >
>> > No topic available for consumer group provided!
>> >
>> > Let me know if you still need the code (have to make changes before
>> > sharing).
>> >
>> > Thanks
>> >
>> >
>> > On Tue, Feb 14, 2017 at 5:57 PM, Cody Koeninger 
>> > wrote:
>> >>
>> >> Can you explain what wasn't successful and/or show code?
>> >>
>> >> On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar 
>> >> wrote:
>> >> > As explained here, direct approach of integration between spark
>> >> > streaming
>> >> > and kafka does not update offsets in Zookeeper, hence Zookeeper-based
>> >> > Kafka
>> >> > monitoring tools will not show progress (details).  We followed the
>> >> > recommended workaround to update the zookeeper with the latest offset
>> >> > after
>> >> > each batch, but no success. Wondering if there's any end-to-end
>> >> > example
>> >> > we
>> >> > can use?
>> >> >
>> >> > Thanks,
>> >> > Mohammad
>> >> >
>> >
>> >

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[Cody Koeninger]

Can you explain what wasn't successful and/or show code?

On Tue, Feb 14, 2017 at 6:03 PM, Mohammad Kargar  wrote:
> As explained here, direct approach of integration between spark streaming
> and kafka does not update offsets in Zookeeper, hence Zookeeper-based Kafka
> monitoring tools will not show progress (details).  We followed the
> recommended workaround to update the zookeeper with the latest offset after
> each batch, but no success. Wondering if there's any end-to-end example we
> can use?
>
> Thanks,
> Mohammad
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: streaming-kafka-0-8-integration (direct approach) and monitoring

[bryan . jeffrey]

Mohammad, 


We store our offsets in Cassandra,  and use that for tracking. This solved a 
few issues for us,  as it provides a good persistence mechanism even when 
you're reading from multiple clusters. 


Bryan Jeffrey 




Get Outlook for Android









On Tue, Feb 14, 2017 at 7:03 PM -0500, "Mohammad Kargar"  
wrote:










As explained here, direct approach of integration between spark streaming and 
kafka does not update offsets in Zookeeper, hence Zookeeper-based Kafka 
monitoring tools will not show progress (details).  We followed the recommended 
workaround to update the zookeeper with the latest offset after each batch, but 
no success. Wondering if there's any end-to-end example we can use?

Thanks,
Mohammad

streaming-kafka-0-8-integration (direct approach) and monitoring

[Mohammad Kargar]

As explained here
,
direct approach of integration between spark streaming and kafka does not
update offsets in Zookeeper, hence Zookeeper-based Kafka monitoring tools
will not show progress (details).  We followed the recommended workaround
to update the zookeeper with the latest offset after each batch, but no
success. Wondering if there's any end-to-end example we can use?

Thanks,
Mohammad