Re: Kafka + Direct + Zookeeper

[Ted Yu]

Please see Approach 2 in:

http://spark.apache.org/docs/latest/streaming-kafka-integration.html




> On May 13, 2015, at 6:10 AM, James King  wrote:
> 
> I'm trying Kafka Direct approach (for consume) but when I use only this 
> config:
> 
> kafkaParams.put("group.id", groupdid);
> kafkaParams.put("zookeeper.connect", zookeeperHostAndPort + "/cb_kafka");
> 
> I get this 
> 
> Exception in thread "main" org.apache.spark.SparkException: Must specify 
> metadata.broker.list or bootstrap.servers
> 
> Zookeeper should have enough information to provide connection details?
> 
> or am I missing something?
> 
> 

RE: Kafka + Direct + Zookeeper

[Shao, Saisai]

Hi James,

For Kafka direct approach, you don’t need to specify “group.id” and 
“zookeeper.connect”, instead you need to specify the broker list through 
“metadata.broker.list” or “bootstrap.servers”. Internally direct approach uses 
low level api of Kafka, so it does not involve ZK.

Thanks
Jerry

From: James King [mailto:jakwebin...@gmail.com]
Sent: Wednesday, May 13, 2015 9:10 PM
To: user
Subject: Kafka + Direct + Zookeeper

I'm trying Kafka Direct approach (for consume) but when I use only this config:

kafkaParams.put("group.id<http://group.id>", groupdid);
kafkaParams.put("zookeeper.connect", zookeeperHostAndPort + "/cb_kafka");

I get this

Exception in thread "main" org.apache.spark.SparkException: Must specify 
metadata.broker.list or bootstrap.servers

Zookeeper should have enough information to provide connection details?

or am I missing something?

Kafka + Direct + Zookeeper

[James King]

I'm trying Kafka Direct approach (for consume) but when I use only this
config:

kafkaParams.put("group.id", groupdid);
kafkaParams.put("zookeeper.connect", zookeeperHostAndPort + "/cb_kafka");

I get this

Exception in thread "main" org.apache.spark.SparkException: Must specify
metadata.broker.list or bootstrap.servers

Zookeeper should have enough information to provide connection details?

or am I missing something?