For setting a list of brokers in kafkaSpout, I believe there are 2 options:
If you use StaticHosts then you need to add GlobalPartitionInformation in
which you have to specify each partition and its corresponding broker host
GlobalPartitionInformation partitions = new GlobalPartitionInformation();
partitions.addPartition(0,new Broker("10.22.2.79", 9092));
//add more partitions here.
BrokerHosts hosts = new StaticHosts(partitions);
Alternatively, If you use ZkHosts then you need to pass in the complete
zookeeper connection string e.g. lolcalhost:9092 , optionally there is a
constructor that allows you to specify a second argument which is zkroot ,
by default it is set to /brokers which should work with default kafka
installation. The code in ZlHosts looks under zkroot/topics/<topic that you
set in SpoutConfig>/partitions to figure out the number of partitions and
leader for each partition.
Thanks
Parth
On Mon, Jul 28, 2014 at 9:54 AM, Adrian Landman <[email protected]>
wrote:
> I am writing a topology that pulls messages from a topic, does some work,
> and then writes them back on a different topic. I have been having some
> issues so I created my own small topology that just pulls a message, prints
> the contents, and then stores them back on a new topic. I finally got this
> to work, but it raised a question.
>
> To create the spout I need to either pass in the kafka location sans port
> (e.g. localhost) or use 2181 as the port.
>
> To create the producer bolt I need to pass in the broker port (e.g. 9092)
> or I get an array out of bounds exception when creating the Producer.
>
> When we were using kafka7 and
>
> https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka/src/jvm/storm/kafka
> for our storm/kafka integration I believe that we used the same broker
> list for both our spout and our producer. Is there anyway to do the same
> with kafka8 and the new storm-kafka project? Also, if w want to pass in a
> list, I know that in the KafkaBolt we can set metadata.broker.list to a
> comma separated list of brokers (1.1.1.1:9092, 1.1.1.2:9092) but can we
> do the same for the spout? Or is there any reason to? ZkHost takes in a
> String, but I didn't see anything that specified the format.
>
--
Thanks
Parth
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
