down votefavorite

Hi Everyone,

I am unable to find good documentation for correctly integrating Kafka with
Apache Storm Trident. I tried to look into the related previously posted
questions here, but no sufficient information.

I would like to connect Trident with Kafka as OpaqueTridentKafkaSpout. Here
is the sample Code which is currently working

            GlobalPartitionInformation globalPartitionInformation  =
new GlobalPartitionInformation(properties.getProperty("topic",
            Broker brokerForPartition0 = new Broker("IP1",9092);
            Broker brokerForPartition1 = new Broker("IP2", 9092);
            Broker brokerForPartition2 = new Broker("IP3:9092");

brokerForPartition0);//mapping from partition 0 to brokerForPartition0
brokerForPartition1);//mapping from partition 1 to brokerForPartition1
brokerForPartition2);//mapping from partition 2 to brokerForPartition2
            StaticHosts staticHosts = new
            TridentKafkaConfig tridentKafkaConfig = new
TridentKafkaConfig(hosts,properties.getProperty("topic", "mytopic"));
            tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new
            OpaqueTridentKafkaSpout kafkaSpout = new

With this I am able to generate streams for my topology as shown in the
code below

TridentTopology topology = new TridentTopology();Stream
analyticsStream  = topology.newStream("spout",

Though I have provided parallelism and my partitions, only 1 executor of
Kafka Spout is running and thereby I am unable to scale it well.

Can anyone please guide me better ways of integrating Apache Storm Trident
(2.0.0) with Apache Kafka (1.0) with 3 node cluster each ?

Also, as soon as it finishes reading from Kafka, I am getting these logs

2018-04-09 14:17:34.119 o.a.s.k.KafkaUtils
Thread-15-spout-spout-executor[79 79] [INFO] Metrics Tick: Not enough
data to calculate spout lag.  2018-04-09 14:17:34.129
o.a.s.k.KafkaUtils Thread-21-spout-spout-executor[88 88] [INFO]
Metrics Tick: Not enough data to calculate spout lag.

And in Storm UI, I can see acks for the messages above. Any suggestion to
ignore metric Ticks ?


Shaik Asifullah

Reply via email to