Integrating Storm Trident and Kafka

[Shaik Asifullah]

down votefavorite
<https://stackoverflow.com/questions/49736439/apache-storm-trident-and-kafka-spout-integration#>
*1*

Hi Everyone,

I am unable to find good documentation for correctly integrating Kafka with
Apache Storm Trident. I tried to look into the related previously posted
questions here, but no sufficient information.

I would like to connect Trident with Kafka as OpaqueTridentKafkaSpout. Here
is the sample Code which is currently working

            GlobalPartitionInformation globalPartitionInformation  =
new GlobalPartitionInformation(properties.getProperty("topic",
"mytopic"));
            Broker brokerForPartition0 = new Broker("IP1",9092);
            Broker brokerForPartition1 = new Broker("IP2", 9092);
            Broker brokerForPartition2 = new Broker("IP3:9092");

            globalPartitionInformation.addPartition(0,
brokerForPartition0);//mapping from partition 0 to brokerForPartition0
            globalPartitionInformation.addPartition(1,
brokerForPartition1);//mapping from partition 1 to brokerForPartition1
            globalPartitionInformation.addPartition(2,
brokerForPartition2);//mapping from partition 2 to brokerForPartition2
            StaticHosts staticHosts = new
StaticHosts(globalPartitionInformation);
            TridentKafkaConfig tridentKafkaConfig = new
TridentKafkaConfig(hosts,properties.getProperty("topic", "mytopic"));
            tridentKafkaConfig.scheme = new SchemeAsMultiScheme(new
StringScheme());
            OpaqueTridentKafkaSpout kafkaSpout = new
OpaqueTridentKafkaSpout(tridentKafkaConfig);

With this I am able to generate streams for my topology as shown in the
code below

TridentTopology topology = new TridentTopology();Stream
analyticsStream  = topology.newStream("spout",
kafkaSpout).parallelismHint(Integer.valueOf(properties.getProperty("spout","6")))

Though I have provided parallelism and my partitions, only 1 executor of
Kafka Spout is running and thereby I am unable to scale it well.

Can anyone please guide me better ways of integrating Apache Storm Trident
(2.0.0) with Apache Kafka (1.0) with 3 node cluster each ?

Also, as soon as it finishes reading from Kafka, I am getting these logs
constantly

2018-04-09 14:17:34.119 o.a.s.k.KafkaUtils
Thread-15-spout-spout-executor[79 79] [INFO] Metrics Tick: Not enough
data to calculate spout lag.  2018-04-09 14:17:34.129
o.a.s.k.KafkaUtils Thread-21-spout-spout-executor[88 88] [INFO]
Metrics Tick: Not enough data to calculate spout lag.

And in Storm UI, I can see acks for the messages above. Any suggestion to
ignore metric Ticks ?



Thanks,

Shaik Asifullah