Domenic Puzio created METRON-569:
------------------------------------
Summary: Enrichment topology duplicates messages
Key: METRON-569
URL: https://issues.apache.org/jira/browse/METRON-569
Project: Metron
Issue Type: Bug
Reporter: Domenic Puzio
When running the 'enrichment' topology, I get duplicate message being indexed.
For example, I put 100 messages into the 'enrichment' Kafka queue and I get 175
messages onto the 'indexing' Kafka queue. This happens when I am running the
'enrichment' topology with one or more enrichment bolt.
This is an acking issue within the JoinBolt class. When a message does not
"complete" the join (like when it is the first message in a pair of message to
get joined) it does not get acked. This means that this message will get
replayed through Storm, causing message duplication further down the road and
tons of additional overhead. Adding the correct acking resolves this problem.
I will add the PR for this shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
