[
https://issues.apache.org/jira/browse/METRON-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671688#comment-15671688
]
ASF GitHub Bot commented on METRON-569:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/359
Hey @DomenicPuzio thanks man :)
The problem with placing the ack there is that in the situation where the
enrichment adapter worker gets killed, we would *like* the data to replay from
the anchor point (the SplitBolt) because Storm will have respawned the worker
elsewhere and we'd like that enrichment to be captured. If you keep the ack
where you have it, the message will never join if the enrichment worker gets
killed, so the message will ultimately get dropped, which isn't ideal.
Ultimately, I think we can all agree that we need to ack at LEAST the tuple
on the `message` stream, but I think that we want to ack it AFTER the join has
happened.
Regarding seeing dup data, I believe you. I am just trying to wrap my head
around how it happens, but, as they say, the proof of the pudding is in the
eating. :)
> Enrichment topology duplicates messages
> ---------------------------------------
>
> Key: METRON-569
> URL: https://issues.apache.org/jira/browse/METRON-569
> Project: Metron
> Issue Type: Bug
> Reporter: Domenic Puzio
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When running the 'enrichment' topology, I get duplicate message being
> indexed. For example, I put 100 messages into the 'enrichment' Kafka queue
> and I get 175 messages onto the 'indexing' Kafka queue. This happens when I
> am running the 'enrichment' topology with one or more enrichment bolt.
> This is an acking issue within the JoinBolt class. When a message does not
> "complete" the join (like when it is the first message in a pair of message
> to get joined) it does not get acked. This means that this message will get
> replayed through Storm, causing message duplication further down the road and
> tons of additional overhead. Adding the correct acking resolves this problem.
> I will add the PR for this shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
