GitHub user DomenicPuzio opened a pull request:
https://github.com/apache/incubator-metron/pull/359
Change acking to prevent duplicate tuples in enrichment topology
Adding this ack statement prevents duplicate messages from being sent
through the _enrichment_ topology. Previously, in the _JoinBolt_, when a
message does not complete a join, it does not get acked - this cause the
message to get re-sent through the topology, resulting in duplicate messages
being indexed.
This is a show-stopping bug, impacting performance and resulting in
duplicate data.
I have tested this in the Quick-Dev enrichment as well as in our production
system at high velocity.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DomenicPuzio/incubator-metron METRON-569
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/359.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #359
----
commit 7920eb575f1848786aeccd3d7d47cf458119f367
Author: DomenicPuzio <[email protected]>
Date: 2016-11-15T20:43:12Z
Adding critical ack statement to prevent duplicate messages
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
