GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/363
METRON-578: Missing error handling bolts for enrichment and threat intel
In the flux files, added outputs for the various `GenericEnrichmentBolt`
used throughout, to tie their error output streams to Kafka topics.
Specifically, enrichment is tied to enrichments_error and threat intel is tied
to threatintel_error.
Quick-dev and the Ambari mpack are updated to ensure the topics are
created, so that no errors occur on usage. Both were run up to ensure that the
topics were created and the flux files don't give errors.
To test, an adjustment was made to the `EnrichmentIntegrationTest` to have
an enrichment bolt that always throws errors. This bolt is derived from
`GenericEnrichmentBolt` (which has been slightly refactored) to ensure that it
uses the same error logic, but ignores the actual logic that leads to errors.
This was done because I couldn't find input that resulted in the error stream
code path in actual practice. Other errors can result, but don't go to the
error stream (e.g. JSON decoding errors).
As a note, the `ErrorEnrichmentBolt` invariably ends up in the stack traces
being printed (which is not handled by `ErrorEnrichmentBolt` or
`GenericEnrichmentBolt`).
Given that the test explicitly calls `UnitTestHelper.verboseLogging()`, I'm
not sure how to suppress that without killing actual errors at the same time
is. Open to suggestions if someone knows a more appropriate way to handle it
(if we care).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron error_handling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/363.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #363
----
commit 1f630472a707d542daff932eb2a9e264ebc38b99
Author: justinjleet <[email protected]>
Date: 2016-11-09T16:45:48Z
Adding enrichment and threatintel error handling to Kafka
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
