[
https://issues.apache.org/jira/browse/METRON-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15683408#comment-15683408
]
ASF GitHub Bot commented on METRON-578:
---------------------------------------
GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/363
METRON-578: Missing error handling bolts for enrichment and threat intel
In the flux files, added outputs for the various `GenericEnrichmentBolt`
used throughout, to tie their error output streams to Kafka topics.
Specifically, enrichment is tied to enrichments_error and threat intel is tied
to threatintel_error.
Quick-dev and the Ambari mpack are updated to ensure the topics are
created, so that no errors occur on usage. Both were run up to ensure that the
topics were created and the flux files don't give errors.
To test, an adjustment was made to the `EnrichmentIntegrationTest` to have
an enrichment bolt that always throws errors. This bolt is derived from
`GenericEnrichmentBolt` (which has been slightly refactored) to ensure that it
uses the same error logic, but ignores the actual logic that leads to errors.
This was done because I couldn't find input that resulted in the error stream
code path in actual practice. Other errors can result, but don't go to the
error stream (e.g. JSON decoding errors).
As a note, the `ErrorEnrichmentBolt` invariably ends up in the stack traces
being printed (which is not handled by `ErrorEnrichmentBolt` or
`GenericEnrichmentBolt`).
Given that the test explicitly calls `UnitTestHelper.verboseLogging()`, I'm
not sure how to suppress that without killing actual errors at the same time
is. Open to suggestions if someone knows a more appropriate way to handle it
(if we care).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/incubator-metron error_handling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/363.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #363
----
commit 1f630472a707d542daff932eb2a9e264ebc38b99
Author: justinjleet <[email protected]>
Date: 2016-11-09T16:45:48Z
Adding enrichment and threatintel error handling to Kafka
----
> Missing error handling bolts for enrichment and threat intel
> ------------------------------------------------------------
>
> Key: METRON-578
> URL: https://issues.apache.org/jira/browse/METRON-578
> Project: Metron
> Issue Type: Improvement
> Affects Versions: 0.2.1BETA
> Reporter: Justin Leet
> Assignee: Justin Leet
>
> TL;DR - we need to add error handling to enrichments/threat intel
> Metron has parsers, enrichment + threat intel, and indexing topologies
> currently. Parsers and and enrichment have bolts that write to error topics
> in Kafka
> # indexing_error
> # parser_error
> # parser_invalid
> The GenericEnrichmentBolt handles errors gracefully by passing along failed
> enrichment tuples un-enriched and additionally emitting the tuple to an
> "error" stream, however there is currently no plumbing to handle the error
> stream.
> {code:java}
> } catch (Exception e) {
> LOG.error("[Metron] Unable to enrich message: " + rawMessage, e);
> JSONObject error = ErrorUtils.generateErrorMessage("Enrichment problem:
> " + rawMessage, e);
> if (key != null) {
> collector.emit(enrichmentType, new Values(key, enrichedMessage,
> subGroup));
> }
> collector.emit("error", new Values(error));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
