[
https://issues.apache.org/jira/browse/METRON-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864067#comment-15864067
]
ASF GitHub Bot commented on METRON-694:
---------------------------------------
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/453
In response to "Is there any reason we didn't just use the normal indexing
topology". Here are the issues I see with doing that. First, I think we
should be careful about putting additional load on the indexing topology. For
parser errors I think it's a wash, an error gets indexed with the original
message instead of the parsed message. But for enrichment errors, new error
messages are generated and passed along. If we send all errors and messages to
the same indexing topology a burst of errors could affect the latency of other
messages.
Here is another scenario that could cause problems. Imagine a message
fails to index and an error is thrown in the indexing topology. If we send all
errors and messages to the same topology, that error could be replayed in the
indexing topology over and over. We could add some logic to keep that from
happening but now writer bolts are more complex and specific to the indexing
topology. You will notice the error indexing topology is different in that it
does not write back out to Kafka.
I feel like by mixing error messages and legitimate messages we are asking
for trouble. Having them separate allows us to tune the topologies separately
and avoid the problems I mention above. Tuning the indexing topology as it
stands now is challenging enough. I can understand not wanting to allocate
additional worker slots though. Maybe I am being too conservative.
> Index Errors from Topologies
> ----------------------------
>
> Key: METRON-694
> URL: https://issues.apache.org/jira/browse/METRON-694
> Project: Metron
> Issue Type: Bug
> Reporter: Ryan Merriman
>
> Need to make sure (and review) that all the bolts write into the error queue.
> Errors should then be consumed from the error queue and indexed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
