[
https://issues.apache.org/jira/browse/METRON-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15873273#comment-15873273
]
ASF GitHub Bot commented on METRON-694:
---------------------------------------
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/453
I lean toward having a separate kafka queue for errors. Errors are lower
in volume, but are a lot more verbose. You are essentially indexing large text
document when you do this and you can easily bring down ES when something goes
wrong and you get lots of errors. I want the ability to first tune my cluster
enough where the volume of errors is rather low. In my daily workflow I would
just monitor the kafka queue via the command line consumer until I reduce the
number of errors. Then if i needed to (and i want this to be optional) I can
index them into ES. Or I can just dump them out to file straight out of kafka
> Index Errors from Topologies
> ----------------------------
>
> Key: METRON-694
> URL: https://issues.apache.org/jira/browse/METRON-694
> Project: Metron
> Issue Type: Bug
> Reporter: Ryan Merriman
>
> Need to make sure (and review) that all the bolts write into the error queue.
> Errors should then be consumed from the error queue and indexed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
