Nick Allen created METRON-1832:
----------------------------------
Summary: Recurrent Large Indexing Error Messages
Key: METRON-1832
URL: https://issues.apache.org/jira/browse/METRON-1832
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
If any index destination like HDFS, Elasticsearch, or Solr goes down while the
Indexing topology is running, an error message is created and sent back to the
user-defined error topic. By default, this is defined to also be the
'indexing' topic.
The Indexing topology then consumes this error message and attempts to write it
again. If the index destination is still down, another error occurs and another
error message is created that encapsulates the original error message. That
message is then sent to the 'indexing' topic, which is later consumed, yet
again, by the Indexing topology.
These error messages will continue to be recycled and grow larger and larger as
each new error message encapsulates all previous error messages in the
"raw_message" field.
Once the index destination recovers, one giant error message will finally be
written that contains massively duplicated, useless information which can
further negatively impact performance of the index destination.
See attachment for example of once of these error messages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
