Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/453
The latest commit includes the changes discussed. Most error messages now
go to the indexing topic by default so there is no need for another error
indexing topology. The error topic setting for the enrichment and indexing
topologies are already exposed through flux properties so I left them there. I
added a "parser.error.topic" property to the global config for parser
topologies (should indexing topic be the default if that property is missing or
should we let it fail?).
For the concern about indexing errors getting in a loop, I thought of a
simple solution that required no extra work. If an error happens in the
indexing topology, simply send it to a different error topic. Would this be
acceptable? I wanted to throw this out there before implementing custom logic
in the indexing topology classes.
The error message structure did change slightly. In my testing I realized
that the type for the "raw_message" field could be different (byte[] vs JSON)
depending on where the error happened (pre vs post parse). If an error message
with byte[] as raw_message was indexed first, Elasticsearch would error on
subsequent error messages with JSON as raw_message. To address this, I
serialized JSON raw messages before storing them in this field. Now
raw_message is always and string and raw_message_bytes is always a byte array.
All tests are passing and it has been tested in quick dev. Some tips on
testing the different cases:
- the mock Bro sensor in quick dev contains a couple unparsable messages so
parser errors should show up in the error index by default
- Invalid parser errors can be generated by adding a field validation to
the global config that will always fail
- for testing indexing errors, copy a message from the indexing topic,
change a field with number type (ip_src_port for example) to a non-number
string, then put that message back into the indexing topic
@cestella, would be good to get your opinion on the MessageGetter classes
and where that ended up. Not sure I love the syntax for default message
getters.
The MPack changes are now minimal but it should still be tested there and
that test is pending. The documentation still needs to be updated so please
don't merge this until that is done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
