Hi,
About our Setup:
We use filebeat as a forwarder and the beats input plugin in graylog and we
get around 5.000 to 10.000 messages per seconds from hundreds of sources.
So we have a elasticsearch cluster and a graylog cluster.
Because we had experience with splunk, we let filebeat tag each message
with additional fields: sourcetype, customer, project
We only have one global input running for the beats input plugin on a
single port (5044) and we want to extract the time into the timestamp
field, so that message time matches the time in the graylog index.
We also want to extract additional attributes dependig on the message.
We now have a lot of extractors with with GROK and a regex condition.
The Problem:
Some of the extractors interact with each other, which sometime leads to
unwanted behaviour.
We want to tell Graylog if sourcetype matches X do extractors 1, 2, 3 if
sourcetype matches Y do extractors 4, 5 and so on.
Possible Solutions:
1. Get Graylog to use different fields for the condition and the grok
pattern.
Not possible right now as far as I know.
2. Do the extractions in the filebeat config.
Not possible right now as far as I know.
3. Somehow combine the sourcetype and the message field to one field, than
match with regex on that field and do grok extraction and finally delete
the combined field.
Seems like an ugly hack, that brings possibly additional problems.
Does anybody know if there is a good working and efficient solution for
this?
Because we are about to scale up even more to more than a Terrabyte of Data
per Day.
best regards,
Peter
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/f33db0cc-c344-42b7-82e9-04ba624680c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
