Hi Peter, the processing pipelines introduced in Graylog 2.0.0 might help you with your use case: http://docs.graylog.org/en/2.0/pages/pipelines.html
Cheers, Jochen On Tuesday, 26 April 2016 13:14:35 UTC+2, Peter Krammer wrote: > > Hi, > > About our Setup: > We use filebeat as a forwarder and the beats input plugin in graylog and > we get around 5.000 to 10.000 messages per seconds from hundreds of > sources. > So we have a elasticsearch cluster and a graylog cluster. > Because we had experience with splunk, we let filebeat tag each message > with additional fields: sourcetype, customer, project > We only have one global input running for the beats input plugin on a > single port (5044) and we want to extract the time into the timestamp > field, so that message time matches the time in the graylog index. > We also want to extract additional attributes dependig on the message. > We now have a lot of extractors with with GROK and a regex condition. > > The Problem: > Some of the extractors interact with each other, which sometime leads to > unwanted behaviour. > We want to tell Graylog if sourcetype matches X do extractors 1, 2, 3 if > sourcetype matches Y do extractors 4, 5 and so on. > > Possible Solutions: > 1. Get Graylog to use different fields for the condition and the grok > pattern. > Not possible right now as far as I know. > 2. Do the extractions in the filebeat config. > Not possible right now as far as I know. > 3. Somehow combine the sourcetype and the message field to one field, than > match with regex on that field and do grok extraction and finally delete > the combined field. > Seems like an ugly hack, that brings possibly additional problems. > > Does anybody know if there is a good working and efficient solution for > this? > Because we are about to scale up even more to more than a Terrabyte of > Data per Day. > > best regards, > Peter > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/db0cce66-be39-4ea1-b1dd-ce4d3ae527a4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
